I am very interested in shader programming, which is likely due to my academic background in mathematical modelling and simulation. I wrote this small primer as a means to help provide an insight into shaders and light simulation to fellow course colleagues who wanted to know more.

What is VFX? How is it achieved?

Lighting

At the heart of all image perception, whether it be simulated or real, is light. Light is said to have the properties of a wave or the properties of individual particles (photons). But more saliently for graphics rendering it has the following properties:

• Light may be reflected
• It may be absorbed
• It may transmit and be refracted through a surface
• It may diffract through narrow apertures.

The first three properties are commonly simulated as their effects are more noticeable. Lighting is simply achieved by adding the light colour (comprising of individual red, green and blue components as in reality) to the pixel colour of a surface. Crucially however, light is not a perceivable wave or stream of photons, instead it is a vector in which the surface colour of geometries may change depending on their orientation to the light. The source of the light is not illuminated like in a bulb, and it will not produce a glow: this is a post effect called bloom, which is superimposed onto a light, averages the glow colour with nearby pixels and decays according to a function of distance. Four types of lights are available for rendering: directional lights, which have an orientation but no position and so translation is ineffective (often used for global scene lighting e.g., the sun); point or spherical lights, which have position and their lighting to an object decays with distance; spot lights for lighting in a specific direction from a single source; and ambient lights, which provide a uniform lighting colour value for all objects in a scene.

There is a dichotomy between local and global illumination models. Unlike the former the latter takes into account the contribution of light reflected off surfaces into a scene and the resulting cost in energy. Ray tracing and physical based lighting are global methods. Global illumination is described by an integral equation known as the rendering equation (https://en.wikipedia.org/wiki/Rendering_equation) which contains a function for incoming light and a bidirectional reflectance distribution function (BRDF), examples of which will be described below.

Graphics programming in general

It would be imprudent to describe graphical visual effects without giving a brief description of the other processes that are occurring whilst a scene is being rendered to the screen. Fletcher and Parberry (2015) describe nine processes in the pipeline. Briefly, these are:

1) Setting the scene, where the camera is positioned, light and fog options are set and the z-buffer is prepared;

2) Visibility determination, where objects visible to the camera are selected for rendering;

3) Setting object-level rendering states, where properties such as texture maps are installed into the rendering context;

4) Geometry generation, where geometry (commonly clusters of triangles), level of detail (LOD) or procedural rendering is submitted to the API;

5) Transformation and lighting, where vertex positions are transformed into camera space for projection and vertex-level lighting is calculated;

6) Backface culling and clipping, where ‘back-facing’ triangles and vertices outside of the view frustrum are culled from rendering;

7) Screen space projection, the 3D vertex information is projected onto the 2D screen space of the view device;

8) Rasterisation, where the pixels for rendering are selected and their shading parameters are extracted;

9) Pixel shading, where the output colour for the pixels within our rendered geometry is determined.

In the past, the aforementioned processes would occur only on the CPU. However, after the introduction of graphics hardware, the most expensive process of rasterisation was accomplished by the new hardware, which brings us to today, where GPU’s may now oversee all rendering processes. Gregory (2014) describes this pipeline. In brief this pipeline encompasses:

• Vertex Shading, including the processes described in point 5 above
• Stream output, where previously processed data may be fed back into the pipeline for further processing. Processed data from hair physics simulation is an example of data that can be fed back into the vertex shading stage
• Clipping, see point 6 above
• Screen mapping, see point 7 above
• Triangle set up and traversal stages within rasterisation (point 8 above)
• Early z-test, in the past depth testing for occluded geometry was performed in the final stage, however it may be performed as an early test
• Pixel shading, see point 9 above
• Merging/blending raster operations stage. This encompasses the depth test for occluding, the alpha test, for transparency, commonly achieved by calculating a weighted average between the colours of two superimposed pixels, and the stencil test, for shadow rendering.

The output of the GPU pipeline is the frame buffer, containing the pixels specifically from solid three dimensional geometries, to be rendered onto the screen. The frame buffer may alternate with another buffer, this is called double buffering to allow complete rendering of a frame whilst the other is being processed. Crucially to this blog, certain stages in the GPU pipeline are now configurable, such as stage 9, whereas some are programmable. These are the vertex and pixel (or fragment) shading programs, and since DirectX 10, geometry shaders. These microprograms do not have access to memory directly,   but apart from the memory supplied within texture maps they may access GPU registers (making certain programming paradigms such as if/else statements inefficient). In VFX, shaders are very important, however, manipulation of the frame buffer and overlaying the rendered scene, such as user interface graphics and particle effects. These are collectively named post-effects.