3DS Max/Vray/Mental Ray: Ambient Occlusion

Before checking out this post you might want to check out a few other popular 3D lighting techniques!

What is Ambient Occlusion?

On top of the methods mentioned above (which are only a few techniques), there are further ways to supplement and provide depth to lighting within an image. A frequent way of adding natural shadow falloff and density into areas of an image is ambient occlusion. Unlike Final Gather and Photon lighting, that are concerned with bounced light and multi-bounce transparency, it focuses on detailing the absence of light in order to develop a successful shading method. Created in 2001 by artists at ILM for the film Pearl Harbour after being challenged by Michael Bay, it was a natural evolution of the reflective occlusion shaders created for the proceeding film, Speed 2, “Ambient occlusion was a way of trying to address global illumination and ray tracing but with far less expense. The key to this technique is to make a single ray-traced occlusion pass that is generated independently of the final lighting and final rendering.”. The theory behind ambient occlusion is that there is an ever present light source in the scene (Irrespective of our lighting system). The amount of shading is distinguished by how much of the geometry is hidden, or blocked, “What happens internally is that the area above the point to be shaded is sampled for blocking geometry. If any is found, the percentage of blockage translates directly to an occlusion factor.” (Van der Steen 2007: 14). It therefore calculates based on their visibility how much shading the pixels should get.

Why use it?

Historically in computer graphics, Ambient Occlusion was created to solve the problem of unnatural looking indirect illumination. Specifically, its ambition was to develop a better model to ‘weight’ shadows and contact areas between geometry and complicated shaded sections of a scene. Ambient renders at the time provided environmental light, but it was very one dimensional and did not take into account the necessary falloff that occurred when light bounced around and under geometry.

But this was over ten years ago? Surely you can generate these shadows using methods now?

Yes! It is possible to achieve a very good lighting setup straight out of commercial renderers, that provide multiple methods to achieve very accurate contact and weighted shadowing. Often this occurs within a material, or directly in the renderer and ways to do both are shown both for VRay and Mental Ray later in this post.

So why still use it if this is possible?

Working in 3D graphics is never as simple as clicking a button to create a beautiful image. People might put forward that facade, especially those that do not use the software on a daily basis to justify a certain approach, but in reality the software packages, shaders and renderers are tools that are only as successful as the people that use them. Project deadlines, problems in a model, or even just a specific look, or challenge requires knowledge of a wide variety of techniques within the software and at many points during a commission constraints may arise that require the use of more than just a ‘brute force’ approach. Consequently, there are multiple reasons why you might want to make use of an Ambient Occlusion shader and an example could be the need to reduce render time during an animation. For instance, Final Gather is an approximated method of lighting, this means that creating a map with a lower point density spread over a larger ‘smoothed’ catchment area will be a very quick (And dirty) way of computing a lighting pass for Mental Ray, but it will also often lead to areas of the scene losing definition. An ambient occlusion pass can therefore be created to supplement this quicker solution and bring back contrast into the image. This allows for a much faster turn around than using a high density Final Gather pass and it puts a lot more of the responsibility for the final look in the hands of the artist. In many areas of the graphics world, having this control is absolutely vital to achieving a look, saving time or providing a professional service (Or all of these).

AO For Mental Ray

Create a separate render pass

In Mental Ray there are two native ways of bringing this shader into effect. The first, and arguably the most flexible way to do it is to place the Ambient/Reflective Occlusion material into the surface shader option of a MR material and then drag this into the material override slot, which can be found in the processing tab of the render dialog (F9). Final Gather, exposure and Global Illumination should all be turned off to allow the shader to create its independent ambient calculation. For visual feedback, this is a good way to achieve this result, although it is also possible to split the Occlusion out into a render element alongside the beauty render.

AO OptionsThe Ambient/Reflective Occlusion Shader in Mental Ray

The options for the shader include providing increased samples to smooth out the solution, specifying the colour of the bright and dark areas of the image, the sharpness of the spread and the max distance that the engine looks to sample occluded points. You can also specify object ID, reflective occlusion and the strength of the falloff. A few examples of Ambient Occlusion passes can be seen below.


LookngSouthAOExamples of ambient occlusion passes

The Arch and Design Shader

A second way of adding Ambient Occlusion is to use the Arch and Design shader, specifically created to work with Mental Ray. If you are not experienced with the historical evolution of the internal shaders, it is advised that you stick to using this material because it is has been built to manage the correct transfer of energy between the different shading parameters within Mental Ray. For example, if you were to increase the reflective nature of an A & D material, it would work out a physically possible amount to take from the diffuse to achieve this result, whilst other shaders may allow the creation of effects that are not constrained by mathematical principles. Therefore, unless you are very knowledgeable about the software (Or aiming for a specific look), it is best to stick with this shader when working in Mental Ray. To enable AO within this shader, you need to go down to the special effects dialog shown below.

AmbientOcclusionArch and Design Special Effects rollout

For Vray


In Vray, there is a very similar shader to the Ambient/Reflective Occlusion found in Mental Ray, but with slightly more features. To activate VRayDirt as an AO shader, it should be plugged into a VRayLightMtl and then in exactly the same way as shown earlier, it is possible to either achieve a scene wide ‘pass’ in the ‘override material’ section of the ‘VRay Global Switches’ window, or as an element using the ‘VRayExtraTex’. Unlike the Arch and Design materials that provide AO as an extra shading option, VRay also has a separate tab for the application of Ambient Occlusion into the native render, found in the Indirect Illumination roll out (Below).

Vray Renderer Rollout

So, why is it called VRayDirt?

It is becoming increasingly common for people to use occlusion driven shaders not just as an extra indirect light supplement, but also as a mask to mimic the effect of wear, or the accumulation of dust within the crevices of objects. VrayDirt is often primarily used for this purpose and it has therefore become a very powerful tool that allows artists to slowly deconstruct complicated materials and build them from the ground up procedurally. This use is actually quite ironic, as the standard way of applying Ambient Occlusion in post production (Multiplying it over an existing beauty render), is actually not the envisioned workflow for it and often it inadvertently creates a mix between supplementing an image and creating a ‘dirty’ look. Examples of this and how to correctly apply it are discussed below.

The ‘purist’ way to composite Ambient Occlusion?

The reason that multiplying AO onto a ‘beauty’ render is not the purist way to implement it, is due to its original purpose. When it was created by ILM (mentioned above), it was as a shader to supplement the environmental lighting. Therefore, it is not in its nature to deal with direct light, or reflection and it shouldn’t be influencing those areas of the image. Often, when people simply multiply an ambient occlusion pass in Photoshop, or After Effects, they are doing so onto elements of a shot that it was not intended to support. This can lead to it clashing with directly illuminated parts of an image and washing out, or simply having an undesired ‘dirty’ look.

AOComparing the common method of compositing AO with the ‘correct’ way

What can be seen in the image above is that when multiplied over the entire beauty pass, the result is overbearing. This often leads to people reducing the opacity of the pass to compensate for how contrasted it makes the image look, instead of simply reordering where it should be included in the compositing process. The ‘correct’ method to apply AO is to multiply it directly onto the diffuse/ambient and under everything else so that it is isolated and does not overrule the reflection, or the direct lighting. When looking at the GIF above it is clear to see the difference between the approaches in the ladder opening. In the image labelled ‘Multiply Ontop’ it has been applied as a final step, whilst in ‘Multiply Under’ it has been layered on-top of the diffuse element only. It is important to remember that when compositing it should be done in a linear fashion so that the underlying lighting adds up correctly. A common side effect of not doing this forces people to use the ‘screen’ method instead of ‘add’, which can be specifically noticed with very bright highlights (hotspots). To achieve a look similar to what you see in your render window, you will also need to apply a gamma correction curve to suit your final output and below this has been set in the adjustment layer to 2.2.

CompExample of the composite workflow


Van der Steen 2007 Rendering with Mental Ray

Leave a Reply