XNA 4.0 Light Pre-Pass: Alpha masking

One important feature of any renderer implementation is the alpha-masking support. Vegetation, chains and wired fences would be a nightmare to model and also a candidate to be a triangle-hungry-monster. The idea of alpha-masking is to decide if a given pixel should be rendered or not using the alpha channel of a texture (we just need one channel, so we can store it on diffuse’s alpha). If the value is bigger then a threshold, we draw it, otherwise it is skipped.

With the introduction of full shader-based pipelines, even this basic behaviour should be implemented on pixel shader. On HLSL we must use the clip(value) function, that discards the pixel if value is negative. Note that the computation for that pixel is still performed, it is just not sent to the render target (or backbuffer). To effectively skip the processing, we could to use dynamic branching and the [branch] notation for Xbox (I will not enter in details here).

In the pixel shader, all we need to do is  clip(diffuse.a – alphaReference), so our values lesser than alphaReference will evaluate a negative result, skipping the result. We can play with the alphaReference value in run-time, to make objects appear/disappear (useful for spawning effects).

Now the cool section: how to integrate it into my light pre pass pipeline (if you don’t know what I’m talking about, take a look at my old posts).  As usual, you can get the full source code here. Use it at your own risk!!

First, we need to do the alpha masking in 3 different stages:

  • when rendering to the G-Buffer;
  • when reconstructing the light;
  • when drawing to a shadow map.

Thinking ahead, we may need to mix alpha masking with fresnel/reflection/skinned meshes/multi-layer materials/etc, so its better to start using a solution to prevent something like “shader_fresnel_alpha_skinned_dual_layer.fx”. I introduce you…uber shaders!!

Uber shader is just a big shader that implements lots of behaviours (fresnel/reflection/etc), and the application decides which path to follow. I will use pre-processor parameters (#define/#ifdef) to construct the shader flow, since it’s a compile-time only process. I must confess I’m not a big fan of uber shaders, since sometimes the code gets messy, tricky to follow and not so human-readable, but for now I’m ok with it.

I’ve added the option to enable/disable alpha-masking and also the alphaReference value on 3DMax®, so we need to find a way to get that information and store into our processed mesh. To accomplish that, we need to make some changes on our Pipeline Processor (it took me a while to have it working properly, so accept this as a good gift :p ):

  • on the model processor (our LightPrePassProcessor.cs), we need to extract the alpha information on the original material, and store a list of “defines” (for now, I’m handling only alpha-masking, but the idea is to gather all kind of information like fresnel/reflection/etc). After that, we put this list into the material’s opaqueData, like “lppMaterial.OpaqueData.Add(“Defines”, listOfDefines);”;
  • we have to extend a material processor: I’ve created a class named LightPrePassMaterialProcessor to handle the “defines” we pushed from the first step, and send it to the effect processor;
  • we need also to extend the EffectProcessor, a job for LightPrePassFXProcessor class. It only reads the “defines” information stored into the context’s parameters and copy it to its “Defines” property.

With these steps working, we can focus on the shader itself. All we need to do is to put the alpha-checking inside “#ifdef ALPHA_MASKING …. #endif” region (ALPHA_MASKING is the key I chose for that, it’s on LightPrePassProcessor.cs). Here is a small snippet, from the technique to render to GBuffer:

//read our diffuse
half4 diffuseMap = tex2D(diffuseMapSampler, input.TexCoord);
clip(diffuseMap.a – AlphaReference);

 Note that as we don’t need the diffuse for the rest of this technique (remember we just output normals/depth on this technique), we can put the texture fetch inside the alpha mask region. We need also to support backface lighting, since almost anything that uses alpha-masking is not a closed mesh. To do that, we need to use the VFACE semantics (available only on SM3+) if we detect that macro, like this:

struct PixelShaderInput
float4 Position   : POSITION0;
float3 TexCoord   : TEXCOORD0;
float Depth    : TEXCOORD1;float3 Normal : TEXCOORD2;
float3 Tangent : TEXCOORD3;
float3 Binormal : TEXCOORD4;#ifdef ALPHA_MASKED
float Face : VFACE;

At this point you’ve probably got the idea of uber shaders (this is just the beginning, though). We need to extend it to the shadow map generation, including texture coordinates to the vertex input/output and performing the clip() inside the pixel shader. Remember also to set the culling to none on the technique declaration.

The trees on this sample were generated by Tree[d], an awesome free tool to generate trees.

I would like to thanks the guys that donated some money. It’s not about the money itself: to get to the point of donating anything, someone has read my blog, downloaded the code, run it, enjoyed it, returned to the blog, and clicked on the button to do a donation. This means that I’m doing a good job in sharing the knowledge, and it motivates me to continue this series of samples.

Thanks guys, see you!



About jcoluna

Game developer and musician
This entry was posted in XNA. Bookmark the permalink.

28 Responses to XNA 4.0 Light Pre-Pass: Alpha masking

  1. This project doesn’t compile for me in the state that you’ve posted it — it produces the error “Unable to find manifest signing certificate in the certificate store.” on build. I haven’t looked into it much, but that’s not an error I’ve ever seen before.

  2. You mentioned a point light shadow implementation at some point, so I was waiting to see if you’d share code related to that.

    Since this post doesn’t include point light shadows, maybe you could help me with a problem I’m having with my implementation of dual parabolic point light shadows.

    Currently the parabolic shadow maps I create seem fine, but reading from the shadow maps does not work yet. I think my problem is that my transformation of a given light point into light space when rendering the light to the light buffer is wrong.

    Here is a description of dual-paraboloid shadow mapping that I’m following:

    Here is my point light shader, which is a slight modification of yours:

    (Look at the “PointLightMeshShadowPS” pixel shader function, where the shadow attenuation is computed.)

    Since you were implementing point light shadows, maybe you already tackled this problem (unless you went the cube map shadow route).

    Do you see any glaring errors in that pixel shader?

    Sorry that I couldn’t donate more, I’m a student though 🙂 I’ll try to make another contribution in a couple of weeks or so.

    If we can solve this issue, I’m happy to share my point light shadow code with you.

  3. Blah, check your comment spam box, I think my previous comment was flagged as spam because I linked to a couple of random sites.

  4. Marco says:


    I would ask you if it’s possible to render translucent geometry with your renderer 😉

    Good work 🙂


  5. Marco says:

    For example water! 😉

  6. Raziel Nomack says:

    I dont Know Why all my Models Exported from 3ds Max 2011 Using Autodesk FBX exporter, are exported with huge scale params. A Cube 1x1x1 in Max results in a mensh in that scene 10x greater than entire sponza.

    Any recomendations?

    • Either scale your scene on export from 3Ds max, or enter a small scale value in the model properties in XNA (i.e. right click the model in your content project and click properties…).

  7. In SpotLightMeshShadowsPS in LightingLPP.fx, after this line:

    “float3 pos = float3(TanAspect*(screenPos*2 – 1)*depthValue, -depthValue);”

    Is “pos” the world space position of the pixel?

  8. They work as I had them by the way. The problem was that I was forgetting to set the world transform when creating the shadow map, so everything was offset in a weird way. Feel free to use those shaders, linked in the gamedev.net post 🙂

  9. Marco says:

    Hi 🙂

    I tried to add 60 trees to my scene and my framerate drops to 20 fps… I’ve a HD4650 with 1 GB ram videocard.

    Is there any way to optimize trees’ drawing?

    Thank you 🙂

    • jcoluna says:

      Probably a couple:
      – Check if the shaders I’ve created use the fastest way to do alpha-masking (I’m using only clip(), but maybe using “if”/alphablend/stencil or another trick could make it faster)
      – Change the mesh, that one is made of a lot of small triangles.
      – Cache the effect parameters, to avoid fetching from the effect’s dictionary (Im using effect.parameters[“World”], you can store that in a variable)
      – Implement instanced rendering
      – Sort meshes front to back, group by material, reduce state changes

      If you do any of those, let me know (and maybe post the code so I can update the sample =)
      See ya!

  10. Do you know what’s causing light’s edges to darken other lights? E.g. when two point lights overlap, you can see a dark area along the overlap. It’s similar to what you might expect to see when alphablending bright sprites on top of one another.

    • jcoluna says:

      Hey buddy, I don’t know what is that. I haven’t seen this issue on my PC, can you post a screenshot? Maybe PIX could help here.


  11. Feedidy says:

    Is it possible to use more than 1-bit alpha?
    I’m not 100% sure how your “ubershader” works.

    • Tom Looman says:

      Non-clipped alpha, aka translucency is an inherit problem in Deferred shading…Solutions include Inferred rendering and drawing your translucent surfaces in a separate forward rendering-pass, but Deferred rendering itself does not support translucency on your geometry.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s