Introduction

Shadows are a critical component of 3D rendering, adding depth and realism to a scene. In this tutorial, we’ll explore how to create basic shadows in OpenGL using the stencil buffer. The stencil buffer allows us to define where a shadow is rendered by restricting its placement to a specific plane.

This method works best for simple shadows that remain on a flat surface, such as shadows cast onto a floor or table.

What is the Stencil Buffer?

The stencil buffer is an additional buffer in OpenGL used for masking certain parts of the scene. It allows you to:

• Select specific areas to render shadows.
• Clip unwanted parts of the shadow outside the defined plane.
• Combine shadows with transparency for realistic effects.

By leveraging the stencil buffer, we can define a “drawing plane” where shadows are rendered, while ensuring other parts of the scene remain unaffected.

Setting Up the Stencil Buffer

To use the stencil buffer, we need to modify the window initialization code:

glutInitDisplayMode(GLUT_DOUBLE | GLUT_RGBA | GLUT_STENCIL);

This adds a stencil buffer to the window.

Next, we clear the stencil buffer at the beginning of the rendering process:

glClearStencil(0);
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);

This ensures the stencil buffer is reset before rendering each frame.

Defining the Shadow Plane

We define the plane where shadows will be rendered using a combination of masking and stencil functions:

glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE); // Disable color rendering
glDepthMask(GL_FALSE);                              // Disable depth updates
glEnable(GL_STENCIL_TEST);                          // Enable stencil testing
glStencilFunc(GL_ALWAYS, 1, 0xFFFFFFFF);            // Set stencil test to always pass
glStencilOp(GL_REPLACE, GL_REPLACE, GL_REPLACE);    // Replace stencil buffer data

The next step is to draw the plane. For this example, we use a custom `bench()` function to define the plane’s geometry:

bench();

After defining the plane, re-enable color and depth masking:

glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE); // Enable color rendering
glDepthMask(GL_TRUE);                           // Enable depth updates
glStencilFunc(GL_EQUAL, 1, 0xFFFFFFFF);         // Only draw where stencil value equals 1
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);         // Keep existing stencil buffer data

Rendering the Shadow

To render the shadow, we use the following steps:

1. Disable texturing and depth testing to ensure the shadow appears as a plain black shape.
2. Flip the shadow vertically to simulate it being cast onto the plane.
3. Translate and rotate the shadow to match the object’s position.

Here’s the code:

glDisable(GL_TEXTURE_2D);
glDisable(GL_DEPTH_TEST);

glPushMatrix();
glScalef(1.0f, -1.0f, 1.0f); // Flip vertically
glTranslatef(0, 2, 0);       // Move shadow to plane
glRotatef(angle, 0, 1, 0);   // Rotate shadow
glColor4f(0, 0, 0, 1);       // Set shadow color to black
square();                    // Draw shadow shape
glPopMatrix();

glEnable(GL_DEPTH_TEST);
glEnable(GL_TEXTURE_2D);
glDisable(GL_STENCIL_TEST); // Disable stencil testing

Rendering the Scene

After rendering the shadow, we draw the rest of the scene as usual. To enhance the realism of the shadow, use alpha blending:

glEnable(GL_BLEND);
glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);
bench(); // Draw the bench with transparency
glDisable(GL_BLEND);

Tutorial Code

Here’s the full implementation of the basic shadows tutorial:

#include <GL/gl.h>
#include <GL/glut.h>

float angle = 0; // Rotation angle
GLuint texture[2]; // Texture IDs

// Load textures (implement as needed)
GLuint loadtextures(const char *filename, float width, float height);

// Draw a square
void square(void) {
    glPushMatrix();
    glBegin(GL_QUADS);
    glVertex3f(-1, -1, 0);
    glVertex3f(-1, 1, 0);
    glVertex3f(1, 1, 0);
    glVertex3f(1, -1, 0);
    glEnd();
    glPopMatrix();
}

// Draw the bench (shadow plane)
void bench(void) {
    glPushMatrix();
    glColor4f(1, 1, 1, 0.7); // Semi-transparent bench
    glBegin(GL_QUADS);
    glVertex3f(-2, -0.5, 1);
    glVertex3f(-2, 0.5, -1);
    glVertex3f(2, 0.5, -1);
    glVertex3f(2, -0.5, 1);
    glEnd();
    glPopMatrix();
}

// Display callback
void display(void) {
    glClearStencil(0);
    glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);
    glLoadIdentity();
    glTranslatef(0, 0, -10);

    // Stencil buffer setup
    glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
    glDepthMask(GL_FALSE);
    glEnable(GL_STENCIL_TEST);
    glStencilFunc(GL_ALWAYS, 1, 0xFFFFFFFF);
    glStencilOp(GL_REPLACE, GL_REPLACE, GL_REPLACE);
    bench(); // Define shadow plane
    glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
    glDepthMask(GL_TRUE);
    glStencilFunc(GL_EQUAL, 1, 0xFFFFFFFF);
    glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);

    // Draw shadow
    glDisable(GL_TEXTURE_2D);
    glDisable(GL_DEPTH_TEST);
    glPushMatrix();
    glScalef(1.0f, -1.0f, 1.0f);
    glTranslatef(0, 2, 0);
    glRotatef(angle, 0, 1, 0);
    glColor4f(0, 0, 0, 1);
    square();
    glPopMatrix();
    glEnable(GL_DEPTH_TEST);
    glEnable(GL_TEXTURE_2D);
    glDisable(GL_STENCIL_TEST);

    // Draw the scene
    glEnable(GL_BLEND);
    glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);
    bench();
    glDisable(GL_BLEND);
    glRotatef(angle, 0, 1, 0);
    square();
    glutSwapBuffers();
    angle++;
}

// Main function
int main(int argc, char **argv) {
    glutInit(&argc, argv);
    glutInitDisplayMode(GLUT_DOUBLE | GLUT_RGBA | GLUT_STENCIL);
    glutInitWindowSize(500, 500);
    glutCreateWindow("Basic Shadows");
    glutDisplayFunc(display);
    glutIdleFunc(display);
    glutMainLoop();
    return 0;
}

If you have any questions or run into issues, feel free to email me at swiftless@gmail.com. Happy coding!