- Vertex Specification
- Vertex Processing
- Vertex Post-Processing
- Primitive Assembly
- Fragment Shader
- Per-Sample Processing
The Stencil Test is a per-sample operation performed after the Fragment Shader. The fragment's stencil value is tested against the value in the current stencil buffer; if the test fails, the fragment is culled.
In order to use the stencil test, the current Framebuffer must have a stencil buffer. The stencil buffer is an image that uses a stencil image format. The Default Framebuffer may have a stencil buffer, and user-defined framebuffers can attach stencil formatted images (either depth/stencil or stencil-only) to the GL_STENCIL_ATTACHMENT attachment point.
If the current framebuffer has no stencil buffer, then the stencil test will always behave as if it is disabled.
If there is a stencil buffer, that buffer has a certain bitdepth. This specifies the number of stencil bits available.
Fragment stencil value
Each Fragment has a stencil value, which is an unsigned integer. The stencil test operation will test this stencil value against the value from the current framebuffer at the fragment's position.
This value is usually defined by the same function that sets the stencil test, below. This means that every fragment from every primitive for every object in a draw call gets the same fragment stencil value (with the front/backface point noted below).
To enable stencil testing, call glEnable with GL_STENCIL_TEST. When the stencil test is enabled, the testing logic of the reference stencil value against the framebuffer stencil value is skipped. The stencil operations are still processed when the test is disabled; the operation proceeds as if the stencil test had passed.
When rendering to a framebuffer that has no a stencil buffer, stenciling will always behave as if it is disabled.
Stenciling operations take into account the fact that triangles have two sides. Therefore, all stencil tests and operation functions have two sets of data: one for the front side of triangles and one for the back. Which side is used depends on whether the fragments generated from the primitive came from the front or back face.
These functions all take a face parameter, which specifies which facing state is set by that function. The face can be GL_FRONT or GL_BACK, but it can also set both sides at once with GL_FRONT_AND_BACK.
For Primitives that have no facing, the front side stencil state are always used.
The stencil test itself is set by this function:
The ref defines the fragment's stencil value for all fragments generated for the given facing. The fragment stencil value will be clamped to the range defined by the stencil buffer's bitdepth.
The first step of the stencil test is to get the destination stencil value from the stencil buffer (called Ds). This is an unsigned integer value. The fragment's stencil value will be called Fs.
The next step is to perform a bitwise AND with both Fs and Ds, against the mask parameter for both. This allows the user to mask off certain stencil bits, reserving them for different conditional tests. This results in two masked unsigned integers, Fm and Dm.
Then the stencil test itself is performed between Fm and Dm, based on the func parameter. The test is of the form (Fm FUNC Dm); the masked fragment value is on the left-hand side. The available functions are:
|GL_NEVER||Always fails.||GL_ALWAYS||Always passes|
glStencilFunc can be used to set both face operations in one call, if you don't like to use GL_FRONT_AND_BACK.
The stencil operation uses the result of the stencil test. If stencil testing is disabled, then the stencil test is counted as passing; the stencil operation detailed below still happens even if stencil testing is disabled.
If the stencil test fails, the fragment is discarded. Normally, discarding a fragment means that it has no visible effects. However, the stencil buffer can be updated even from discarded fragments, if the depth or stencil tests discard them.
There are three possible cases involved here:
- The stencil test fails. The fragment will be discarded. Remember that the stencil test happens before the depth test, so the result of that is never computed if the stencil test fails.
- The stencil test passes, but the Depth Test fails. The fragment will be discarded (due to the depth failure). This case only applies if the depth test is enabled.
- The stencil test passes, and the Depth Test passes. If the depth test is disabled, then it is always assumed to pass.
To define how the stencil value in the framebuffer is modified in each case, use the following function:
The parameters sfail, dpfail, and dppass define the stencil update operations to perform in the three cases above, respectively. Each case can use any of the following operations ("current value" here means the value already in the stencil buffer):
|GL_KEEP||Don't modify the current value (default)||GL_INVERT||Invert the current value|
|GL_ZERO||Set it to zero||GL_REPLACE||Replace with the masked fragment value|
|GL_INCR||Increment the current value, saturating1 if it would overflow||GL_INCR_WRAP||Increment the current value, wrapping if it would overflow|
|GL_DECR||Decrement the current value, setting to zero if it would underflow||GL_DECR_WRAP||Decrement the current value, wrapping if it would underflow|
1: Meaning that it stops at the maximum representable integer at the stencil buffer's bitdepth. For an 8-bit stencil buffer, that would be 255.
glStencilOp can be used to set both face operations in one call, if you don't like to use GL_FRONT_AND_BACK.