Please help me understand the concept of how stencil buffering works in Vulkan.

Unfortunately, the example(s) I have seen go too deep into the weeds, obscuring what I’m trying to understand. Not sure what is really needed or not.

Here I’ll detail what I have come to understand which may or may not be correct.

  1. I want to render a graphic that will punch a hole into the depth/stencil image so scissors is not what I want to do
  2. I have the depth part working so I’m good there.
  3. I created a separate pipeline for just depth buffer use and one for depth/stencil use.
  4. I’m assuming that if I render with the depth/stencil pipeline, it will punch a hole into depth/stencil image.
  5. Now that there’s a hole in the depth/stencil image, further renders within the same frame buffer will only be visible through the hole.

This is what I have come to understand but I’m unable to get it working. Thanks.

What does it mean to “punch a hole into the depth/stencil image”? Also, the way stencil works in Vulkan is functionally no different than how it works in any other API.

What I mean by “punch a hole into the depth/stencil image” is with a pipeline that has stencilTestEnable set to true, if you render with an image that is black with white spots, the white areas will allow other rendering to pass.

I just had a thought, does the image I want to render to the stencil buffer need it’s aspect mask set to VK_IMAGE_ASPECT_STENCIL_BIT? Right now the image I’m using as a stencil mask is visible in the render even though stencilTestEnable = true in that pipeline. Not sure what I’m doing wrong.

Been looking this demo over and over and I don’t see what I’m missing.
https://github.com/SaschaWillems/Vulkan/blob/master/examples/stencilbuffer/stencilbuffer.cpp

if you render with an image that is black with white spots, the white areas will allow other rendering to pass.

Two things. First, I don’t understand what you mean by “allow other rendering to pass”. Do you want previous rendered stuff to be visible through these white areas? Do you want it to affect subsequently rendered objects?

Second, the stencil buffer doesn’t care about colors. And since the fragment shader cannot manipulate the stencil buffer, you can’t do a thing where the FS detects “white” and causes something stencil-related to happen.

The stencil test is about [i]math[/i], operations done based on the stencil value for the fragment (constant for all fragments in the rendering command) and the stencil value in the corresponding sample in the stencil buffer. And some depth stuff too.

the image I want to render to the stencil buffer

You cannot render “colors” to the stencil. And, as previously stated, the fragment shader cannot affect the fragment’s stencil value. The stencil buffer’s value can only be affected by the stencil reference value you set when you rendered those objects. It’s a single, unchanging value over the course of the rendering command.

Do you want previous rendered stuff to be visible through these white areas? Do you want it to affect subsequently rendered objects?

Yes. In OpenGL, I would disable the depth and color buffers, enable the stencil buffer, this would now render to the stencil buffer, re-enable the color and depth buffers and what I rendered would only show through what I did to the stencil buffers. I know it’s all math but I’m a visual thinker and communicator. For my needs, I only used the stencil buffer for simple masking.

The stencil buffer’s value can only be affected by the stencil reference value you set when you rendered those objects.

Can you tell me where in the example I sited it’s doing that? Do you have any example code?

Then do that in Vulkan. It has all of the same switches as OpenGL does. Vulkan has write masks, just like OpenGL. Vulkan has the same stencil reference state, as well as stencil function state.

By “disable/enable the color and depth buffers”, I assume you mean turning off write masks. Because changing the FBO itself is highly unnecessary for such a task.

You said “if you render with an image that is black with white spots, the white areas will allow other rendering to pass.”. “White” and “black” are colors; ergo, you want to use colors to determine the value of the stencil. That’s not allowed.

So either you haven’t explained very well what you’re trying to do, or what you’re trying to do isn’t possible, in either OpenGL or Vulkan.

You said “if you render with an image that is black with white spots, the white areas will allow other rendering to pass.”

Here’s some of my OpenGL code using a stencil buffer to mask the items in a scroll box. Since a stencil is an 8 bit buffer, you can think of it as a grey scale image or perhaps something similar to an alpha channel.


    // Disable color and depth buffers
    glColorMask( GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE );
    glDepthMask( GL_FALSE );

    // Start using the stencil
    glEnable( GL_STENCIL_TEST );

    glStencilFunc( GL_ALWAYS, 0x1, 0x1 );
    glStencilOp( GL_REPLACE, GL_REPLACE, GL_REPLACE );

    // Render to the stencil
    m_upStencilMaskSprite->render( matrix );

    // Re-enable color and depth
    glColorMask( GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE );
    glDepthMask( GL_TRUE );

    // Where a 1 was not rendered
    glStencilFunc( GL_EQUAL, 0x1, 0x1 );

    // Keep the pixel
    glStencilOp( GL_KEEP, GL_KEEP, GL_KEEP );

    for( int i = m_visStartPos; i < m_visEndPos; ++i )
        m_pScrollControlVec[i]->render( matrix );

    // Finished using stencil
    glDisable( GL_STENCIL_TEST );

Vulkan has the same stencil reference state, as well as stencil function state.

I assume you mean the VkStencilOpState. I understand that but the problem is I assume I need to write something to the stencil buffer. I just don’t know how that is done and I don’t see that happening in the example I sited.

Working with the Vulkan stencil buffer is very different then what I did with OpenGL or DirectX. I need a simple, stencil buffer example because I’m just not getting it.

No, it isn’t. Vulkan stencil operations work in exactly the same way as it does in OpenGL or D3D. If you can understand what that OpenGL code you posted is doing, then you understand exactly how it works in Vulkan too. The only difference is where you put the stuff. All you need to do is map those things to stuff in Vulkan.

You say “I need to write something to the stencil buffer.” Is that not exactly what “m_upStencilMaskSprite->render( matrix );” does? The state setting before that rendering call in OpenGL becomes state that is part of the pipeline object in Vulkan. The state set by the glStencil* and glEnable(GL_STENCIL_TEST) are in VkStencilOpState. The depth write mask is found in VkPipelineDepthStencilStateCreateInfo. And the color write masks are found in the array of VkPipelineColorBlendAttachmentState in VkPipelineColorBlendStateCreateInfo.

There is pretty much a 1:1 mapping between OpenGL function calls and Vulkan pipeline structure state when it comes to stencil stuff. Yes, you have to have a separate pipeline object for the mask render compared to the render using the stencil mask. But that’s it.

I see now. Thanks to you, I now have my stencil buffer working! Awesome, you’re the best!

I’ve always had trouble wrapping my head around the stencil test functions (compareOp, failOp, depthFailOp) and the stencil buffer in general.