Results 1 to 3 of 3

Thread: Understanding stencil buffer and masking

  1. #1
    Member Contributor
    Join Date
    Mar 2017
    Posts
    53

    Understanding stencil buffer and masking

    In the Stencil Testing tutorial on Learn OpenGL, I'm having trouble understanding how the use of glStencilFunc() and glStencilMask() can be used draw only around the borders of an object (like a highlighting effect).

    From what I know about stencil buffers so far, a stencil buffer with only 0s could look like this on a 4x4 pixel screen, with the given drawing code:

    Code :
    glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);
    glStencilFunc(GL_ALWAYS, 1, 0xFF);
    glStencilMask(0xFF); //0xFF would be new stencil value on stencil test success
     
    glDisable(GL_DEPTH_TEST); //would also mean depth test always passes 
    glDrawArrays(<center 4 pixels>);

    0000 0000 0000 0000
    0000 0xFF 0xFF 0000
    0000 0xFF 0xFF 0000
    0000 0000 0000 0000

    I'd be interested in drawing a different color object on the pixels with only a "0000" value in the stencil buffer. Based on the tutorial, I could just call glDrawArrays() to draw across all the pixels, and the stencil buffer code below should discard the center 4 pixel fragments.

    Code :
    glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);
    glStencilFunc(GL_NOTEQUAL, 1, 0xFF);
    glStencilMask(0x00); //on stencil success or fail, no stencil values will be changed
     
    glDisable(GL_DEPTH_TEST);
    glDrawArrays(<draw to the entire 4x4 pixel screen>);

    Doing glStencilFunc for a stencil value of 0xFF in the stencil buffer doesn't seem to discard the fragment though when I think of the math behind it (though the code output in the tutorial shows discarding). From the docs to glStencilFunc, the stencil test would be like this with the arguments above:

    func=GL_NOTEQUAL, ref=1, mask=0xFF, stencil=0xFF (since the stencil value is 0xFF)
    (ref & mask) != (stencil & mask) ---> (1 & 0xFF) != (0xFF & 0xFF) ---> 1 != 0xFF ---> stencil test passes

    I feel like this would make sense and the fragments would be drawn where the stencil value is 0xFF, but the tutorial output shows pixels being drawn like only where the stencil buffer values are 0 I think (hence the highlighting effect). Is there something wrong with my logic with the stencil buffer, maybe in the first or second drawing code? I feel like I summarized the code right from the tutorial.

  2. #2
    Junior Member Newbie
    Join Date
    Feb 2015
    Posts
    15
    glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);

    In this, GL_REPLACE(Stencil pass & Depth pass) will replace old stencil value to Reference value.

    glStencilFunc(GL_ALWAYS, 1, 0xFF);
    Ref Value: 1
    Mask: 0xFF

    so after rendering your first fragment your stencil plane will become:

    0x00 0x00 0x00 0x00
    0x00 0x01 0x01 0x00
    0x00 0x01 0x01 0x00
    0x00 0x00 0x00 0x00

    Then Second fragment calculation:
    (ref & mask) != (stencil & mask) ---> (1 & 0xFF) != (1 & 0xFF) ---> 1 != 1 ---> stencil test fails.

  3. #3
    Member Contributor
    Join Date
    Mar 2017
    Posts
    53
    That makes sense. Looks like I got the first drawing code wrong. The docs in glStencilOp say GL_REPLACE means replace the stencil value with ref, like you said. Thanks!

Similar Threads

  1. Replies: 1
    Last Post: 09-11-2013, 11:35 AM
  2. copy app generated stencil data to stencil buffer
    By guilder in forum OpenGL: Basic Coding
    Replies: 7
    Last Post: 12-23-2010, 06:46 PM
  3. Masking depth buffer
    By V-man in forum OpenGL: Advanced Coding
    Replies: 3
    Last Post: 01-15-2006, 07:21 AM
  4. Stencil Buffer Not Masking
    By Swiftless in forum OpenGL: Basic Coding
    Replies: 2
    Last Post: 09-16-2005, 10:35 PM
  5. How to clear stencil buffer after stencil test?
    By in forum OpenGL: Basic Coding
    Replies: 5
    Last Post: 06-14-2003, 09:04 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Proudly hosted by Digital Ocean