How to render an object to the stencil buffer

[FlightBlaze]

Hi! How can I render an object to the stencil buffer? Do I need to change some parameters in the pipeline state to start writing to the stencil?

[TheMostDiligent]

You need to enable stencil and set the necessary stencil properties for the front and back faces.

[FlightBlaze]

Here is my code for the stencil object PSO:

PSOCreateInfo.GraphicsPipeline.DepthStencilDesc.StencilEnable = Diligent::True;

Diligent::StencilOpDesc StencilDesc;
StencilDesc.StencilFunc = Diligent::COMPARISON_FUNC_ALWAYS;
StencilDesc.StencilPassOp = Diligent::STENCIL_OP_REPLACE;

PSOCreateInfo.GraphicsPipeline.DepthStencilDesc.FrontFace = StencilDesc;
PSOCreateInfo.GraphicsPipeline.DepthStencilDesc.BackFace = StencilDesc;

And for the other objects PSO:

PSOCreateInfo.GraphicsPipeline.DepthStencilDesc.StencilEnable = Diligent::True;
Diligent::StencilOpDesc StencilDesc;
StencilDesc.StencilFunc = Diligent::COMPARISON_FUNC_NOT_EQUAL;
PSOCreateInfo.GraphicsPipeline.DepthStencilDesc.FrontFace = StencilDesc;
PSOCreateInfo.GraphicsPipeline.DepthStencilDesc.BackFace = StencilDesc;

Other objects should discard their pixels at the stencil area, but they don't. Where I am wrong?

[TheMostDiligent]

There is also a depth test. You may need to disable it

[FlightBlaze]

I draw 2D graphics, so I disabled it

[TheMostDiligent]

Which stencil reference value do you write? You need to set it with SetStencilRef

[FlightBlaze]

I tried DeviceContext.SetStencilRef(0xFF) and 0x0 but it doesn't work. I don't even understand what stencil reference value is

[TheMostDiligent]

You can find the information on stencil operation on the Web. That one seems to be reasonable. Stencil works similarly in all graphics APIs.

[FlightBlaze]

I don't know if it's a bug or not, but I have done a lot of trial and error, but it doesn't work. Finally, I tried to hide the object completely by setting stencil comparison function to NEVER but it still doesn't work.

PSOCreateInfo.GraphicsPipeline.DepthStencilDesc.StencilEnable = Diligent::True;
Diligent::StencilOpDesc StencilDesc;
StencilDesc.StencilFunc = Diligent::COMPARISON_FUNC_NEVER;
PSOCreateInfo.GraphicsPipeline.DepthStencilDesc.FrontFace = StencilDesc;
PSOCreateInfo.GraphicsPipeline.DepthStencilDesc.BackFace = StencilDesc;

And before drawing the mesh I specified the stencil reference value:

deviceCtx->SetStencilRef(1);

Tested on Mac (MoltenVk) and Windows (Direct3D11)

If you want, I can share with you my demo project in which the stencil doesn't work, and help to compile it as well. What if it really is a bug?

[TheMostDiligent]

I don't know if it's a bug or not, but I have done a lot of trial and error, but it doesn't work.

What are you trying to do with stencil?

Stencil testing is used in atmospheric scattering effect in particular and everything works properly.

[FlightBlaze]

I am creating a 2D graphics library for Diligent. Look at the semi-transparent corners of the triangle (I marked one with a red circle). There is an overlap of two lines connected at one point. I want to remove this overlap by writing pixels of the first line to the stencil buffer to prevent the same pixels from being overwritten by the second line.

[TheMostDiligent]

First of all, you will need to use a depth buffer with stencil (e.g. D32F_S8 or D24U_S8).
Enable stencil in the depth-stencil state.
Clear the stencil value to 0 and set stencil ref to 0.
Set stencil function to equal and stencil op to increment. This way every primitive will increment the stencil counter and only one primitive will be rendered as stencil will be incremented.

On the other hand, you don't really need stencil buffer at all to make sure primitives do not overlap. Simply use regular depth buffer - use depth function LESS and write some fixed depth from your shader (e.g. 0.5 or 0.0). This way every pixel will only be written by a single primitive

[FlightBlaze]

I'm using the depth buffer. It works, hooray! Thanks!