Custom color passthrough

[tretyp]

Hi, recently i bought HTC Vive Elite hmd, mostly for passthrough features and it turned out a way more complicated than i expected.

I had tried ALXR and it works great, except... it uses 'black' color for passthrough as default, which is killing half of fun of it.
I did some research and found custom [old] builds with 'green' or 'blue' colors instead, but only for Pico and Quest headsets. None of these worked on HTC XR.

Then i bought VD, since it's 'super easy' and everyone reccomend it. Yeah, it doesn't support passthrough for Vive headsets at all.

So, is it possible to request here 'an custom' build with green color passthrough? I tested a few latest nightly builds and 'android' versions [non quest or pico specific] works just fine.

I know there is a way to compile it myself, but i never had any interest in any android apps at all, so it would take days for me to even install all the correct prerequisites. So i thought maybe someone, or korejan himself, could help here.

I found this code for custom color people were using in theirs quest\pico custom builds:

alvr/openxr-client/alxr-engine-sys/cpp/ALVR-OpenXR-Engine/src/alxr_engine/vulkan_shaders/passthroughMask_frag.glsl.

#version 460
#ifdef ENABLE_ARB_INCLUDE_EXT
#extension GL_ARB_shading_language_include : require
#else
// required by glslangValidator
#extension GL_GOOGLE_include_directive : require
#endif
#pragma fragment
#include "common/baseVideoFrag.glsl"
#include "common/color-functions.glsl" //added from Rachmanin0xF github
vec3 RGB_TO_LAB(vec3 rgb) {
return XYZ_TO_LAB(RGB_TO_XYZ(rgb));
}
float when_lt(float x, float y) {
return max(sign(y - x), 0.0);
}
layout(location = 0) out vec4 FragColor;
const vec3 KeyColor_LAB = vec3(21.0486,-5.2067,21.8668); //Modify as you wish in LAB format this code is between brown/green/yellow
void main()
{
vec4 sampleRGB = SampleVideoTexture();
vec3 sampleLAB = RGB_TO_LAB(sampleRGB.rgb);
float deltaE = LAB_DELTA_E_CIE2000(sampleLAB, KeyColor_LAB);
sampleRGB.a -= when_lt(deltaE, 10.0); //tolerance, try 10 for instance
FragColor = sampleRGB;
}

[korejan]

@tretyp Hi, I'm currently working on client-side UI support and re-worked the mask mode shader that you can configure and updates in real-time, it's not publicly available yet but you can try this build. it is a work-in-progress and the settings are not yet persistent.

In this build, green is the default chroma-key, the controls to toggle passthrough modes is the same, to show/hide the UI double click the menu button.

Trigger button for "mouse clicks", grip & hold to move/translate/scale UI, if you move the cursor to the top you can see some placement modes.

In the UI you will see two alpha controls, one controls the alpha of the keyed out area and the other controls stream area, if you invert those two values you will have an inverted mask for example.

There are multiple chroma-key modes in a drop-down list, this list is ordered by colour compare accuracy vs performance, the first entry and default selected (CBCR_BT2020) is the fastest mode and works well for green chroma-keys but not good for arbitrary colour keys, the LAB_CIExxxx entries are much better for arbitrary colour keys but are more expensive to run on the GPU, LAB_CIE2020 is the most accurate and most expensive.

I have optimized these shaders a lot but even with the fastest mode I do not recommend using them with the video resolution set to 140% (== 1.4 x native res), especially on XR2 gen1 based devices. Using CBCR_BT2020 / BT709 mode is just about ok at 140% but you will notice the client frame-rate starting to dip a little, LAB_CIE2020 mode at 140% will kill the client performance. So it is best that you change the video res to 100-125%.

The threshold slider, when changing the chroma-key modes each one will need different thresholds for the same chroma key, a good default will be set when changing between the modes.

Lastly this APK can be used for Quest and HTC headsets

[tretyp]

Thanks for the reply, unfrortunatelly provided apk is just crashing instantly on my Elite XR.
I just quickly tested latest nightly build [v0.38.0+nightly.2024.05.27 ] and it does crash too.

So i started trying every build since v0.36.0+nightly.2024.05.18 [the last one with separated quest and 'generic' android apks], and it worked until v0.38.0+nightly.2024.05.25.
Starting from this build alxr is crashing at app launch.

[korejan]

Thanks, I'll look into it but it will be a bit difficult to figure out as I don't have a F3/xre.

If you could send me a logcat log it would help

[elbadcode]

Hi, recently i bought HTC Vive Elite hmd, mostly for passthrough features and it turned out a way more complicated than i expected.

I had tried ALXR and it works great, except... it uses 'black' color for passthrough as default, which is killing half of fun of it. I did some research and found custom [old] builds with 'green' or 'blue' colors instead, but only for Pico and Quest headsets. None of these worked on HTC XR.

Then i bought VD, since it's 'super easy' and everyone reccomend it. Yeah, it doesn't support passthrough for Vive headsets at all.

So, is it possible to request here 'an custom' build with green color passthrough? I tested a few latest nightly builds and 'android' versions [non quest or pico specific] works just fine.

I know there is a way to compile it myself, but i never had any interest in any android apps at all, so it would take days for me to even install all the correct prerequisites. So i thought maybe someone, or korejan himself, could help here.

I found this code for custom color people were using in theirs quest\pico custom builds:

alvr/openxr-client/alxr-engine-sys/cpp/ALVR-OpenXR-Engine/src/alxr_engine/vulkan_shaders/passthroughMask_frag.glsl.

#version 460
#ifdef ENABLE_ARB_INCLUDE_EXT
#extension GL_ARB_shading_language_include : require
#else
// required by glslangValidator
#extension GL_GOOGLE_include_directive : require
#endif
#pragma fragment
#include "common/baseVideoFrag.glsl"
#include "common/color-functions.glsl" //added from Rachmanin0xF github
vec3 RGB_TO_LAB(vec3 rgb) {
return XYZ_TO_LAB(RGB_TO_XYZ(rgb));
}
float when_lt(float x, float y) {
return max(sign(y - x), 0.0);
}
layout(location = 0) out vec4 FragColor;
const vec3 KeyColor_LAB = vec3(21.0486,-5.2067,21.8668); //Modify as you wish in LAB format this code is between brown/green/yellow
void main()
{
vec4 sampleRGB = SampleVideoTexture();
vec3 sampleLAB = RGB_TO_LAB(sampleRGB.rgb);
float deltaE = LAB_DELTA_E_CIE2000(sampleLAB, KeyColor_LAB);
sampleRGB.a -= when_lt(deltaE, 10.0); //tolerance, try 10 for instance
FragColor = sampleRGB;
}

lol that original shader you sent is undoubtedly based on mine which I have no problem with since I distanced myself from the community it was produced for and made it free license anyway, but it is kind of funny to see they took mine and just made it worse since you get all or nothing while mine had a secondary smoothing factor like virtual desktop. My repo is way out of date but if you do try compiling from public branch you might try my shaders.

The new changes sound interesting too though I'm not going to bother using that build until I can build my own. I'm guessing this is all on a local/private branch still? I'll definitely revisit it check out your new shaders once that's available

[korejan]

@tretyp i've uploaded another version to the same gdrive link, can you try that

[tretyp]

@tretyp i've uploaded another version to the same gdrive link, can you try that

Yay! This one works perfectly!
Thank you, much appreciated