IncludeInfrared crashes the app on HoloLensCaptureApp

[petergu684]

Thanks for this great repo for sensor data collection. I am not sure it is a bug but when I set the IncludeInfrared flag to true, HoloLens app would crash when trying to setup the pipeline. Below is the flags I changed in HoloLensCaptureApp.cs:

        private static readonly bool IncludeDepth = true;
        private static readonly bool IncludeDepthCalibrationMap = true;
        private static readonly bool IncludeAhat = false;
        private static readonly bool IncludeAhatCalibrationMap = false;
        private static readonly bool IncludeInfrared = true;
        private static readonly bool EncodeInfrared = true;

I suppose the infrared image here means the active brightness image from the IR camera. Could you suggest the proper way to get the active brightness stream? Plus, I notice the include infrared flag is only used in long throw mode. In Ahat mode, OutputInfraredImage is set to false. Is there any reason writing it in such way? Thanks!

My HoloLens is on 21H1, 20348.1432.

[sandrist]

I think I see the bug, and will PR a quick fix shortly. Just to be sure though, can you see what exception gets thrown exactly (and from where) when trying to run the app with this configuration?

Yes, Infrared refers to the AB image. Setting IncludeInfrared = true should be all you need to do (once this bug fix is checked in).

I'll check with the team to see if there was a good reason for not outputting the Infrared image from the depth camera in AHAT mode, and report back. I know we've had issues with the AHAT streams in general, which seem to be a bit finnicky depending on which other sensor streams are being accessed.

(Also, I would recommend leaving IncludedDepthCalibrationMap to false unless you have a good reason to emit that stream, i.e., you are re-computing calibration intrinsics for the depth camera yourself. It's a fairly heavy stream, and probably only needed in extreme special cases for debugging purposes)

[petergu684]

Thanks for the advice!

As for the crash, it happens when the label shows "Capturing..." for less a second and then immediately switched to the initial state, then crashes in a few seconds. I switched to the debug mode and changed the debug type to managed only, a null reference exception was raised at the Dispose function of RemoteClockExporter.cs file (line 53):

this.listener.Stop();

Adding a question marker after listener could stop the crash but still won't capturing. (The app just returns to initial state)
Below is more log outputs:

...
Attempting to connect to 169.254.48.108:16000
Connected to 169.254.48.108:16000.
'HoloLensCaptureApp.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\data\Programs\WindowsApps\Microsoft.NET.CoreFramework.Debug.2.2_2.2.29301.2_arm__8wekyb3d8bbwe\System.Runtime.Serialization.Formatters.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'HoloLensCaptureApp.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'Anonymously Hosted DynamicMethods Assembly'. 
'HoloLensCaptureApp.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\data\Programs\WindowsApps\Microsoft.NET.CoreFramework.Debug.2.2_2.2.29301.2_arm__8wekyb3d8bbwe\System.ObjectModel.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'HoloLensCaptureApp.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\data\Programs\WindowsApps\Microsoft.NET.CoreFramework.Debug.2.2_2.2.29301.2_arm__8wekyb3d8bbwe\System.Threading.Tasks.Parallel.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
Exception thrown: 'System.Exception' in Microsoft.Psi.MixedReality.UniversalWindows.dll
Exception thrown: 'System.AggregateException' in System.Private.CoreLib.dll
Pipeline Error: One or more errors occurred. (The buffer allocated is insufficient. (Exception from HRESULT: 0x88982F8C))
The thread 0x6dc has exited with code 0 (0x0).
The thread 0xfd0 has exited with code 0 (0x0).
The thread 0x1678 has exited with code 0 (0x0).
Exception thrown: 'System.Net.Sockets.SocketException' in System.Net.Sockets.dll
The thread 0x1dd0 has exited with code 0 (0x0).
The thread 0x142c has exited with code 0 (0x0).
Exception thrown: 'System.IO.IOException' in System.Net.Sockets.dll
Exception thrown: 'System.IO.IOException' in System.Net.Sockets.dll
Exception thrown: 'System.Exception' in Microsoft.Psi.MixedReality.UniversalWindows.dll
RemoteClockExporter Exception: A blocking operation was interrupted by a call to WSACancelBlockingCall
RendezvousServer error: RendezvousServer disconnected.
The thread 0x2338 has exited with code 0 (0x0).
The thread 0x2184 has exited with code 0 (0x0).
The thread 0x9ac has exited with code 0 (0x0).
The thread 0x1a3c has exited with code 0 (0x0).
Exception thrown: 'System.AggregateException' in System.Private.CoreLib.dll
Pipeline Error: One or more errors occurred. (The buffer allocated is insufficient. (Exception from HRESULT: 0x88982F8C))
Exception thrown: 'System.NullReferenceException' in Microsoft.Psi.dll
Object reference not set to an instance of an object.

The program '[7644] HoloLensCaptureApp.exe' has exited with code -1 (0xffffffff).

Regarding the AHAT camera, it could enable some interesting application because of its higher framerate and spatial resolution although being quite noisy. I also have trouble enabling both AHAT and PV in the latest build 22H1 as is mentioned in this issue.

[sandrist]

Thanks for the additional info. PR #242 should hopefully fix the bug in encoding infrared images, while also extending configuration options to allow for AHAT infrared images.

Unfortunately, however, we are also seeing the same issues with enabling AHAT in recent OS builds. It also seems to no longer be possible to enable AHAT and long-throw depth streams at the same time. We're not sure if/when the AHAT issues will be fixed, as it seems to be a problem deep in the OS/firmware, but will post on that thread if we (or anyone else) can discover a workaround...

[petergu684]

Sorry, I just got the time to test it. It works and I can see the Infrared stream in PSI Studio! Thanks for the update. There are some minor related issues:

• [PSI Studio] Similar to the depth image, could there also be the range fields (range min, range max, range mode) in properties tab of PSI Studio? Now I can see the image updating but it is mostly black. It will be cool if we can set the lookup range for easier visualization. I suppose it won't be too different from the depth image considering they are both 16-bit images.
• [HoloLensCaptureExporter] The exporter does not export both ahat and long throw infrared stream. Long throw infrared is missing because it has a wrong name here. InfraredImageCameraView should be DepthInfraredImageCameraView, same for the encoded one. Ahat infrared is missing because the code is not added. Copy/paste the long throw infrared code can fix the problem but I think it is worth mentioning here. These are minor changes but I can submit a PR if needed.
• Infrared stream does not always have the same number of frames as its depth stream. In my last test, 2 out of 2217 infrared frames is missing in ahat mode and 1 out of 208 depth frame is missing in long throw mode (i.e. 2215 infrared vs 2217 depth for ahat; 208 infrared vs 207 depth for long throw). It is not a big issue since we have timestamp file that can help align frames, but I am wondering whether that is normal considering these two streams are from the same sensor.

[sandrist]

Glad that it works, and thanks for the PR bringing the exporter in line with the recent changes!

For your first question, only the DepthImage visualizer has properties for things like min/max range. Those properties wouldn't make sense for other kinds of RGB images in general, but you're right that they could be useful for Infrared/brightness images. We might consider eventually changing the infrared streams to be represented as DepthImages rather than Images, or perhaps creating a more generic type for "data images", but that will be a big rippling change for sometime later in our roadmap. In the meantime, you could try creating a little script/app that converts the Infrared images to DepthImage, using the CopyFrom method (which only works if the image to copy from is Gray 16 bit). Feel free to follow up here if you go down that path and get stuck or need help.

For your last question, that sounds normal. By default, the app is using "LatestMessage" delivery policies on most streams (except audio), which means messages might drop if a component can't keep up. For example, if there is any hiccup in the network and the TCP writer gets blocked, some incoming messages will be dropped. You can always change delivery policies to be "Unlimited" if you don't want messages to ever drop, but then you run the risk of ever-increasing message queues and latencies. So it's always a tradeoff.

[petergu684]

Thanks for your reply. I will try it when I have time!