Incompatibility issue of AHAT sensor and PV camera

[JuyiZhang]

Hello,

I am right now having some trouble on getting the AHAT sensor and PV camera working at the same time. I have tried standalone AHAT sensor and long throw sensor + PV camera and it works properly. However, when I use AHAT sensor + PV camera the application reported an error while initiating.

The code I modified

std::vector<ResearchModeSensorType> AppMain::kEnabledRMStreamTypes = { ResearchModeSensorType::DEPTH_AHAT }; std::vector<StreamTypes> AppMain::kEnabledStreamTypes = { StreamTypes::PV };

The console output

I have also tried to use this repository https://github.com/petergu684/HoloLens2-ResearchMode-Unity and added code of media capture from the microsoft website https://docs.microsoft.com/en-us/windows/mixed-reality/develop/unity/locatable-camera-in-unity and it also fails in short throw mode (which utilizes AHAT sensor). However, when I shift to long-throw mode, it worked properly. Therefore, I am suspecting a compatibility issue between the AHAT sensor and the PV camera. However, I am not sure if it is a hardware problem or can be repaired by software update. Do anyone know how to work around the situation?

Thank you so much for developing the app and resolving my issue

Best,

[petergu684]

Same here, but I can confirm it used to work a few weeks ago. Now the behavior is: 1) if PV camera is on before starting the AHAT sensor, the AHAT sensor will fail to OpenStream; 2) if the AHAT camera is on first, it will work properly until I turn on PV camera and then AHAT sensor's GetNextBuffer method will start to return null pointer.
HoloLens OS Build: 20348.1501

[JuyiZhang]

Same here, but I can confirm it used to work a few weeks ago. Now the behavior is: 1) if PV camera is on before starting the AHAT sensor, the AHAT sensor will fail to OpenStream; 2) if the AHAT camera is on first, it will work properly until I turn on PV camera and then AHAT sensor's GetNextBuffer method will start to return null pointer. HoloLens OS Build: 20348.1501

Yes, I was using 20348.1515 and the problem appears. I have downgraded the hololens to 20348.1432 and the app works again. (However, I guess this is just a temporary solution since staying at one version forever does not sound like a great idea)

[petergu684]

Good to know. I guess it has something to do with

Fixed an issue where graphics memory is leaked during some camera usage scenarios

in the 22H1 update.

[JuyiZhang]

Good to know. I guess it has something to do with

Fixed an issue where graphics memory is leaked during some camera usage scenarios

in the 22H1 update.

Thank you so much for the info! I have tried to use PV camera with your unity plugin as well and in 20348.1432 it worked like a charm :) Let's hope microsoft can resolve this problem soon

[kasumman]

@JuyiZhang @petergu684
Can you modify the VideoFrameProcessor to keep MediaCapture and MediaFrameReader as member variables and see if that helps?

[petergu684]

Hi @kasumman, MediaCapture and MediaFrameReader are already the member variable of my class when the problem exists.

The implementation in my cpp file is mostly copy/paste of the VideoFrameProcessor::InitializeAsync method but saving these variables as member variable:

mediaCapture = MediaCapture();
co_await mediaCapture.InitializeAsync(settings);
...
co_await selectedSource.SetFormatAsync(preferredFormat);
m_mediaFrameReader = co_await mediaCapture.CreateFrameReaderAsync(selectedSource);
auto status = co_await m_mediaFrameReader.StartAsync()

In header file, these are declared as member variables of the class:

winrt::Windows::Media::Capture::MediaCapture mediaCapture = nullptr;
winrt::Windows::Media::Capture::Frames::MediaFrameReader m_mediaFrameReader = nullptr;
winrt::event_token m_OnFrameArrivedRegistration;
static void CameraWriteThread(HL2ResearchMode* pProcessor);
std::thread* m_pWriteThread = nullptr;

[petergu684]

Any updates?

[JWRoboticsVision]

@JuyiZhang

Hi, I have same issue with latest OS version 20348.1511, but I cannot find the link for downloading 20348.1432, could you please provide the moethod to download this version package?

[petergu684]

https://aka.ms/hololens2download/10.0.20348.1432
You can also change it to any older version you want.

[joshanderson-kw]

Hi, I believe we've run into the same issue. After accessing the AHAT and PV camera at the same time, the app crashes immediately on OS 20348.1511, but works fine on 20348.1432. Any updates on this?

[petergu684]

Just want to check whether this will be fixed in the next 22H2 update, which I guess should be around the corner? @kasumman

[kasumman]

Just want to check whether this will be fixed in the next 22H2 update, which I guess should be around the corner? @kasumman
According to the info I have it should be fixed in 22H2.

[petergu684]

Hi @kasumman, I just updated my HL2 to 22H2 (20348.1528) released today but it doesn't seem to have fixed the issue.

[Michelvl92]

Any updates or workarounds besides sticking to 20348.1432?

[luffy-yu]

I met the same issue.

[petergu684]

Just get reply from their support team that switching Windows Insider Program to Dev Channel and updating HoloLens to latest version should fix this issue. I tried and it works without any issue so far. I am on 22621.1057 but it is possible that this is fixed in an earlier insider version I don't know. More data points on how it works would be helpful :)