Unhandled exception when executing IBasicVideoEffect methods in WinUI 3

[kaesardev]

Describe the bug

Crash during the execution of custom effects implementations (from the IBasicVideoEffect or IBasicAudioEffect interfaces) in a WinUI 3 application. Some properties get executed, such as IsReadOnly and SupportedMemoryTypes, but fail and no stack trace is provided.

System.MissingMethodException: 'Method not found: 'System.Collections.Concurrent.ConcurrentDictionary`2<System.RuntimeTypeHandle,WinRT.IObjectReference> WinRT.IWinRTObject.get_QueryInterfaceCache()'.'

Note: The same implementation of custom effects works correctly in Console Application.

To Reproduce

1. Create a Class Library (WinUI 3 in Desktop)
2. Install packages: Microsoft.Graphics.Win2D, Microsoft.Windows.CsWinRT
3. Add to .csproj:

<PropertyGroup>
	<AllowUnsafeBlocks>true</AllowUnsafeBlocks>
	<CsWinRTComponent>true</CsWinRTComponent>
	<CsWinRTWindowsMetadata>10.0.19041.0</CsWinRTWindowsMetadata>
</PropertyGroup>

5. Implement BlurVideoEffect.cs and FadeVideoEffect.cs as in the tutorial.
6. Implement EchoAudioEffect.cs as in the tutorial.
7. Create a Blank App, Packaged (WinUI 3 in Desktop)
8. Add reference to library.
9. Add to Package.appxmanifest:

<file name="WinRT.Host.dll" xmlns:winrt="urn:schemas-microsoft-com:winrt.v1">
	<winrt:activatableClass name="WinRTLibrary.BlurVideoEffect" threadingModel="both" />
	<winrt:activatableClass name="WinRTLibrary.FadeVideoEffect" threadingModel="both" />
	<winrt:activatableClass name="WinRTLibrary.EchoAudioEffect" threadingModel="both" />
</file>

11. Execute the following method when launch the application or clicking a button

private static async Task RenderMediaCompositionWithEffects()
{
    var composition = new MediaComposition();

    // Add clip
    var clip = MediaClip.CreateFromColor(Color.FromArgb(255, 0, 0, 0), TimeSpan.FromSeconds(5));
    composition.Clips.Add(clip);

    // Add effect
    var blurVideoEffectDefinition = new VideoEffectDefinition(typeof(BlurVideoEffect).FullName);
    clip.VideoEffectDefinitions.Add(blurVideoEffectDefinition);

    var fadeVideoEffectDefinition = new VideoEffectDefinition(typeof(FadeVideoEffect).FullName);
    clip.VideoEffectDefinitions.Add(fadeVideoEffectDefinition);

    var echoAudioEffectDefinition = new AudioEffectDefinition(typeof(EchoAudioEffect).FullName);
    clip.AudioEffectDefinitions.Add(echoAudioEffectDefinition);

    // Render
    var folder = Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location);
    var path = Path.Join(folder, "output_video.mp4");
    if (!File.Exists(path))
    {
        File.Create(path).Close();
    }

    var outputFile = await StorageFile.GetFileFromPathAsync(path);
    var profile = MediaEncodingProfile.CreateMp4(VideoEncodingQuality.HD720p);
    await composition.RenderToFileAsync(outputFile, MediaTrimmingPreference.Precise, profile);
}

Expected behavior

Export media composition with custom effects.

Version Info

.NET 6
Microsoft.Graphics.Win2D 1.0.4
Microsoft.Windows.CsWinRT 2.0.1
Microsoft.WindowsAppSDK 1.2.221116.1
Microsoft.Windows.SDK.BuildTools 10.0.22621.755

Additional context

Source code: https://github.com/kaesardev/CustomEffectsWinRT

[vvetma]

Are there any news regarding this issue or any known workarounds?

[brabebhin]

Any news on this?

[manodasanW]

I tried out the above repro provided under additional context, but it seems to run fine and generates a mp4 file. Are folks still able to repro it?

I know in the past we had some issues with MissingMethodException from mismatched projections and so on, so I wonder if you try it out again with the latest CsWinRT and have the latest .NET SDK installed, if you still see the issue. If you do hit it, are there any errors related to source generator in the build output.

[brabebhin]

Let's put it the other way around, can you provide a sample of it working?

[lhak]

I tested it with the lastest available cswinrt and .net versions and still observe the same error message as in the bug description. I did not see any errors/warnings in the build output.

[lukasf]

Some good news on this: I have updated the sample to latest WindowsAppSdk, Win2d and CsWinRT (remember to update the packages both in the lib and in the app). And the sample works: I put breakpoints in the process methods and they get called as expected! I also tested the audio effect and it works as well.

So I think this issue is resolved. I tested both packaged and unpackaged deployment.

[lhak]

I tried it again in my application and still see the reported error, even when updating to the newest packages. So this issue is apparently still not fixed.

[lukasf]

I tried it again in my application and still see the reported error, even when updating to the newest packages. So this issue is apparently still not fixed.

Just because it does not currently work in your app does not mean that the issue is not solved. If you still see the issue despite having latest CsWinRT, AppSdk, BuildTools and Win2d, then you should compare the project files and manifests, to find out what is different in your app, compared to the sample. It could also be a problem with outdated files in your obj/bin folder. Try deleting all intermediate files from all projects and rebuild from scratch.

Here is the sample with updated packages. Check it out, put a breakpoint in BlurVideoEffect.ProcessFrame, press the button in the sample and see that it works indeed, without any errors:
https://github.com/lukasf/CustomEffectsWinRT

[lhak]

@lukasf: Thanks for the updated project. I tested it and it works correctly. So I compared the project files to those of my application and found an issue: The .csproj file references a publish profile (this is the default in the appsdk templates). Unfortunately it seems that these files are not checked into git by default (I have run into this issue multiple times) and are also missing in the sample project.

This causes the SelfContained parameter (self-contained deployment for .net) to be set to false instead of true. I can confirm that using framework-dependent deployment for .net works, but self-contained does not work. So it seems that a fix for the latter configuration is still required. This is important because if you want to submit an application to the microsoft store, this deployment method is required. I guess this is also the reason why it is enabled by default in the templates.

[lukasf]

@lhak That's really interesting. Indeed if I add a publish file and put it as self contained, I get the MissingMethodException!

@manodasanW I think you should really look into this. Generally, the bug seems to be fixed. But when a "self contained" publish profile is used, the MissingMethodException is still there. So it looks like using self contained deployment somehow breaks CsWinRT code generation, at least for this case. Or the generated code/binaries are somehow not correctly deployed to the output folder anymore. Something must be different.

Check out my sample here:
https://github.com/lukasf/CustomEffectsWinRT

Please select Debug x64, because I only added a profile for that. On master branch, everything works fine. Button press results in file being generated, if you put a breakpoint in BlurVideoEffect, it is hit. Now check out the "self-contained" branch, which adds the publish profile. Run again, press the button, boom, MissingMethodException! If you edit the publish profile and put SelfContained as false, the exception is gone.

This looks like a bug in CsWinRT, which does not work correctly with self contained deployments (which is probably the main way to deploy .net core apps, since for whatever reason, Microsoft does not seem interested at all in adding automatic .net core installation in msix). I do not see any suspicious messages in the build output. No clue what is going on.

[manodasanW]

Thanks for the additional insights, I will try to repro this again.

[whiskhub]

See #1141 (comment)

I found out the error has something to do with the AssemblyLoadContext in the WinRT.Host.Shim. https://github.com/microsoft/CsWinRT/blob/master/src/Authoring/WinRT.Host.Shim/Module.cs#L76-L106:

Normally, the WinRT.Runtime assembly should be loaded for each AssemblyLoadContext in the protected override Assembly Load(AssemblyName assemblyName), while for all other assemblies null is returned. This should trigger the AssemblyLoadContext.Default.Resolving event, causing the other assemblies to be loaded there in the main ALC. But this event is never raised in the self-contained scenario. So no assembly except the WinRT runtime and System.Private.CoreLib assembly are actually loaded, causing the missing methods.

Replacing the entire construct with a simple Assembly.LoadFrom() fixes the error. But there is a reason the ALC construct was added after all...

I don't know what the reason is and how to fix it yet, though.

[nltmm]

@manodasanW there is another bug with later versions of CsWinRT. The sample from @lukasf https://github.com/lukasf/CustomEffectsWinRT works perfectly with Microsoft.Windows.CsWinRT 2.0.1. However, after updating to Microsoft.Windows.CsWinRT 2.1.x or 2.2.x, the video effect no longer works. This happens even when all other packages Microsoft.WindowsAppSDK, Microsoft.Windows.SDK.BuildTools are updated to the latest version.

Could you please look into this?