Performance problem on Firefox if video is placed in a room

[takahirox]

Description

FPS number in a room is significantly worse on Firefox if video is placed in a room.

To Reproduce

Access this "room with video"

https://hubs.mozilla.com/rc7gbkQ/crazy-honored-nation

And also access this "room without video"

https://hubs.mozilla.com/PsvaJ6i/clear-authentic-space

And compare the fps number displayed at the bottom right.

In my Desktop Windows with NVIDIA GeForce GTX 1080

Browser	With video	Without video
Firefox Nightly	30fps	60fps
Chrome	60fps	60fps

Expected behavior
The fps number in a "room with video" on Firefox should be as good as the one on Chrome.

Hardware

• Device: Desktop
• OS: Windows
• Browser: Firefox

Additional context
Add any other context about the problem here.

┆Issue is synchronized with this Jira Task

[takahirox]

Jim's case: Windows, Nvidia RTX 2080

Browser	With video	Without video
Firefox	90fps	135fps
Chrome	144fps	144fps

Dom's case: Linux, NVidia RTX 2080

Browser	With video	Without video
Firefox	39-45fps	42-47fps
Chrome	120fps	120fps

(In Dom's case, I guess the performance problem on Firefox seems from generic (JS engine?) issue rather than video stuffs.)

[takahirox]

I took the profile on Firefox.

The remarkable thing is .texImage2D() cpu time is huger with video. I guess Firefox has a problem uploading video to GPU?

The result of another Firefox profile which can profile the lower level. It also shows that uploading video seems to waste long time.

[takahirox]

Reported to bugzilla https://bugzilla.mozilla.org/show_bug.cgi?id=1680851

[netpro2k]

The upstream VideoTexture in Three also makes this gap even worse since they now use requestVideoFrameCallback to reduce the work to only upload when there are actually new video frames, but only Chrome supports this currently https://github.com/mrdoob/three.js/blob/dev/src/textures/VideoTexture.js#L22.

I do think we should re-introduce a FPS cap on video frame uploads.. We had one at some point but it has been removed, and I can't really remember why. Hard limiting to something like 30 FPS should at least help mitigate this problem a bit.

Ultimately this problem eventually goes away with https://www.khronos.org/registry/webgl/extensions/proposals/WEBGL_video_texture/ since this will prevent the need for the GPU -> CPU -> GPU copy of video frames going on now with the current pipeline, but that's probably a long ways off from being usable.

[mattrossman]

360 videos are especially bad, they bring Firefox down to ~5 FPS which makes that feature unusable for me.

[takahirox]

From mrdoob/three.js#21746

There might be a chance that video play performance on Firefox can be improved by setting the video texture format RGBA.

[takahirox]

Wow, RGBA format works really well on my end. 15 fps became 60fps.

11.2MB mp4 video + desktop Windows.

	RGB	RGBA
Firefox	15fps	60fps
Chrome	60fps	60fps

[mattrossman]

Holy cow, same result here on the big 360 video I was having trouble with. Went from 5 FPS to just under 60 FPS. Great find @takahirox!

[takahirox]

The root issue may have been resolved on Firefox end.

https://bugzilla.mozilla.org/show_bug.cgi?id=1680851

We should test again and remove the workaround if we confirm it's fixed.

[takahirox]

I tested on Windows Firefox Nightly. Unfortunately the problem hasn't been resolved yet.

[netpro2k]

We may want to aim to make most things RGBA anyway even if this issue does get fixed in Firefox, as many platforms end up emulating RGB (not specific to videos): https://developer.mozilla.org/en-US/docs/Web/API/WebGL_API/WebGL_best_practices#some_formats_e.g._rgb_may_be_emulated

[takahirox]

Upstream Three.js seemed to have removed RGBFormat. The workaround can be removed when upgrading our Three.js anyways.

https://github.com/mrdoob/three.js/wiki/Migration-Guide#136--137

[johnshaughnessy]

Will remove the workaround in b89f1e7