Gamescope causing stuttering issues

[Arbitrate3280]

Is there an existing issue for this?

• I have searched the existing issues

Is your issue described in the documentation?

• I have read the documentation

Is your issue present in the nightly release?

• This issue is present in the nightly release

Describe the Bug

Every game is a micro-stutter mess on a linux host using vaapi, tried it with both x11 and Wayland.

The same games on Windows, with the same host hardware at the same bitrate runs smooth.

I don't know if this means that this issue is with vaapi as a whole, or if the way vaapi enconding is being done within sunshine can be further improved or if AMF is simply better than VAAPI and it is what it is.

Expected Behavior

Given that this is not a hardware neither a network limitation as I initially thought, a smooth stream should be expected if using VAAPI.

Additional Context

The client is an android tv stick.

Host Operating System

Linux

Operating System Version

Fedora 38

Architecture

64 bit

Sunshine commit or version

20.0

Package

Linux - rpm

GPU Type

AMD

GPU Model

5700 xt

GPU Driver/Mesa Version

Mesa 23.1.3

Capture Method (Linux Only)

KMS

Config

global_prep_cmd = [{"do":"kscreen-doctor output.DP-1.mode.1920x1080@60 output.HDMI-A-1.disable","undo":"kscreen-doctor output.DP-1.mode.2560x1080@75 output.HDMI-A-1.enable"}]

Apps

No response

Relevant log output

Logs don't really say much there's no errors there.

[KuleRucket]

Works great for me as long as I don't let the GPU load approach 100%. AMD cards don't seem to separate encoding from normal GPU work as well as nvidia cards do.

[Arbitrate3280]

It's so weird, it happens to all games for me, even light ones that don't come close to using 100% gpu load.

Only when using VAAPI though, AMF in Windows works fine.

[Smoukus]

Talked to numerous people, and they all said that VAAPI is inferior to AMF.

There should be an option to encode via AMF on Linux.
Upvote this request if you feel the same: https://ideas.moonlight-stream.org/posts/221/amf-encoding-on-linux

[Arbitrate3280]

Some more info: Just tried with another client (Moonlight QT) and VAAPI is working fine, the stream was smooth, so its nothing inherently wrong with vaapi HEVC encoding in my system.

It's still weird though, because it should't be an issue with decoding on the main client (Moonlight Android TV) either, because in both the VAAPI and AMF experiments in the main client I was using HEVC, so the decoder was the same for both, and yet the AMF one was smooth and the VAAPI one was not.

Weird...

Trying to think of some other things that might be the cause of the issue related to the linux session, in case its not related to VAAPI: it happens in both Wayland vs X11, native and wine games alike. I can't think of anything else for now...

Edit: I don't think it's the capture method either, both x11 and KMS felt similar.

[Smoukus]

Just tried with another client (Moonlight QT) and VAAPI is working fine, the stream was smooth, so its nothing inherently wrong with vaapi HEVC encoding in my system.

What was the client that you initially tested with where you have problems?
I have problems with Moonlight Android (my client is Nvidia Shield).

[Arbitrate3280]

What was the client that you initially tested with where you have problems? I have problems with Moonlight Android (my client is Nvidia Shield).

https://www.mi.com/global/product/xiaomi-tv-stick-4k/specs

Also Moonlight Android.

[Arbitrate3280]

Smoukus, are you by any chance using gamescope?

I've been doing some more testing thinking of things that could be cause the stuttering and decide to stop launching my games through gamescope.

This improved the micro-stuttering a lot! It's not completely gone, but the difference is amazing. I've noticed that gamescope was introducing stutter for the host, I just never noticed because I was always looking into the client screen.

Now without it there's no stuttering on the host and only some stuttering left on the client, but now the stutter is client-only at least.

Now I'll do some tweaking with bitrate and frame pacing options and see if anything helps, because I think that this time it's indeed something client side, and there's nothing wrong with VAAPI, at least on my machine.

Because of that I'll close this issue.

[Smoukus]

Nope, I don't use gamescope.

If you find perfect settings, please post them here.

[KuleRucket]

Didn't think of suggesting that but I also had problems when using gamescope. Sunshine and gamescope compete for encoding resources.

[Arbitrate3280]

Didn't think of suggesting that but I also had problems when using gamescope. Sunshine and gamescope compete for encoding resources.

This seems like a useful peace of information to have written down somewhere, I'll see if I the devs want this written in the documentation.

[gschintgen]

See also: #2286 and #2333