feat(vaapi): add option to enable strict enforcement of frame size

[cgutman]

Description

This PR introduces a configuration option for AMD GPUs using the VAAPI encoder to strictly enforce frame bitrate limits (rc_buffer_size) to balance the concerns of video quality (#2864) and frame bitrate excursions (#2788). Intel GPUs appear fine with VBR mode, so this option can be ignored for them.

This PR sets the default to the v0.23.1 (and pre-#2821) behavior which is to not enforce a strict RC buffer size. Even though this isn't ideal for packet loss reasons, the immediate symptom of large frames exceeding the FEC maximum in #2788 was resolved separately by #2803 anyway.

Screenshot

Issues Fixed or Closed

Fixes #2864

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Dependency update (updates to dependencies)
• Documentation update (changes to documentation)
• Repository update (changes to repository files, e.g. .github/...)

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have added or updated the in code docstring/documentation-blocks for new or existing methods/components

[codecov]

Codecov Report

Attention: Patch coverage is 8.60215% with 85 lines in your changes missing coverage. Please review.

Project coverage is 11.06%. Comparing base (9662f05) to head (61362c1).
Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
src/platform/linux/vaapi.cpp	0.00%	74 Missing ⚠️
src/video.cpp	37.50%	4 Missing and 6 partials ⚠️
src/config.cpp	50.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3332      +/-   ##
==========================================
- Coverage   11.18%   11.06%   -0.13%     
==========================================
  Files          99       99              
  Lines       17184    17231      +47     
  Branches     8008     8034      +26     
==========================================
- Hits         1922     1906      -16     
+ Misses      12719    12641      -78     
- Partials     2543     2684     +141     

Flag	Coverage Δ
Linux	8.35% <6.59%> (-0.17%)	⬇️
Windows	5.23% <0.00%> (+<0.01%)	⬆️
macOS-13	13.62% <23.52%> (-0.03%)	⬇️
macOS-14	12.58% <18.75%> (-0.06%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/config.h	0.00% <ø> (ø)
src/platform/common.h	34.04% <100.00%> (ø)
src/video.h	54.94% <ø> (+0.59%)	⬆️
src/config.cpp	6.00% <50.00%> (+0.30%)	⬆️
src/video.cpp	29.95% <37.50%> (-0.47%)	⬇️
src/platform/linux/vaapi.cpp	2.35% <0.00%> (-0.93%)	⬇️

... and 21 files with indirect coverage changes

[cgutman]

It turns out that not even all encoders support VBR mode. The same low-power modes that provide great performance also only support CQP rate control on hardware prior to Alder Lake (and intel-media-driver with the HuC firmware). I had to plumb a bunch of functions to allow us to pre-determine which rate control is supported before requesting it, otherwise FFmpeg would treat lack of support for our RC mode as a fatal error.

On the bright side, this allowed me to get rid of the blind probing we did for low_power=1 previously and add support for slices without breaking older hardware.

[cgutman]

I was able to confirm the CQP limitation for the low-power H.264 entrypoint on my Skylake test system. Looking further, VAAPI is actually the only encoder that requires a fallback to CQP rate control. Since we now know ahead of time whether we'll need to use CQP on VAAPI, we can remove the QP fallback logic in the encoder probing code and just set QP if neeeded in the init_codec_options() callback with the other RC stuff.

[sonarqubecloud]

Quality Gate failed

Failed conditions
15 New issues
15 New Code Smells (required ≤ 0)

See analysis details on SonarCloud

Catch issues before they fail your Quality Gate with our IDE extension SonarLint