[Perf] Linux/x64: 6 Regressions on 8/28/2024 6:34:28 PM

[performanceautofiler]

Run Information

Name	Value
Architecture	x64
OS	ubuntu 22.04
Queue	TigerUbuntu
Baseline	7ff684af49cdf008bebbe10efcc10a2649795d65
Compare	063ae05c63307becef004eefbc8704ccb03c7f04
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.IO.Compression.Brotli

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
Compress_WithoutState - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	406.57 ms	473.62 ms	1.16	0.01	False
Compress_WithState - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	405.38 ms	468.93 ms	1.16	0.01	False
Compress_WithState - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	183.27 ms	207.36 ms	1.13	0.02	False
Compress_WithoutState - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	182.72 ms	209.69 ms	1.15	0.01	False
Compress_WithoutState - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	43.38 ms	49.93 ms	1.15	0.02	False
Compress_WithState - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	43.54 ms	49.48 ms	1.14	0.02	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

• Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
• CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
• Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
• Dotnet SDK installed for dotnet commands
• Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Compression.Brotli*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.IO.Compression.Brotli*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Compression.Brotli*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.IO.Compression.Brotli*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

System.IO.Compression.Brotli.Compress_WithoutState(level: Optimal, file: "TestDocument.pdf")

ETL Files

Histogram

JIT Disasms

System.IO.Compression.Brotli.Compress_WithState(level: Optimal, file: "TestDocument.pdf")

ETL Files

Histogram

JIT Disasms

System.IO.Compression.Brotli.Compress_WithState(level: Optimal, file: "alice29.txt")

ETL Files

Histogram

JIT Disasms

System.IO.Compression.Brotli.Compress_WithoutState(level: Optimal, file: "alice29.txt")

ETL Files

Histogram

JIT Disasms

System.IO.Compression.Brotli.Compress_WithoutState(level: Optimal, file: "sum")

ETL Files

Histogram

JIT Disasms

System.IO.Compression.Brotli.Compress_WithState(level: Optimal, file: "sum")

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[jkurdek]

I think it was dotnet/runtime#106994. fyi: @jkoritzinsky

[jkoritzinsky]

cc: @carlossanlop looks like upgrading Brotli may have introduced some regressions. What do you think is the best course of action here?

[carlossanlop]

Are those regressions expected according to the brotli update release notes or the owners?

The recent zlib-ng perf regressions were accepted because they were clearly described by the zlib-ng owners as expected.

[jkoritzinsky]

The release notes for brotli are pretty bare bones. They didn’t describe any known regressions.

https://github.com/google/brotli/releases/tag/v1.1.0

Here’s a link to more regressions as well as some improvements from the upgrade: dotnet/runtime#107302 (comment)

[carlossanlop]

You could ask the brotli owners if they're aware of this perf regression in their tests. If they can justify it, then we could probably take the update if no one objects. I did that with the zlib-ng owners and they provided detailed explanations to justify some of the perf regressions.

Also, it does not look like this brotli update contains any security fixes, but I am unsure if they publish them in the release tag notes. If we are absolutely sure there were no security fixes, should we just skip this update until we get significant enough improvements that would justify the perf regression?

Any thoughts on my suggestions, @jkotas?

[jkotas]

Do we understand what caused the improvements in May 2024 and July 2024? It sounds like that this benchmark may be sensitive to unrelated factors.

should we just skip this update until we get significant enough improvements that would justify the perf regression?

I do not think we can afford the stay behind.

15% regression is not prohibitive. If you include the earlier improvements, it is ~7% for release-to-release comparison.

[jkotas]

RunKind:micro_mono

Do we see these regressions on CoreCLR as well?

[jkurdek]

Do we understand what caused the improvements in May 2024 and July 2024? It sounds like that this benchmark may be sensitive to unrelated factors.

The only commit in range for May is : dotnet/runtime#101630
The July issue should be this: #38789

[jkoritzinsky]

It looks like we see some of the regressions on CoreCLR as well, as recorded in #107302 and the linked regressions.

There's also some improvements for some of the benchmarks linked there. Really weird grouping of improvements vs regressions though at a high level glance.

[jkotas]

The only commit in range for May is : dotnet/runtime#101630

We may want to include System.IO.Compression.Native in the PGO training set (with the matching training scenario) to help C/C++ compilers to generate better code.

[ericstj]

I agree with @jkotas that we cannot afford to stay behind. I also agree with @carlossanlop that we should notify the maintainers of brotli about the performance differences we see between 1.0.9 and 1.1.0. @jkoritzinsky can you summarize the differences we see in a way that might be informative or actionable to the brotli owners?

[carlossanlop]

@jkoritzinsky can you summarize the differences we see in a way that might be informative or actionable to the brotli owners?

@ericstj I opened an issue in google/brotli with our results to try to understand the regressions: google/brotli#1202

[carlossanlop]

We may want to include System.IO.Compression.Native in the PGO training set (with the matching training scenario) to help C/C++ compilers to generate better code.

@jkotas I can open an issue to track your request, but in which repo is this done? runtime or performance?

[carlossanlop]

@jkotas I opened dotnet/runtime#108073 but if the correct repo was dotnet/performance, please feel free to move it there.

Let's close this as we're taking the brotli update regardless, and the follow ups have been addressed (opening an issue for google/brotli to continue the regression conversation, and open the PGO issue).

[carlossanlop]

I don't have permission to close issues in this repo. @jkurdek can you please help?