Fix and refactor FCSBenchmarks #13419

safesparrow · 2022-06-29T23:13:25Z

Major cleanup and fixes of the benchmarks in tests/FCSBenchmarks projects.

Changes in all benchmarks

All benchmarks should now build and work
Use BDN's console API to make the benchmarks more flexible (end-user can tweak the config without having to change code and build)
General cleanup

Changes in historical benchmark (`BenchmarkComparison`)

Changes in the notebook (`runner.ipynb`)

Fix it
Fix the commit-based benchmark which should now work at least with recent versions (it might still fail when building an old codebase)
Replace multiple projects + String.Replace with a single project using MSBuild properties
Refactor/cleanup
Store results in JSON instead of CSV, put artifacts in separate directories (inside .artifacts/, read them at the end

Changes in the BDN benchmark (now renamed to `HistoricalBenchmark.fsproj`)

Refactor
Add it to VisualFSharp.sln
Add it to the InternalsVisibleTo list in FSharp.Compiler.Service

Changes in FSharp.Compiler.Benchmarks

Deduplicate some of the code by reusing code from HistoricalBenchmark
Now depends on HistoricalBenchmark mainly to reuse the SingleFileCompiler. This might be somewhat controversial - HistoricalBenchmark needs to support old versions of FCS to allow doing a comparison, which can be limiting. However, currently this isn't a problem - once it is, we could either duplicate the code again or extract the shared bits to a separate project.

* extract all types into separate files * deduplicate the shared source file * start creating a script for running all benchmarks

* Refactor all the benchmarks a bit

- use JSON reports instead of CSV - put results in separate directories so that they can be accessed after the benchmark - read the results after all benchmarks finish rather than keeping them in memory * Add `BenchmarkComparison/README.md`

* refactor runner.ipynb further * Clear notebook outputs * Rename entrypoint file to `Program.fs` * Finish the smoke test script

src/Compiler/FSharp.Compiler.Service.fsproj

tests/benchmarks/FCSBenchmarks/decentlySizedStandAloneFile.fs

tests/benchmarks/FCSBenchmarks/CompilerServiceBenchmarks/SourceText.fs

tests/benchmarks/FCSBenchmarks/CompilerServiceBenchmarks/Program.fs

tests/benchmarks/FCSBenchmarks/CompilerServiceBenchmarks/Helpers.fs

safesparrow · 2022-06-30T00:10:20Z

tests/benchmarks/FCSBenchmarks/BenchmarkComparison/setup_commit_template.fsproj

@@ -1,27 +0,0 @@
-<Project Sdk="Microsoft.NET.Sdk">


Replaced by a single project file with MSBuild properties

safesparrow · 2022-06-30T00:10:24Z

tests/benchmarks/FCSBenchmarks/BenchmarkComparison/setup_current.fsproj

@@ -1,25 +0,0 @@
-<Project Sdk="Microsoft.NET.Sdk">


Replaced by a single project file with MSBuild properties

safesparrow · 2022-06-30T00:10:27Z

tests/benchmarks/FCSBenchmarks/BenchmarkComparison/setup_version_template.fsproj

@@ -1,25 +0,0 @@
-<Project Sdk="Microsoft.NET.Sdk">


Replaced by a single project file with MSBuild properties

tests/benchmarks/FCSBenchmarks/BenchmarkComparison/run_current.fsproj

safesparrow · 2022-06-30T00:11:31Z

tests/benchmarks/FCSBenchmarks/BenchmarkComparison/run.fsproj

@@ -1,23 +0,0 @@
-<Project Sdk="Microsoft.NET.Sdk">


No need for an extra top-level project now

safesparrow · 2022-06-30T06:49:41Z

@vzarytovskii Would you be able to review? Thanks!

vzarytovskii · 2022-06-30T16:35:26Z

@vzarytovskii Would you be able to review? Thanks!

Thanks for changes, they look good will do more thorough review tomorrow, once at my laptop :)

One thing which comes to mind is to add benchmarks to FSharp.sln too (and maybe FSharp.Compiler.Service.sln, @auduchinok wdyt?). Some people are not using the VisulaFSharp.sln.

auduchinok · 2022-06-30T16:39:29Z

One thing which comes to mind is to add benchmarks to FSharp.sln too (and maybe FSharp.Compiler.Service.sln, @auduchinok wdyt?). Some people are not using the VisulaFSharp.sln`.

Yes, thanks. We only use the FSharp.Compiler.Service solution, so it would be useful.

…t benchmark files to solution folders

safesparrow · 2022-07-07T11:40:35Z

@vzarytovskii Let me know if this is something worth merging and any changes you'd like to see :)

… F# project `HistoricalBenchmark.Runner`, call that logic from `runner.ipynb` * Add a few customisations (mostly directories) * Reuse existing checkouts/builds/benchmark reports * Provide simple `Runner.runAll` method for doing everything * Include sample results with three historical comparisons

safesparrow · 2022-07-13T20:28:28Z

I did a few changes:

moved all the logic from BenchmarkComparison/runner.ipynb into a new F# project HistoricalBenchmark.Runner
added a few customisations
made the benchmarking more robust and easier to run (one line of code in the notebook)
added three sample results

@vzarytovskii @KevinRansom Would it be possible to get the changes reviewed? If this is not a change you are willing to consider at this time that's fine - but I would be keen to know whether there is any point spending more time on this👍

vzarytovskii · 2022-07-15T12:09:25Z

I did a few changes:

moved all the logic from BenchmarkComparison/runner.ipynb into a new F# project HistoricalBenchmark.Runner

added a few customisations

made the benchmarking more robust and easier to run (one line of code in the notebook)

added three sample results

@vzarytovskii Vlad Zarytovskii FTE @KevinRansom Kevin Ransom FTE Would it be possible to get the changes reviewed? If this is not a change you are willing to consider at this time that's fine - but I would be keen to know whether there is any point spending more time on this👍

It looks great, sorry it was taking long time, we were a bit preoccupied with some other stuff :)

safesparrow added 11 commits June 24, 2022 23:49

Fix benchmark source files

020c113

* cleanup benchmarks

f3b5ebd

* extract all types into separate files * deduplicate the shared source file * start creating a script for running all benchmarks

* Fix BenchmarkComparison notebook

cb4984b

* Refactor all the benchmarks a bit

Refactor BenchmarkComparison

d34fc2b

Cleanup benchmarks a bit more

e45c28c

More cleanup (WIP)

a247e82

changes

27ecbc3

More cleanup, rename a few files.

bedc09a

* Refactor runner.ipynb:

e1af655

- use JSON reports instead of CSV - put results in separate directories so that they can be accessed after the benchmark - read the results after all benchmarks finish rather than keeping them in memory * Add `BenchmarkComparison/README.md`

* Small cleanup

ed2872b

* refactor runner.ipynb further * Clear notebook outputs * Rename entrypoint file to `Program.fs` * Finish the smoke test script

Merge remote-tracking branch 'upstream/main' into benchmarks-cleanup

d9d7abb

safesparrow commented Jun 30, 2022

View reviewed changes

safesparrow marked this pull request as ready for review June 30, 2022 01:17

safesparrow added 3 commits June 30, 2022 18:40

Add benchmark projects to FSharp.Compiler.Service.sln, add non-projec…

5c049f6

…t benchmark files to solution folders

Add benchmark projects to FSharp.sln too

cffd7ac

Fix invalid filepath in CompilerServiceBenchmarks.fs

2d68e24

safesparrow added 5 commits July 11, 2022 01:46

Merge remote-tracking branch 'upstream/main' into benchmarks-cleanup

0baff42

Apply IL Cache Clearing in FSharp.Compiler.Benchmarks as before

5a16d42

Merge remote-tracking branch 'upstream/main' into benchmarks-cleanup

0d0a5d4

Merge main into benchmarks-cleanup

e332fae

Allow skipping local build, add some comments

ab274f8

vzarytovskii approved these changes Jul 15, 2022

View reviewed changes

vzarytovskii merged commit c19ebd5 into dotnet:main Jul 15, 2022

vzarytovskii mentioned this pull request Aug 1, 2022

Reliable FCS benchmarks which run on every release #13606

Open

This was referenced Jan 3, 2024

Facelifting FCS benchmarks #16484

Merged

Removing a dead bench notebook #16496

Merged

psfinaki mentioned this pull request Jan 19, 2024

Smoke test some benchmarks in CI #16554

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix and refactor FCSBenchmarks #13419

Fix and refactor FCSBenchmarks #13419

safesparrow commented Jun 29, 2022 •

edited

Loading

safesparrow Jun 30, 2022

safesparrow Jun 30, 2022

safesparrow Jun 30, 2022

safesparrow Jun 30, 2022

safesparrow commented Jun 30, 2022

vzarytovskii commented Jun 30, 2022 •

edited

Loading

auduchinok commented Jun 30, 2022 •

edited

Loading

safesparrow commented Jul 7, 2022

safesparrow commented Jul 13, 2022 •

edited

Loading

vzarytovskii commented Jul 15, 2022

Fix and refactor FCSBenchmarks #13419

Fix and refactor FCSBenchmarks #13419

Conversation

safesparrow commented Jun 29, 2022 • edited Loading

Changes in all benchmarks

Changes in historical benchmark (BenchmarkComparison)

Changes in the notebook (runner.ipynb)

Changes in the BDN benchmark (now renamed to HistoricalBenchmark.fsproj)

Changes in FSharp.Compiler.Benchmarks

safesparrow Jun 30, 2022

Choose a reason for hiding this comment

safesparrow Jun 30, 2022

Choose a reason for hiding this comment

safesparrow Jun 30, 2022

Choose a reason for hiding this comment

safesparrow Jun 30, 2022

Choose a reason for hiding this comment

safesparrow commented Jun 30, 2022

vzarytovskii commented Jun 30, 2022 • edited Loading

auduchinok commented Jun 30, 2022 • edited Loading

safesparrow commented Jul 7, 2022

safesparrow commented Jul 13, 2022 • edited Loading

vzarytovskii commented Jul 15, 2022

safesparrow commented Jun 29, 2022 •

edited

Loading

Changes in historical benchmark (`BenchmarkComparison`)

Changes in the notebook (`runner.ipynb`)

Changes in the BDN benchmark (now renamed to `HistoricalBenchmark.fsproj`)

vzarytovskii commented Jun 30, 2022 •

edited

Loading

auduchinok commented Jun 30, 2022 •

edited

Loading

safesparrow commented Jul 13, 2022 •

edited

Loading