stdlibs benchmark regressions 2025-02-24

[Akirathan]

Several severe regressions in stdlib benchmarks on 2025-02-24. The non-exhaustive list of regressed benchmarks:

• https://enso-org.github.io/engine-benchmark-results/stdlib-benchs.html#org_enso_benchmarks_generated_Column_Numeric_ceil_decimals
• https://enso-org.github.io/engine-benchmark-results/stdlib-benchs.html#org_enso_benchmarks_generated_Column_Numeric_ceil_ints
• https://enso-org.github.io/engine-benchmark-results/stdlib-benchs.html#org_enso_benchmarks_generated_Column_Numeric_ceil_large_ints
• https://enso-org.github.io/engine-benchmark-results/stdlib-benchs.html#org_enso_benchmarks_generated_Column_Numeric_ints_round_decimal_places_0_use_bankers_true
• https://enso-org.github.io/engine-benchmark-results/stdlib-benchs.html#org_enso_benchmarks_generated_Column_Numeric_ints_round_decimal_places_2_use_bankers_false
• https://enso-org.github.io/engine-benchmark-results/stdlib-benchs.html#org_enso_benchmarks_generated_Column_Numeric_ints_round_decimal_places_2_use_bankers_true
• https://enso-org.github.io/engine-benchmark-results/stdlib-benchs.html#org_enso_benchmarks_generated_Column_Numeric_round_ints
• https://enso-org.github.io/engine-benchmark-results/stdlib-benchs.html#org_enso_benchmarks_generated_Column_Numeric_truncate_ints

All of these benchmarks used to be stable, and regressed by cca 1000% on 2025-02-24.

Good revision: 76f5f70
Bad revision: e74db0b
Range: 76f5f70...e74db0b

Looking into the benchmark run for ceil_decimal benchmark, it seems that that particular benchmark has insufficient warmup.

[Akirathan]

Is it possible that #12295 caused these huge regressions?? cc @JaroslavTulach

[JaroslavTulach]

Is it possible that #12295 caused these huge regressions?? cc @JaroslavTulach

I cannot exclude such an alternative. It might, for example, change the warmup characteristics.

Looking into the benchmark run for ceil_decimal benchmark, it seems that that particular benchmark has insufficient warmup.

Yeah, that's possible. There is this new transferToInterpreter() - there were recently doubts about usage of such construct.

[Akirathan]

Yeah, that's possible. There is this new transferToInterpreter() - there were recently doubts about usage of such construct.

Problems with transferToInterpreter revealed in another issue at #11437 (comment). So the very first step is to run with JAVA_OPTS='-Dpolyglot.engine.TraceTransferToInterpreter=true'