6.1.0 looks regressed compared to 5.2.0

[theapache64]

Macrobenchmark Data

Device

• Redmi A1

Result

3 runs of 15 iterations (both 5.2.0 and 6.1.0)

# 5_2_0 - first run
LottieBenchmark_startup
frameDurationCpuMs   P50     12.6,   P90     15.0,   P95     16.4,   P99     36.3
frameOverrunMs       P50     -4.4,   P90      4.3,   P95      5.0,   P99     22.6
Traces: Iteration 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

# 5_2_0 - second run
LottieBenchmark_startup
frameDurationCpuMs   P50     12.6,   P90     15.0,   P95     16.7,   P99     36.4
frameOverrunMs       P50     -5.0,   P90      4.3,   P95      5.0,   P99     25.8
Traces: Iteration 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

# 5_2_0 - third run
LottieBenchmark_startup
frameDurationCpuMs   P50   12.6,   P90   15.0,   P95   16.6,   P99   35.1
frameOverrunMs       P50   -4.0,   P90    4.3,   P95    5.0,   P99   22.1
Traces: Iteration 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

# 6_1_0 - first run
LottieBenchmark_startup
frameDurationCpuMs   P50   13.3,   P90   15.2,   P95   17.2,   P99   37.5
frameOverrunMs       P50    1.2,   P90    4.4,   P95    5.1,   P99   21.1
Traces: Iteration 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

# 6_1_0 - second run
LottieBenchmark_startup
frameDurationCpuMs   P50     13.6,   P90     15.3,   P95     16.9,   P99     37.1
frameOverrunMs       P50      1.7,   P90      4.5,   P95      5.3,   P99     19.6
Traces: Iteration 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

# 6_1_0 - third run
LottieBenchmark_startup
frameDurationCpuMs   P50     13.3,   P90     15.2,   P95     17.3,   P99     38.6
frameOverrunMs       P50      1.3,   P90      4.5,   P95      5.3,   P99     23.7
Traces: Iteration 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Benchart

Copy and paste the above data to https://theapache64.github.io/benchart/ to get the below charts

Benchmark Code

• https://github.com/theapache64/android-benchmark/tree/lottie-benchmark

[gpeal]

Interesting investigations. From your analysis, it would appear that there might be a slight regression but the delta is so small that it's likely still within the margin of error. Performance testing is notoriously difficult to do with high precision.

I just ran your benchmark on my S23 and it doesn't appear to indicate that there has been a regression:

5.2.0
LottieBenchmark_startup
frameDurationCpuMs   P50      6.5,   P90      8.8,   P95      9.5,   P99     13.8
frameOverrunMs       P50     -0.0,   P90      8.4,   P95      9.0,   P99     11.0

LottieBenchmark_startup
frameDurationCpuMs   P50      7.1,   P90      9.0,   P95      9.6,   P99     13.4
frameOverrunMs       P50      0.7,   P90      8.9,   P95      9.2,   P99     10.9
Traces: Iteration 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

LottieBenchmark_startup
frameDurationCpuMs   P50      7.1,   P90      9.0,   P95      9.6,   P99     13.8
frameOverrunMs       P50      0.9,   P90      8.8,   P95      9.2,   P99     11.4
Traces: Iteration 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

6.1.0
LottieBenchmark_startup
frameDurationCpuMs   P50      7.1,   P90      9.0,   P95      9.7,   P99     13.1
frameOverrunMs       P50      0.8,   P90      8.7,   P95      9.1,   P99     10.9
Traces: Iteration 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

LottieBenchmark_startup
frameDurationCpuMs   P50      7.4,   P90      9.1,   P95      9.9,   P99     16.4
frameOverrunMs       P50      1.4,   P90      8.8,   P95      9.2,   P99     12.1
Traces: Iteration 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

LottieBenchmark_startup
frameDurationCpuMs   P50      6.9,   P90      9.0,   P95      9.7,   P99     13.3
frameOverrunMs       P50      0.3,   P90      8.5,   P95      9.1,   P99     11.3
Traces: Iteration 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

It's possible that the updated baseline profiles that will be in the next PR will account for any difference if there is any.

If you do think there is a legitimate issue here, feel free to take a stab at finding and fixing it yourself.