Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use exact multinomial for bulk rand #114

Open
wants to merge 62 commits into
base: lh/anew-dev-2
Choose a base branch
from
Open

Conversation

Tortar
Copy link
Collaborator

@Tortar Tortar commented Mar 9, 2025

still wip, but it improves benchmarks locally, we need some tests in CI for this too

@Tortar Tortar marked this pull request as draft March 9, 2025 19:10
Copy link

github-actions bot commented Mar 9, 2025

Benchmark Results

ca5d92d 58844fb ca5d92d / 58844fb
TTFX excluding time to load 8.6 ± 0 ms 8.79 ± 0 ms 1.08,0.958,0.979
code size in bytes 1.33e+04 ± 0 h 1.56e+04 ± 0 h 0.851,0.851,0.849
code size in lines 459 ± 0 h 565 ± 0 h 0.817,0.817,0.812
code size in syntax nodes 3.44e+03 ± 0 h 4.18e+03 ± 0 h 0.823,0.823,0.822
constructor n=100 σ=0.1 6.15 ± 2.6 μs 9.06 ± 1.1 μs 0.665,0.95,0.679
constructor n=100 σ=1.0 6.88 ± 2.8 μs 7.03 ± 2.5 μs 1.07,1.01,0.979
constructor n=100 σ=10.0 6.72 ± 0.28 μs 7.01 ± 0.43 μs 1.08,1.01,0.958
constructor n=100 σ=100.0 7.07 ± 3.1 μs 6.84 ± 2.1 μs 1.25,0.976,1.03
constructor n=1000 σ=0.1 0.0433 ± 0.0017 ms 0.0454 ± 0.0052 ms 1.04,0.977,0.954
constructor n=1000 σ=1.0 0.0453 ± 0.0015 ms 0.045 ± 0.0014 ms 1.04,0.979,1.01
constructor n=1000 σ=10.0 0.0585 ± 0.0038 ms 0.0574 ± 0.0032 ms 1.04,0.985,1.02
constructor n=1000 σ=100.0 0.0595 ± 0.0015 ms 0.0591 ± 0.0012 ms 1.04,0.991,1.01
constructor n=10000 σ=0.1 0.432 ± 0.032 ms 0.428 ± 0.024 ms 1.09,0.984,1.01
constructor n=10000 σ=1.0 0.435 ± 0.019 ms 0.438 ± 0.022 ms 1.04,0.971,0.994
constructor n=10000 σ=10.0 0.463 ± 0.023 ms 0.461 ± 0.02 ms 1.03,0.971,1
constructor n=10000 σ=100.0 0.579 ± 0.02 ms 0.576 ± 0.023 ms 1.01,0.994,1
delete ∘ rand n=100 σ=0.1 4.71 ± 0.19 μs 4.68 ± 0.2 μs 1,0.996,1.01
delete ∘ rand n=100 σ=1.0 5 ± 0.2 μs 4.96 ± 0.19 μs 1,0.998,1.01
delete ∘ rand n=100 σ=10.0 5.18 ± 0.19 μs 5.16 ± 0.19 μs 0.998,0.996,1
delete ∘ rand n=100 σ=100.0 9.31 ± 0.77 μs 9.23 ± 0.74 μs 1.01,0.998,1.01
delete ∘ rand n=1000 σ=0.1 0.0467 ± 0.00088 ms 0.0463 ± 0.00079 ms 1.01,0.997,1.01
delete ∘ rand n=1000 σ=1.0 0.051 ± 0.0016 ms 0.0506 ± 0.0016 ms 1.01,0.999,1.01
delete ∘ rand n=1000 σ=10.0 0.052 ± 0.00091 ms 0.0516 ± 0.00086 ms 1,0.999,1.01
delete ∘ rand n=1000 σ=100.0 0.0612 ± 0.0014 ms 0.0608 ± 0.0012 ms 1,1,1.01
delete ∘ rand n=10000 σ=0.1 0.503 ± 0.011 ms 0.5 ± 0.012 ms 1.01,0.982,1.01
delete ∘ rand n=10000 σ=1.0 0.545 ± 0.0088 ms 0.542 ± 0.011 ms 1,0.996,1.01
delete ∘ rand n=10000 σ=10.0 0.556 ± 0.019 ms 0.546 ± 0.011 ms 1,0.994,1.02
delete ∘ rand n=10000 σ=100.0 0.546 ± 0.011 ms 0.542 ± 0.01 ms 1.01,1,1.01
empty constructor 1.92 ± 0.29 μs 2.02 ± 0.53 μs 1.18,1.01,0.948
intermixed_h n=100 σ=0.1 14.9 ± 1.7 μs 15 ± 1.9 μs 1.02,1.02,0.994
intermixed_h n=100 σ=1.0 12.1 ± 1.1 μs 12.1 ± 1.3 μs 1.07,0.978,1
intermixed_h n=100 σ=10.0 11.7 ± 1.2 μs 11.8 ± 1.4 μs 1.03,0.966,0.998
intermixed_h n=100 σ=100.0 13.2 ± 1.6 μs 13.2 ± 1.6 μs 1.06,0.986,1
intermixed_h n=1000 σ=0.1 0.113 ± 0.01 ms 0.11 ± 0.0097 ms 1.01,0.986,1.03
intermixed_h n=1000 σ=1.0 0.114 ± 0.0084 ms 0.114 ± 0.0095 ms 1.02,0.997,1
intermixed_h n=1000 σ=10.0 0.109 ± 0.0088 ms 0.108 ± 0.0087 ms 1.01,0.988,1.01
intermixed_h n=1000 σ=100.0 0.121 ± 0.014 ms 0.122 ± 0.014 ms 1.01,0.984,0.992
intermixed_h n=10000 σ=0.1 1.18 ± 0.21 ms 1.22 ± 0.16 ms 0.976,0.977,0.969
intermixed_h n=10000 σ=1.0 1.21 ± 0.15 ms 1.21 ± 0.17 ms 0.995,0.976,1
intermixed_h n=10000 σ=10.0 1.14 ± 0.2 ms 1.09 ± 0.16 ms 0.991,0.982,1.04
intermixed_h n=10000 σ=100.0 1.2 ± 0.24 ms 1.19 ± 0.17 ms 1.01,0.972,1.01
pathological 1 0.0456 ± 0.0002 μs 0.0457 ± 0.00022 μs 0.999,0.993,0.999
pathological 1′ 0.212 ± 0.0017 μs 0.2 ± 0.0018 μs 1.04,1.04,1.06
pathological 2 0.0629 ± 0.0003 μs 0.063 ± 0.00026 μs 1,0.999,0.998
pathological 2′ 0.182 ± 0.0016 μs 0.177 ± 0.0033 μs 1,1.04,1.03
pathological 3 19.5 ± 0.36 ns 18.6 ± 0.24 ns 1,1.01,1.05
pathological 4 0.0623 ± 0.0003 μs 0.0624 ± 0.00028 μs 0.999,1,0.999
pathological 4′ 0.241 ± 0.0016 μs 0.219 ± 0.0029 μs 1.07,1.08,1.1
pathological 5a 0.0558 ± 0.00011 μs 0.0448 ± 0.0002 μs 1,0.995,1.25
pathological 5b 0.045 ± 0.00023 μs 0.0451 ± 0.00023 μs 1,1,0.998
pathological 5b′ 0.474 ± 0.0069 μs 0.451 ± 0.0084 μs 1.08,1.06,1.05
sample (bulk) n=1000 k=10000 σ=0.1 0.239 ± 0.049 ms 0.152 ± 0.043 ms 1.54,1.38,1.57
sample (bulk) n=1000 k=10000 σ=1.0 0.332 ± 0.023 ms 0.204 ± 0.02 ms 1.68,1.61,1.63
sample (bulk) n=1000 k=10000 σ=10.0 0.202 ± 0.06 ms 0.152 ± 0.029 ms 1.26,1.31,1.33
sample (bulk) n=1000 k=10000 σ=100.0 0.166 ± 0.042 ms 0.168 ± 0.045 ms 0.994,1.02,0.986
sample (bulk) n=1000 k=100000 σ=0.1 2.46 ± 0.41 ms 1.44 ± 0.66 ms 1.32,1.62,1.7
sample (bulk) n=1000 k=100000 σ=1.0 3.31 ± 0.25 ms 1.95 ± 0.17 ms 1.76,1.76,1.7
sample (bulk) n=1000 k=100000 σ=10.0 1.95 ± 0.42 ms 1.4 ± 0.26 ms 1.48,1.46,1.39
sample (bulk) n=1000 k=100000 σ=100.0 1.59 ± 0.39 ms 1.51 ± 0.34 ms 1,1.08,1.05
sample (bulk) n=1000 k=1000000 σ=0.1 23.4 ± 0.13 ms 19.2 ± 3.4 ms 1.46,1.62,1.22
sample (bulk) n=1000 k=1000000 σ=1.0 0.0392 ± 0.0015 s 19.4 ± 1 ms 1.66,1.63,2.02
sample (bulk) n=1000 k=1000000 σ=10.0 20.4 ± 2.8 ms 14 ± 2.1 ms 1.45,1.49,1.46
sample (bulk) n=1000 k=1000000 σ=100.0 15.9 ± 3.4 ms 17.8 ± 2.1 ms 1.04,1.13,0.892
sample (bulk) n=1000000 k=10000 σ=0.1 0.628 ms 0.255 ms 2.91,2.27,2.47
sample (bulk) n=1000000 k=10000 σ=1.0 1.04 ms 0.372 ms 3.58,2.25,2.79
sample (bulk) n=1000000 k=10000 σ=10.0 0.179 ms 0.209 ms 1.04,1.18,0.857
sample (bulk) n=1000000 k=10000 σ=100.0 0.238 ms 0.19 ms 1.14,0.318,1.25
sample (bulk) n=1000000 k=100000 σ=0.1 5.77 ± 0.3 ms 3.1 ± 0.31 ms 2.58,1.95,1.86
sample (bulk) n=1000000 k=100000 σ=1.0 7.39 ± 0.25 ms 3.12 ± 0.031 ms 3.35,2.45,2.37
sample (bulk) n=1000000 k=100000 σ=10.0 2.8 ± 0.15 ms 1.41 ± 0.1 ms 1.89,1.86,1.98
sample (bulk) n=1000000 k=100000 σ=100.0 1.91 ± 0.054 ms 2 ± 0.01 ms 0.933,0.848,0.953
sample (bulk) n=1000000 k=1000000 σ=0.1 0.0399 ± 0.0013 s 20.8 ± 0.21 ms 3.57,1.83,1.92
sample (bulk) n=1000000 k=1000000 σ=1.0 0.0604 s 23.1 ms 3.07,2.39,2.61
sample (bulk) n=1000000 k=1000000 σ=10.0 21.2 ± 5.1 ms 14.9 ± 0.26 ms 1.94,2.91,1.43
sample (bulk) n=1000000 k=1000000 σ=100.0 14.4 ms 13.3 ms 0.716,1.36,1.08
sample (single) n=100 σ=0.1 28 ± 0.64 ns 27.5 ± 0.61 ns 1.02,1.03,1.02
sample (single) n=100 σ=1.0 0.0325 ± 0.0024 μs 0.0321 ± 0.0023 μs 0.998,1.02,1.01
sample (single) n=100 σ=10.0 21.4 ± 5.3 ns 21.4 ± 5.1 ns 0.998,1.01,1
sample (single) n=100 σ=100.0 18.5 ± 4.2 ns 18.5 ± 3.8 ns 1,0.978,1
sample (single) n=1000 σ=0.1 25.9 ± 4.5 ns 25.5 ± 4.5 ns 1.01,1.02,1.02
sample (single) n=1000 σ=1.0 0.0354 ± 0.0023 μs 0.0347 ± 0.0022 μs 1.02,1.03,1.02
sample (single) n=1000 σ=10.0 22.4 ± 6.3 ns 22.5 ± 5.6 ns 1.01,1.01,0.993
sample (single) n=1000 σ=100.0 18.8 ± 4.2 ns 18.5 ± 4 ns 1.01
sample (single) n=10000 σ=0.1 0.0342 ± 0.0012 μs 0.0336 ± 0.0013 μs 1.02
sample (single) n=10000 σ=1.0 0.0378 ± 0.0012 μs 0.0368 ± 0.001 μs 1.03,1.01,1.03
sample (single) n=10000 σ=10.0 22.8 ± 6.8 ns 23.9 ± 6.5 ns 0.975,1.04,0.953
sample (single) n=10000 σ=100.0 18.7 ± 4.2 ns 19.3 ± 3.9 ns 0.965,0.991,0.972
summarysize n=100 σ=0.1 1.13e+05 ± 0 h 1.13e+05 ± 0 h 1
summarysize n=100 σ=1.0 1.13e+05 ± 0 h 1.13e+05 ± 0 h 1
summarysize n=100 σ=10.0 1.13e+05 ± 0 h 1.13e+05 ± 0 h 1
summarysize n=100 σ=100.0 1.13e+05 ± 0 h 1.13e+05 ± 0 h 1
summarysize n=1000 σ=0.1 1.42e+05 ± 0 h 1.42e+05 ± 0 h 1
summarysize n=1000 σ=1.0 1.42e+05 ± 0 h 1.42e+05 ± 0 h 1
summarysize n=1000 σ=10.0 1.42e+05 ± 0 h 1.42e+05 ± 0 h 1
summarysize n=1000 σ=100.0 1.42e+05 ± 0 h 1.42e+05 ± 0 h 1
summarysize n=10000 σ=0.1 1e+06 ± 0 h 1e+06 ± 0 h 1
summarysize n=10000 σ=1.0 1e+06 ± 0 h 1e+06 ± 0 h 1
summarysize n=10000 σ=10.0 1e+06 ± 0 h 1e+06 ± 0 h 1
summarysize n=10000 σ=100.0 1e+06 ± 0 h 1e+06 ± 0 h 1
update ∘ rand n=100 σ=0.1 0.0872 ± 0.0024 μs 0.0853 ± 0.0023 μs 1.03,0.935,1.02
update ∘ rand n=100 σ=1.0 0.0932 ± 0.0028 μs 0.0922 ± 0.003 μs 1.02,0.94,1.01
update ∘ rand n=100 σ=10.0 0.1 ± 0.0036 μs 0.104 ± 0.004 μs 0.999,0.971,0.959
update ∘ rand n=100 σ=100.0 0.228 ± 0.024 μs 0.231 ± 0.027 μs 1.03,0.996,0.986
update ∘ rand n=1000 σ=0.1 0.0878 ± 0.0022 μs 0.086 ± 0.0024 μs 1.02,0.938,1.02
update ∘ rand n=1000 σ=1.0 0.0941 ± 0.003 μs 0.0925 ± 0.0027 μs 1.02,0.938,1.02
update ∘ rand n=1000 σ=10.0 0.0993 ± 0.0017 μs 0.0988 ± 0.0019 μs 1.01,0.943,1
update ∘ rand n=1000 σ=100.0 0.22 ± 0.0083 μs 0.224 ± 0.0093 μs 1.02,1,0.979
update ∘ rand n=10000 σ=0.1 0.0955 ± 0.0013 μs 0.0934 ± 0.00099 μs 1.02,0.944,1.02
update ∘ rand n=10000 σ=1.0 0.0986 ± 0.0014 μs 0.0979 ± 0.0027 μs 1.04,0.938,1.01
update ∘ rand n=10000 σ=10.0 0.0986 ± 0.0015 μs 0.0967 ± 0.001 μs 1.02,0.925,1.02
update ∘ rand n=10000 σ=100.0 0.196 ± 0.0025 μs 0.196 ± 0.003 μs 1.01,0.977,0.997
time_to_load 0.0742 ± 0.00045 s 0.368 ± 0.0012 s 0.202,0.205,0.202

Benchmark Plots

A plot of the benchmark results have been uploaded as an artifact to the workflow run for this PR.
Go to "Actions"->"Benchmark a pull request"->[the most recent run]->"Artifacts" (at the bottom).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant