Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

release: 23.1 microbenchmark regression investigation and sign-off (early edition) #96960

Closed
4 tasks done
herkolategan opened this issue Feb 10, 2023 · 2 comments
Closed
4 tasks done

release: 23.1 microbenchmark regression investigation and sign-off (early edition) #96960

herkolategan opened this issue Feb 10, 2023 · 2 comments
Labels
C-performance Perf of queries or internals. Solution not expected to change functional behavior. T-testeng TestEng Team

Comments

@herkolategan
Copy link
Collaborator

herkolategan commented Feb 10, 2023 •
edited
Loading

Revisions and Binaries

Master: cc99062
./dev test-binaries ./pkg/sql/... --output=test_binaries.tar.gz
22.2.3: 81a114c
./dev test-binaries ./pkg/sql/... --output=test_binaries_compare.tar.gz

Execution

Cluster: ./bin/roachprod create herko-bench -n 16 --lifetime "24h" --clouds gce --gce-machine-type "n1-standard-8" --gce-zones="us-east4-c" --os-volume-size=200
Timeout: 20m
Iterations: 10
Benchtime: 1s
Random Seed: 1

./dev build roachprod-microbench
./bin/roachprod-microbench ./artifacts/log -cluster herko-bench -compare-binaries ./bin/test_binaries_compare.tar.gz -shell="export COCKROACH_RANDOM_SEED=1" -iterations=10 -timeout=20m -- -test.benchtime=1s

Analytics

Iterations per microbenchmark, duration of each iteration: https://docs.google.com/spreadsheets/d/1IJ9-lbMo2mJ_O-NoQU77iRYvuElRwjaIbajPD5BNUhU/edit?usp=sharing

Timeouts:

Timeouts are on a per microbenchmark function level.
Either the whole microbenchmark timed out, or some of the permutations.

BenchmarkDatabaseBackup
BenchmarkDatabaseRestore
BenchmarkIntentRangeResolution
BenchmarkMVCCGarbageCollect
BenchmarkMVCCGet_Pebble
BenchmarkMVCCReverseScan_Pebble
BenchmarkMVCCScanGarbage_Pebble
BenchmarkMVCCScan_Pebble

Open Issues:

Failures:

Sheets

Generated sheet for pkg/sql/...
https://docs.google.com/spreadsheets/d/1uDYotBARIqdL5nSquMoF7Fy3qqoMfNigwHAcB8NPR7M/edit
Generated sheet for pkg/storage/...
https://docs.google.com/spreadsheets/d/1kagTm9WXv9CSn78nDg3iD0bsnFXhc-r4a0L29hP_-No/edit
Generated sheet for pkg/col/...
https://docs.google.com/spreadsheets/d/1VZPc_Ft9RKK60OcjO_x2zKv9LttZ--U3SVsxeEwFcik/edit
Generated sheet for pkg/workload/...
https://docs.google.com/spreadsheets/d/1jgaXqS_3DikzbTxP5WsJ_7gnDrX6jzTNr0ufmCrGK_8/edit
Generated sheet for pkg/ts/...
https://docs.google.com/spreadsheets/d/1r2IXm1gZviRVPIwZZHGQCZUTwY_N12WZrdX1TDpx4yI/edit
Generated sheet for pkg/roachpb/...
https://docs.google.com/spreadsheets/d/1Ejd09DpvXszNYHYf82Sg857tZscJURaScipU_UtXO6E/edit
Generated sheet for pkg/server/...
https://docs.google.com/spreadsheets/d/1phImav7bvnQEzd8okFjDRjoxPMlptevEs7M7a9kgtIQ/edit
Generated sheet for pkg/bench/...
https://docs.google.com/spreadsheets/d/1l3BnrnfTYL6BudvLfksRzccoQwYX3ShwaV36-OU8Hhk/edit
Generated sheet for pkg/gossip/...
https://docs.google.com/spreadsheets/d/17ls1Zk23E3_27aP4supiXbph-kOsFYe0DsMpNqHPRg4/edit
Generated sheet for pkg/geo/...
https://docs.google.com/spreadsheets/d/1G0rfnnBNPZYjgpiRMokNB4diXo7rrV2VkspqXwZclOQ/edit
Generated sheet for pkg/blobs/...
https://docs.google.com/spreadsheets/d/1ksRJCTSFVBm0Y-whw0_1JXBf_GHWDTqdMISuOHpzG34/edit
Generated sheet for pkg/spanconfig/...
https://docs.google.com/spreadsheets/d/18oTvIY28XEZAEtajTxETM8xrj_cP2g5-j6_cB6dbrjs/edit
Generated sheet for pkg/ccl/...
https://docs.google.com/spreadsheets/d/1HxTZbtEMscgzt_4f-z1D2KB6HdvXw-qGaZF4zkXV1ik/edit
Generated sheet for pkg/kv/...
https://docs.google.com/spreadsheets/d/10GhYr_91CANCNKOM_gPy7Sk9hQkTyQGNgNwNgfHeUtI/edit
Generated sheet for pkg/util/...
https://docs.google.com/spreadsheets/d/1yhzKN-A5iwllJqG_Z9VnD7r1gzvwfh_0J8cklginBFE/edit
Generated sheet for pkg/rpc/...
https://docs.google.com/spreadsheets/d/10hoxvxcdiN79fKDfWj6uLO-iemETDbLxCUI0V8xFWuE/edit

Jira issue: CRDB-24434
@herkolategan herkolategan added C-performance Perf of queries or internals. Solution not expected to change functional behavior. T-testeng TestEng Team labels Feb 10, 2023
@blathers-crl
Copy link

blathers-crl bot commented Feb 10, 2023

cc @cockroachdb/test-eng

@herkolategan herkolategan changed the title early edition: release: 22.3* microbenchmark regression investigation and sign-off early edition: release: (master) 22.3* microbenchmark regression investigation and sign-off Feb 10, 2023
@jbowens jbowens mentioned this issue Feb 23, 2023
Closed
@herkolategan herkolategan changed the title early edition: release: (master) 22.3* microbenchmark regression investigation and sign-off release: (master) 22.3* microbenchmark regression investigation and sign-off (early edition) Mar 3, 2023
@nvanbenschoten nvanbenschoten mentioned this issue Mar 6, 2023
Closed
7 tasks
@srosenberg srosenberg changed the title release: (master) 22.3* microbenchmark regression investigation and sign-off (early edition) release: 23.1 microbenchmark regression investigation and sign-off (early edition) Mar 9, 2023
@rafiss
Copy link
Collaborator

rafiss commented Mar 9, 2023

ORMQueries/django_table_introspection_8_tables and ORMQueries/django_table_introspection_1_table both had ~1000% regressions in time/op. #96397 was merged after the benchmarks were run and should improve this a lot.

@mgartner mgartner mentioned this issue Mar 9, 2023
Closed
18 tasks
jbowens added a commit to jbowens/cockroach that referenced this issue Mar 15, 2023
@jbowens
storage: batch pebbleResults allocation
7965e26 
Embed a `pebbleResults` struct on the `pebbleMVCCScanner` to avoid an
allocation.

```
                                                                     │  23.1.txt   │           23.1-wip.txt            │
                                                                     │   sec/op    │   sec/op     vs base              │
MVCCGet_Pebble/batch=true/versions=100/valueSize=8/numRangeKeys=0-24   109.3µ ± 1%   105.2µ ± 3%  -3.73% (p=0.009 n=6)

                                                                     │   23.1.txt   │          23.1-wip.txt          │
                                                                     │     B/s      │      B/s       vs base         │
MVCCGet_Pebble/batch=true/versions=100/valueSize=8/numRangeKeys=0-24   68.36Ki ± 0%   78.12Ki ± 12%  ~ (p=0.061 n=6)

                                                                     │  23.1.txt   │            23.1-wip.txt            │
                                                                     │    B/op     │    B/op      vs base               │
MVCCGet_Pebble/batch=true/versions=100/valueSize=8/numRangeKeys=0-24   354.0 ± 22%   228.5 ± 28%  -35.45% (p=0.002 n=6)

                                                                     │  23.1.txt  │           23.1-wip.txt            │
                                                                     │ allocs/op  │ allocs/op   vs base               │
MVCCGet_Pebble/batch=true/versions=100/valueSize=8/numRangeKeys=0-24   3.000 ± 0%   2.000 ± 0%  -33.33% (p=0.002 n=6)
```

Informs cockroachdb#96960.
Epic: None
Release note: None
@jbowens jbowens mentioned this issue Mar 15, 2023
Merged
jbowens added a commit to jbowens/cockroach that referenced this issue Mar 16, 2023
@jbowens
storage: batch pebbleResults allocation
d8515b1 
Embed a `pebbleResults` struct on the `pebbleMVCCScanner` to avoid an
allocation.

```
                                                                     │  23.1.txt   │           23.1-wip.txt            │
                                                                     │   sec/op    │   sec/op     vs base              │
MVCCGet_Pebble/batch=true/versions=100/valueSize=8/numRangeKeys=0-24   109.3µ ± 1%   105.2µ ± 3%  -3.73% (p=0.009 n=6)

                                                                     │   23.1.txt   │          23.1-wip.txt          │
                                                                     │     B/s      │      B/s       vs base         │
MVCCGet_Pebble/batch=true/versions=100/valueSize=8/numRangeKeys=0-24   68.36Ki ± 0%   78.12Ki ± 12%  ~ (p=0.061 n=6)

                                                                     │  23.1.txt   │            23.1-wip.txt            │
                                                                     │    B/op     │    B/op      vs base               │
MVCCGet_Pebble/batch=true/versions=100/valueSize=8/numRangeKeys=0-24   354.0 ± 22%   228.5 ± 28%  -35.45% (p=0.002 n=6)

                                                                     │  23.1.txt  │           23.1-wip.txt            │
                                                                     │ allocs/op  │ allocs/op   vs base               │
MVCCGet_Pebble/batch=true/versions=100/valueSize=8/numRangeKeys=0-24   3.000 ± 0%   2.000 ± 0%  -33.33% (p=0.002 n=6)
```

Informs cockroachdb#96960.
Epic: None
Release note: None
jbowens added a commit to jbowens/cockroach that referenced this issue Mar 16, 2023
@jbowens
storage: batch pebbleResults allocation
d93a851 
Embed a `pebbleResults` struct on the `pebbleMVCCScanner` to avoid an
allocation.

```
                                                                     │  23.1.txt   │           23.1-wip.txt            │
                                                                     │   sec/op    │   sec/op     vs base              │
MVCCGet_Pebble/batch=true/versions=100/valueSize=8/numRangeKeys=0-24   109.3µ ± 1%   105.2µ ± 3%  -3.73% (p=0.009 n=6)

                                                                     │   23.1.txt   │          23.1-wip.txt          │
                                                                     │     B/s      │      B/s       vs base         │
MVCCGet_Pebble/batch=true/versions=100/valueSize=8/numRangeKeys=0-24   68.36Ki ± 0%   78.12Ki ± 12%  ~ (p=0.061 n=6)

                                                                     │  23.1.txt   │            23.1-wip.txt            │
                                                                     │    B/op     │    B/op      vs base               │
MVCCGet_Pebble/batch=true/versions=100/valueSize=8/numRangeKeys=0-24   354.0 ± 22%   228.5 ± 28%  -35.45% (p=0.002 n=6)

                                                                     │  23.1.txt  │           23.1-wip.txt            │
                                                                     │ allocs/op  │ allocs/op   vs base               │
MVCCGet_Pebble/batch=true/versions=100/valueSize=8/numRangeKeys=0-24   3.000 ± 0%   2.000 ± 0%  -33.33% (p=0.002 n=6)
```

Informs cockroachdb#96960.
Epic: None
Release note: None
craig bot pushed a commit that referenced this issue Mar 16, 2023
@jbowens
Merge #98153 #98718
82fd797 
98153: sqlproxyccl: change denylist into an access control list r=pjtatlow a=pjtatlow

To support an IP Allowlist in the sqlproxy, this change extends
the denylist code to make the Watcher support multiple AccessControllers.
Each AccessController is consulted before allowing a connection through,
and rechecked on any changes to the underlying files.

Part of: [CC-8136](https://cockroachlabs.atlassian.net/browse/CC-8136)

98718: storage: batch pebbleResults allocation r=sumeerbhola a=jbowens

Embed a `pebbleResults` struct on the `pebbleMVCCScanner` to avoid an allocation.

```
                                                                     │  23.1.txt   │           23.1-wip.txt            │
                                                                     │   sec/op    │   sec/op     vs base              │
MVCCGet_Pebble/batch=true/versions=100/valueSize=8/numRangeKeys=0-24   109.3µ ± 1%   105.2µ ± 3%  -3.73% (p=0.009 n=6)

                                                                     │   23.1.txt   │          23.1-wip.txt          │
                                                                     │     B/s      │      B/s       vs base         │
MVCCGet_Pebble/batch=true/versions=100/valueSize=8/numRangeKeys=0-24   68.36Ki ± 0%   78.12Ki ± 12%  ~ (p=0.061 n=6)

                                                                     │  23.1.txt   │            23.1-wip.txt            │
                                                                     │    B/op     │    B/op      vs base               │
MVCCGet_Pebble/batch=true/versions=100/valueSize=8/numRangeKeys=0-24   354.0 ± 22%   228.5 ± 28%  -35.45% (p=0.002 n=6)

                                                                     │  23.1.txt  │           23.1-wip.txt            │
                                                                     │ allocs/op  │ allocs/op   vs base               │
MVCCGet_Pebble/batch=true/versions=100/valueSize=8/numRangeKeys=0-24   3.000 ± 0%   2.000 ± 0%  -33.33% (p=0.002 n=6)
```

Informs #96960.
Epic: None
Release note: None

Co-authored-by: PJ Tatlow <[email protected]>
Co-authored-by: Jackson Owens <[email protected]>
@timgraham timgraham mentioned this issue Apr 1, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C-performance Perf of queries or internals. Solution not expected to change functional behavior. T-testeng TestEng Team
Projects
No open projects
Test Engineering
Status: Done
Milestone
No milestone
Development

No branches or pull requests
3 participants
@herkolategan @rafiss @srosenberg