tools: Created benchmark suite #160
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jsimnz
added
feature
Codecov Report
@@ Coverage Diff @@
## develop #160 +/- ##
===========================================
- Coverage 62.85% 59.06% -3.80%
===========================================
Files 81 87 +6
Lines 7737 8344 +607
===========================================
+ Hits 4863 4928 +65
- Misses 2351 2890 +539
- Partials 523 526 +3
|
|
I suggest adding to the main Posted here as I'm unsure of what's the process to contribute a small change like this to the branch. |
AndrewSisley
requested changes
Feb 1, 2022
There was a problem hiding this comment.
Looks good, with lots small comments and nothing too serious. Would however suggest that this could have been a series of much smaller PRs and bits could have been merged earlier - even if it meant a bit of extra reviewing due to changes you will have made after any hypothetical early merges.
|
Also - outputted metrics seem either confused, or we have an embedded time machine?
EDIT: I guess this is default output from go.bench - hopefully we can find a way to override that. |
AndrewSisley
reviewed
Feb 1, 2022
AndrewSisley
reviewed
Feb 1, 2022
AndrewSisley
reviewed
Feb 1, 2022
AndrewSisley
reviewed
Feb 1, 2022
jsimnz
force-pushed
the
jsimnz/feat/benchmark-suite
branch
from
97febc0 to
bfe35dd
Compare
AndrewSisley
approved these changes
Feb 1, 2022
…inistic seed for the RNG system to produce consistent results
jsimnz
force-pushed
the
jsimnz/feat/benchmark-suite
branch
from
bfe35dd to
f6e06d0
Compare
Closed
jsimnz
added a commit
that referenced
this pull request
Feb 7, 2022
* Added bench readme * Started fixture framework to generate doc data * basic bench runner script. * started collection benchmark outline * Updated DB Collection client interface API * Added db cleanup routines * Switched from ResetTimer, to Start/Stop so it can be composable * Updated collection client API to include new context arg * updated gitignore to ignore benchmark artifacts * Updated test suite structure * Renamed bench file, added sync/async benchmarks * Reorganized benchmark folder to match db/tests structure. Removed old db-init files for benchmark * Reorganized collection tests into smaller units. Added async read tests. WIP: async backfill writes * Updated fixture data * Renamed get bench tests to read * Added Create/Write tests, updated utils structure, implemented async backfilling for benchmarks * Added CreateMany tests * Fixed issue where discarded transactions we're causing side-effect mutation on the document object * Fixed async transaction conflict issue * Started underlying storage engine benchmarks * Started put tests on storage, added more get tests * Updated collection benchmark utilities, extracted into dedicated package * Added new FieldDescription Meta data entry to indicate 'synthetic' internal IDs for managed related types * Added more query benchmarks, added query generation to fixtures package * Fixed tests from BREAKING change resulted from IPLD delta payload restructure * Removed auto query generation for fixtures, manual for now * Restructured benchmarks to further match db test structure. Removed opcount * Added filter query benchmarks * Added offset/limit + sort benchmarks. Removed old untracked deleted benchmark files * Updated DB and planner to expose make plan functions as public * Added benchmarks for planner/parser * Added query formatting with dockey replacement from inserted set * Added benchmarks for single and multi point lookups by dockey(s) * Fixed missing func args resulting from rebase * Added makefile * Added some comments to the async implementations. Also added a deterministic seed for the RNG system to produce consistent results * Updated CIDs from tests based on breaking change of IPLD format * Added put and put many benchmarks to storage * updated dependencies required via go mod tidy (CI issue) * Removing deadcode or useless comments * Replaced WithSchema with ForSchema * Renamed fixture.Context to fixture.Generator * Updated benchmark runner utils to take a context instead of creating a new one * Updated all references to benchmark runners for new context arg * Refactored StartTimer() calls and numType arg * removed dead/commented code * Updated public API for query planner * Added StopTimer() calls to add accuracy to the benchmarks * Updated benchmark loop to use ResetTimer for more accuracy * Moved valueSize array out * Replaced custom interface for TempDir with proper testing.TB interface * Minor comments and stuff * Moved the testutil NewDB related funcs from /tests to /bench * Removed rando comment * fixed linter errors BREAKING CHANGE: Changes internal IPLD format and resulting delta CIDs.
shahzadlone
pushed a commit
to shahzadlone/defradb
that referenced
this pull request
Feb 23, 2024
* Added bench readme * Started fixture framework to generate doc data * basic bench runner script. * started collection benchmark outline * Updated DB Collection client interface API * Added db cleanup routines * Switched from ResetTimer, to Start/Stop so it can be composable * Updated collection client API to include new context arg * updated gitignore to ignore benchmark artifacts * Updated test suite structure * Renamed bench file, added sync/async benchmarks * Reorganized benchmark folder to match db/tests structure. Removed old db-init files for benchmark * Reorganized collection tests into smaller units. Added async read tests. WIP: async backfill writes * Updated fixture data * Renamed get bench tests to read * Added Create/Write tests, updated utils structure, implemented async backfilling for benchmarks * Added CreateMany tests * Fixed issue where discarded transactions we're causing side-effect mutation on the document object * Fixed async transaction conflict issue * Started underlying storage engine benchmarks * Started put tests on storage, added more get tests * Updated collection benchmark utilities, extracted into dedicated package * Added new FieldDescription Meta data entry to indicate 'synthetic' internal IDs for managed related types * Added more query benchmarks, added query generation to fixtures package * Fixed tests from BREAKING change resulted from IPLD delta payload restructure * Removed auto query generation for fixtures, manual for now * Restructured benchmarks to further match db test structure. Removed opcount * Added filter query benchmarks * Added offset/limit + sort benchmarks. Removed old untracked deleted benchmark files * Updated DB and planner to expose make plan functions as public * Added benchmarks for planner/parser * Added query formatting with dockey replacement from inserted set * Added benchmarks for single and multi point lookups by dockey(s) * Fixed missing func args resulting from rebase * Added makefile * Added some comments to the async implementations. Also added a deterministic seed for the RNG system to produce consistent results * Updated CIDs from tests based on breaking change of IPLD format * Added put and put many benchmarks to storage * updated dependencies required via go mod tidy (CI issue) * Removing deadcode or useless comments * Replaced WithSchema with ForSchema * Renamed fixture.Context to fixture.Generator * Updated benchmark runner utils to take a context instead of creating a new one * Updated all references to benchmark runners for new context arg * Refactored StartTimer() calls and numType arg * removed dead/commented code * Updated public API for query planner * Added StopTimer() calls to add accuracy to the benchmarks * Updated benchmark loop to use ResetTimer for more accuracy * Moved valueSize array out * Replaced custom interface for TempDir with proper testing.TB interface * Minor comments and stuff * Moved the testutil NewDB related funcs from /tests to /bench * Removed rando comment * fixed linter errors BREAKING CHANGE: Changes internal IPLD format and resulting delta CIDs.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #71 + Closes #70 + Closes #72
Create the initial benchmark suite to be used to track relative differences in performance between commits and PRs.
Currently, the following suites are implemented:
The Raw Storate API is used as a base line reference for how the storage backend is performing relative to the higher level operations of Defra.
Theres a LOT of changes here, including several non benchmark related bugs that were found while implementing the benchmarks. Fixes #113
Unrelated, but I mentioned before that I got myself into git hell by accidentally forking this branch off of another major feat for the Time Travel queries. Turns out that was false, and this branch is clean from a older develop.
Additionally, this branch was recently rebased against the most recent develop changes, so it should be pretty clean. Only notable conflicts from the rebase was in
db/tests/utils.gosince both Andy and I made some changes for how test DBs are created, but I think I merged them pretty cleanly.Todo after merge:
bench/folderBREAKING CHANGE: 1) Updates the CRDT Delta payloads, causing different CIDs to be associated for the same input compared to older version. 2) Updated client/core interfaces.