tests: external evm benchmarks #24050
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
chfast
force-pushed
the
evm_benchmarks
branch
from
98fc9ab to
ef0ac5f
Compare
|
Comparison between |
chfast
force-pushed
the
evm_benchmarks
branch
2 times, most recently
from
cf433bd to
f4603b1
Compare
|
I changed the implementation to use Using one of the |
|
@holiman, please check if jumpdest analysis is cached for these runs. |
chfast
force-pushed
the
evm_benchmarks
branch
from
6dca006 to
d545a21
Compare
Yeah, so apparently the |
|
It might be possible to fix this via a hack. Instead of You make the |
holiman
requested changes
Dec 14, 2021
chfast
force-pushed
the
evm_benchmarks
branch
from
d545a21 to
d766533
Compare
Maybe the way is to be have API with more explicit code analysis handling. |
|
Any resolution to this? |
Do you have any proposal? I suggested a way to hack around it -- do you think there's some better way to do it? I don't really have any good alternatives. |
|
Wrapping with another Alternative is to have more explicit code analysis cache management, i.e. analyze the code with one API outside of the loop and pass it to |
|
BTW, how do you collect "analysis" stats for benchmarks? |
|
I did one more commit which looks a bit better (and looks better IMHO) but in general the loop seems to be getting little faster but this is not visible in benchmarks doing more than just JUMPDEST. |
The |
|
This makes the analysis become reused: diff --git a/core/vm/analysis.go b/core/vm/analysis.go
index 449cded2a8..5a4fe12d03 100644
--- a/core/vm/analysis.go
+++ b/core/vm/analysis.go
@@ -66,8 +66,11 @@ func (bits *bitvec) codeSegment(pos uint64) bool {
return ((*bits)[pos/8] & (0x80 >> (pos % 8))) == 0
}
+var AnalysisCount = uint64(0)
+
// codeBitmap collects data locations in code.
func codeBitmap(code []byte) bitvec {
+ AnalysisCount++
// The bitmap is 4 bytes longer than necessary, in case the code
// ends with a PUSH32, the algorithm will push zeroes onto the
// bitvector outside the bounds of the actual code.
diff --git a/tests/state_test.go b/tests/state_test.go
index 8ffa001c0b..9905a6cf9f 100644
--- a/tests/state_test.go
+++ b/tests/state_test.go
@@ -184,6 +184,11 @@ func runBenchmark(b *testing.B, t *StateTest) {
key := fmt.Sprintf("%s/%d/Run", subtest.Fork, subtest.Index)
b.Run(key, func(b *testing.B) {
+ vm.AnalysisCount = 0
+ defer func() {
+ b.ReportMetric(float64(vm.AnalysisCount), "analysis")
+ }()
+
vmconfig := vm.Config{}
config, eips, err := GetChainConfig(subtest.Fork)
if err != nil {
@@ -235,6 +240,11 @@ func runBenchmarkCall(b *testing.B, t *StateTest) {
key := fmt.Sprintf("%s/%d/Call", subtest.Fork, subtest.Index)
b.Run(key, func(b *testing.B) {
+ vm.AnalysisCount = 0
+ defer func() {
+ b.ReportMetric(float64(vm.AnalysisCount), "analysis")
+ }()
+
vmconfig := vm.Config{}
config, eips, err := GetChainConfig(subtest.Fork)
@@ -284,7 +294,8 @@ func runBenchmarkCall(b *testing.B, t *StateTest) {
context.BaseFee = baseFee
evm := vm.NewEVM(context, txContext, statedb, config, vmconfig)
- sender := vm.AccountRef(msg.From())
+ sender := vm.NewContract(vm.AccountRef(msg.From()), vm.AccountRef(msg.From()),
+ nil, 0)
b.ResetTimer()
for n := 0; n < b.N; n++ {
// Execute the message. |
|
I compared two implementation variants: Run (old) and Call (new). The difference is small and does not which variant is faster. So I'm for staying with "Call" variant. |
holiman
reviewed
Jan 21, 2022
tests/state_test.go
Outdated
| @@ -61,6 +68,7 @@ func TestState(t *testing.T) { | |||
| for _, dir := range []string{ | |||
| stateTestDir, | |||
| legacyStateTestDir, | |||
| benchmarksDir, // FIXME: This does not seem to work, but we want to test benchmarks! | |||
There was a problem hiding this comment.
What is this fixme about? Do the tests in the benchmarks dir still not work? Something we need to look into or can this comment be removed?
There was a problem hiding this comment.
I believe it actually works as expected. Let me find it in the logs...
There was a problem hiding this comment.
These run, however the "benchmark" dir name is omitted.
--- PASS: TestState (0.05s)
--- PASS: TestState/main/weierstrudel.json (0.05s)
--- PASS: TestState/main/weierstrudel.json/London/0/trie (0.00s)
--- PASS: TestState/main/weierstrudel.json/London/0/snap (0.00s)
--- PASS: TestState/main/weierstrudel.json/London/1/trie (0.00s)
--- PASS: TestState/main/weierstrudel.json/London/1/snap (0.00s)
--- PASS: TestState/main/weierstrudel.json/London/2/trie (0.00s)
--- PASS: TestState/main/weierstrudel.json/London/2/snap (0.00s)
--- PASS: TestState/main/weierstrudel.json/London/3/trie (0.01s)
--- PASS: TestState/main/weierstrudel.json/London/3/snap (0.01s)
--- PASS: TestState/main/weierstrudel.json/London/4/trie (0.01s)
--- PASS: TestState/main/weierstrudel.json/London/4/snap (0.01s)
--- PASS: TestState/main/blake2b_huff.json (0.17s)
chfast
force-pushed
the
evm_benchmarks
branch
from
8f52d41 to
8b5f3ec
Compare
sidhujag
pushed a commit
to syscoin/go-ethereum
that referenced
this pull request
Jan 28, 2022
* tests: add ipsilon/evm-benchmarks git submodule * tests: plug-in evm-benchmarks
JacekGlen
pushed a commit
to JacekGlen/go-ethereum
that referenced
this pull request
May 26, 2022
* tests: add ipsilon/evm-benchmarks git submodule * tests: plug-in evm-benchmarks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I converted evmone's benchmarks to StateTest format and they are available at https://github.com/ipsilon/evm-benchmarks. Here added as a git submodule. We are still discussing how exactly extract and maintain the benchmark suite outside of the evmone project.
Then I copied a lot of code from
TestState()to createBenchmarkState()with the goal to use the lowest EVM execute function possible. I stopped atEVM.Call()but I believe evenInterpreter.Run()is possible.If some refactoring is done, some code can be shared between
TestState()andBenchmarkState(). But the test runner is quite convoluted withreflectbeing used at some level...