Aurora Engine Benchmarks

[birchmd]

The Near runtime has the following requirements:

1. Regardless of what work is done in a transaction (computation, io, etc), Near gas used should always be proportional to the wall clock execution time
2. This proportionality should be such that 1000 Tgas can be consumed in < 1s (otherwise Near's blocktime can be thrown off)

The purpose of this PR is to create benchmarks involving the EVM wasm contract to test these requirements hold.

Wall-clock time is an inherently volatile metric because many factors contribute to it, so results from this should be taken with a grain of salt; though great deviations from expectations should be investigated more closely.

The EVM contract is executed within the near-vm-runner. This also records the Near gas spent on the computation. The wall clock measurements are done using criterion.

Two work loads are considered:

• Transferring eth between two accounts
• Deploying an eth smart contract (of various sizes)

Note: these work loads are done as eth transactions sent to the EVM contract, so both also involve checking the validity of the transaction which includes a Keccak256 hash computation and Secp256k1 signature check. (In fact, this is likely the bulk of the work involved).

Using the timing measurements from criterion and the gas measurements from the near-vm-runner, we can measure the relationship between wall-clock time and gas used.

Known limitation: all state is kept in memory, so IO time may not be accurately accounted for.

Closes #31

[birchmd]

I think this is in good enough shape to review (and merge if there are no changes requested). There are a couple remaining TODO items that I have captured as issues, but they are not blocking to this PR in my opinion.

This PR is important because it contains some of the only tests we have for some features. Creating the PR even revealed some bugs which could then be fixed (#42 #44). We will focus more on testing in the future as well of course.

The original impetus for this PR was to benchmark the Near runtime using the EVM contract, as well as get a sense for the cost of common EVM operations. the results of this are summarized in the plot below (note: I've called out a few points of interest in orange, but all the points are present from the benchmarks in this PR):

Some takeaways from this:

• Due to signature checking currently happening in wasm itself (instead of being a host function in the runtime), everything is very expensive in terms of Near gas. This makes getting the new host functions in place very important for usability. Extend the Math API with EVM precompiles near/nearcore#3954
• The points generally lie on a straight line, which is good for our assumption that time is proportional to gas usage. However, the slope of the line looks too steep to me. It means it would take ~1.3 seconds to consume 1000 Tgas, which I think is too long based on Near's target block time (please correct me if I am wrong cc @bowenwang1996 @ailisp @olonho)
• There is some deviation from the straight line at the top end gas usage, which I am not sure how to reconcile.

[joshuajbouw]

Interesting. Shoot. I didn't expect them to be so high, but makes sense given they are backed into the contract. That really adds a lot more importance on near/nearcore#3954.

[alexauroradev]

@birchmd can you please create a plot NEAR Gas vs. Ethereum Gas. This will help us to determine the recalculation formula + estimate what size of a transaction we can accommodate within 300 Tgas.

[joshuajbouw]

Looks really good. Nothing to really note or point out from me. Good work.

[birchmd]

@djsatok The graph you asked for is below:

The correlation here is not very strong. The differences seem to be that we charge much more for pre-compiles, and much less for deploying code. Both of these make sense to me. Our "pre-compiles" are actually executing in wasm right now, so the gas cost is far off what it should be; again pointing to the importance of near/nearcore#3954 . And I think Near storage is supposed to be much cheaper in terms of gas than on Eth because of the "storage staking" concept.

The Eth transfer and ERC20 transfer both happen to lie close-ish to the linear regression line, so maybe it could be used to give a rough estimate of the relationship between Near and Eth gas (obviously it totally does not work in the extreme cases of transactions that are all storage or all pre-compiles). Using that measure, you should be able to fit approximately 230,000 Eth gas into 300 Near Tgas.

[artob]

@birchmd Assigning this back to you to wrap up review discussions. Please assign to me once ready to merge.

[birchmd]

Back to you @artob

Should be good to merge once CI finishes.

Note: there will need to be a follow-up PR after #48 lands because it changes the output format of the contract, and I rely on the output to learn the address of deployed contracts (e.g. ERC-20). So once that PR is merged we'll need to fix the benchmarks to consume the new output format. I can take care of that though, should be quite small.

[joshuajbouw]

New changes LGTM.

[artob]

Note: there will need to be a follow-up PR after #48 lands because it changes the output format of the contract, and I rely on the output to learn the address of deployed contracts (e.g. ERC-20). So once that PR is merged we'll need to fix the benchmarks to consume the new output format. I can take care of that though, should be quite small.

@birchmd I've merged #48 just now. Please update this as needed and then reassign back to me for final review and merge.

[joshuajbouw]

New changes look fine.

[birchmd]

@artob should be good to go now

[artob]

LGTM 👍 Really great work, @birchmd!

[bowenwang1996]

@birchmd any idea what the benchmark would be if the precompiles are executed as host functions? I am not sure what host functions are needed, but I believe some are already available on nightly protocol.

[birchmd]

@bowenwang1996
@djsatok also asked me to get an estimate with precompiles as host functions. I thought the biggest contribution to current gas costs was probably ecrecover, so I ran a benchmark with that one as a host function using the work from near/nearcore#3954 . With that change, an eth transfer drops to around 1Tgas and and erc20 transfer drops to around 15 Tgas. In short, it was a very big improvement.

I didn't run all the benchmarks or re-generate the Tgas vs time graph, but I can do that if you are interested.

The code with ecrecover integrated as a host function is in a branch on my fork of the repo. https://github.com/birchmd/aurora-engine/tree/evm-benchmark-with-math-api

[bowenwang1996]

@birchmd I see. Do we need any other host functions that have not been implemented in nearcore yet?

[birchmd]

@bowenwang1996 As far as I know everything in that nearcore PR is needed. I'm not sure how often each precompile is used in real Eth contracts, so I don't know if some could be considered higher priority than others. But I do know that the EVM contract itself uses ecrecover to validate transactions it receives which is why it is very important it can be executed efficiently.

I don't know if there are other host functions which are needed and not implemented. Maybe @artob or @joshuajbouw would know if there is.