Skip to content
This repository has been archived by the owner on Jul 1, 2021. It is now read-only.

tests: update to match py-evm: don't skip Constantinople, add Petersburg, use --fork; break out long-running categories #608

Merged
merged 8 commits into from
May 21, 2019
Merged

tests: update to match py-evm: don't skip Constantinople, add Petersburg, use --fork; break out long-running categories #608

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
1146f83
fixtures: update git submodule to same commit as in `py-evm`.
veox May 15, 2019
13e4f48
tests: don't skip fixtures for 'Constantinople{,fix}' + mark SLOW_TES…
veox May 14, 2019
137e287
ci: add `petersburg` CI job.
veox May 15, 2019
712f16e
ci/tests: ignore tests not matching `--fork` arg given to pytest.
veox May 16, 2019
adf43f6
tests: update SLOW_TESTS now that more forks are tested + remove dupl…
veox May 16, 2019
020778e
ci: break out `zeroKnowledge{,2}` and `sstore` tests into separate CI…
veox May 17, 2019
61a25fc
ci: run rpc-state-{quadratic,sstore,zero_knowledge} in py37 env.
veox May 21, 2019
c3fe964
tests: minor, don't re-write `passed_fork`.
veox May 21, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 31 additions & 11 deletions .circleci/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -131,15 +131,13 @@ jobs:
- image: circleci/python:3.6
environment:
TOXENV: py36-lint

py37-lint:
<<: *common
docker:
- image: circleci/python:3.7
environment:
TOXENV: py37-lint


py36-docs:
<<: *common
docker:
Expand Down Expand Up @@ -171,6 +169,13 @@ jobs:
- image: circleci/python:3.6
environment:
TOXENV: py36-rpc-state-homestead
py36-rpc-state-petersburg:
<<: *common
docker:
- image: circleci/python:3.6
environment:
TOXENV: py36-rpc-state-petersburg

py36-rpc-state-tangerine_whistle:
<<: *common
docker:
Expand Down Expand Up @@ -221,12 +226,6 @@ jobs:
- image: circleci/python:3.6
environment:
TOXENV: py36-p2p
py36-rpc-state-quadratic:
<<: *common
docker:
- image: circleci/python:3.6
environment:
TOXENV: py36-rpc-state-quadratic
py36-eth2-core:
<<: *common
docker:
Expand Down Expand Up @@ -264,6 +263,24 @@ jobs:
environment:
TOXENV: py36-plugins

py37-rpc-state-quadratic:
<<: *common
docker:
- image: circleci/python:3.7
environment:
TOXENV: py37-rpc-state-quadratic
py37-rpc-state-sstore:
<<: *common
docker:
- image: circleci/python:3.7
environment:
TOXENV: py37-rpc-state-sstore
py37-rpc-state-zero_knowledge:
<<: *common
docker:
- image: circleci/python:3.7
environment:
TOXENV: py37-rpc-state-zero_knowledge

py37-core:
<<: *common
Expand Down Expand Up @@ -320,7 +337,6 @@ jobs:
environment:
TOXENV: py37-plugins


docker-image-build-test:
machine: true
steps:
Expand All @@ -343,13 +359,17 @@ workflows:
# - py37-libp2p
- py37-plugins

- py37-rpc-state-quadratic
- py37-rpc-state-sstore
- py37-rpc-state-zero_knowledge

- py36-rpc-state-byzantium
- py36-rpc-state-constantinople
- py36-rpc-state-frontier
- py36-rpc-state-homestead
- py36-rpc-state-tangerine_whistle
- py36-rpc-state-petersburg
- py36-rpc-state-spurious_dragon
- py36-rpc-state-quadratic
- py36-rpc-state-tangerine_whistle
- py36-rpc-blockchain

- py36-core
Expand Down
2 changes: 1 addition & 1 deletion fixtures
Open in desktop
Submodule fixtures updated 23665 files
114 changes: 76 additions & 38 deletions tests/json-fixtures-over-rpc/test_rpc_fixtures.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
is_hex,
is_integer,
is_string,
to_tuple,
)

from eth.chains.mainnet import (
Expand Down Expand Up @@ -47,16 +48,12 @@
BASE_FIXTURE_PATH = os.path.join(ROOT_PROJECT_DIR, 'fixtures', 'BlockchainTests')

SLOW_TESTS = (
'Call1024PreCalls_d0g0v0_Byzantium',
'Call1024PreCalls_d0g0v0_EIP150',
'Call1024PreCalls_d0g0v0_EIP158',
'Call1024PreCalls_d0g0v0',
'ContractCreationSpam_d0g0v0_Homestead',
'ContractCreationSpam_d0g0v0_Frontier',
'ForkStressTest_EIP150',
'ForkStressTest_EIP158',
'ForkStressTest_Homestead',
'ForkStressTest_Frontier',
'ForkStressTest_Byzantium',
'Create2Recursive_d0g0v0',
'Create2Recursive_d0g1v0',
'ForkStressTest',
Copy link
Contributor Author

@veox veox May 16, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test is slow across all forks - however, removal of fork qualifier means that any fixture that has this string will be marked SLOW.

Is this acceptable?

'stQuadraticComplexityTest/Call50000_d0g1v0.json',
'stQuadraticComplexityTest/QuadraticComplexitySolidity_CallDataCopy_d0g1v0.json',
'stQuadraticComplexityTest/Return50000_2_d0g1v0.json',
Expand All @@ -66,39 +63,37 @@
'stQuadraticComplexityTest/Call50000_ecrec_d0g1v0.json',
'walletReorganizeOwners',
'bcExploitTest/SuicideIssue.json',
'DelegateCallSpam_Homestead',
'static_Call50000_sha256_d0g0v0_Byzantium',
'static_Call50000_rip160_d0g0v0_Byzantium',
'static_Call50000_rip160_d1g0v0_Byzantium',
'static_Call50000_sha256_d1g0v0_Byzantium',
'static_Call50000_ecrec_d1g0v0_Byzantium',
'static_Call50000_d1g0v0_Byzantium',
'static_Call50000_d0g0v0_Byzantium',
'static_Call50000_ecrec_d0g0v0_Byzantium',
'static_Call50000_identity2_d0g0v0_Byzantium',
'static_Call50000_identity2_d1g0v0_Byzantium',
'static_Call50000_identity_d1g0v0_Byzantium',
'static_Call50000_identity_d0g0v0_Byzantium',
'static_Call50000bytesContract50_1_d1g0v0_Byzantium',
'static_Call50000bytesContract50_2_d1g0v0_Byzantium',
'static_LoopCallsThenRevert_d0g0v0_Byzantium',
'static_LoopCallsThenRevert_d0g1v0_Byzantium',
'Call1024PreCalls_d0g0v0_Byzantium',
'Call1024PreCalls_d0g0v0_EIP158',
'Call1024PreCalls_d0g0v0_EIP150',
'Call1024PreCalls_d0g0v0_Byzantium',
'Call1024PreCalls_d0g0v0_EIP150',
'Call1024PreCalls_d0g0v0_EIP158',
Copy link
Contributor Author

@veox veox May 16, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The above six lines are duplicates of the first three lines in the list: therefore removed.

'static_Call1024PreCalls_d1g0v0',
'static_Call1024PreCalls2_d0g0v0',
'static_Call1024PreCalls2_d1g0v0',
'static_Call1024PreCalls3_d1g0v0',
'static_Call50000bytesContract50_1_d0g0v0',
'static_Call50000_ecrec_d0g0v0',
'static_Call50000_ecrec_d1g0v0',
'static_Call50000_rip160_d0g0v0',
'static_Call50000_rip160_d1g0v0',
'static_Call50000_sha256_d0g0v0',
'static_Call50000_sha256_d1g0v0',
'static_Call50000_d0g0v0',
'static_Call50000_d1g0v0',
'static_Call50000_identity2_d0g0v0',
'static_Call50000_identity2_d1g0v0',
'static_Call50000_identity_d0g0v0',
'static_Call50000_identity_d1g0v0',
'static_Call50000bytesContract50_1_d1g0v0',
'static_Call50000bytesContract50_2_d1g0v0',
'static_LoopCallsThenRevert_d0g0v0',
'static_LoopCallsThenRevert_d0g1v0',
'static_Return50000_2_d0g0v0',
'stQuadraticComplexityTest/Call50000_identity2_d0g1v0.json',
'stQuadraticComplexityTest/Call50000_identity_d0g1v0.json',
'stQuadraticComplexityTest/Call50000_rip160_d0g1v0.json',
'stQuadraticComplexityTest/Call50000bytesContract50_1_d0g1v0.json',
'stQuadraticComplexityTest/Call50000bytesContract50_2_d0g1v0.json',
'stQuadraticComplexityTest/Create1000_d0g1v0.json',
'ShanghaiLove_Homestead',
'ShanghaiLove_Frontier',
'DelegateCallSpam_EIP158',
'DelegateCallSpam_Byzantium',
'DelegateCallSpam_EIP150',
'DelegateCallSpam',
)

# These are tests that are thought to be incorrect or buggy upstream,
Expand All @@ -112,6 +107,28 @@
# The result is in conflict with the yellow-paper:
# * https://github.com/ethereum/py-evm/pull/1224#issuecomment-418800369
('GeneralStateTests/stRevertTest/RevertInCreateInInit_d0g0v0.json', 'RevertInCreateInInit_d0g0v0_Byzantium'), # noqa: E501
('GeneralStateTests/stRevertTest/RevertInCreateInInit_d0g0v0.json', 'RevertInCreateInInit_d0g0v0_Constantinople'), # noqa: E501
Copy link
Contributor Author

@veox veox May 14, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This INCORRECT_UPSTREAM_TESTS blob is the same as the one in py-evm. It matches 1:1 the particular py-evm version that trinity is pinned to.

Wonder if we should make it more accessible in py-evm so it can be imported here instead of copy-pasting.

Added a TODO comment so this is not forgotten.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you up for making the changes and/or making an issue in py-evm to expose this API?

Copy link
Contributor Author

@veox veox May 17, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll make issues, as you outlined in:

Yeah, to avoid committing the TODO's to master, how about an issue in trinity that links to an issue in py-evm which lists the methods to expose in the API.

('GeneralStateTests/stRevertTest/RevertInCreateInInit_d0g0v0.json', 'RevertInCreateInInit_d0g0v0_ConstantinopleFix'), # noqa: E501

# The CREATE2 variant seems to have been derived from the one above - it, too,
# has a "synthetic" state, on which py-evm flips.
# * https://github.com/ethereum/py-evm/pull/1181#issuecomment-446330609
('GeneralStateTests/stCreate2/RevertInCreateInInitCreate2_d0g0v0.json', 'RevertInCreateInInitCreate2_d0g0v0_Constantinople'), # noqa: E501
('GeneralStateTests/stCreate2/RevertInCreateInInitCreate2_d0g0v0.json', 'RevertInCreateInInitCreate2_d0g0v0_ConstantinopleFix'), # noqa: E501

# Four variants have been specifically added to test a collision type
# like the above; therefore, they fail in the same manner.
# * https://github.com/ethereum/py-evm/pull/1579#issuecomment-446591118
# Interestingly, d2 passes in Constantinople after a py-evm refactor of storage handling,
# the same test is already passing in ConstantinopleFix. Since the situation is synthetic,
# not much research went into why, yet.
('GeneralStateTests/stSStoreTest/InitCollision_d0g0v0.json', 'InitCollision_d0g0v0_Constantinople'), # noqa: E501
('GeneralStateTests/stSStoreTest/InitCollision_d1g0v0.json', 'InitCollision_d1g0v0_Constantinople'), # noqa: E501
('GeneralStateTests/stSStoreTest/InitCollision_d2g0v0.json', 'InitCollision_d2g0v0_Constantinople'), # noqa: E501
('GeneralStateTests/stSStoreTest/InitCollision_d3g0v0.json', 'InitCollision_d3g0v0_Constantinople'), # noqa: E501
('GeneralStateTests/stSStoreTest/InitCollision_d0g0v0.json', 'InitCollision_d0g0v0_ConstantinopleFix'), # noqa: E501
('GeneralStateTests/stSStoreTest/InitCollision_d1g0v0.json', 'InitCollision_d1g0v0_ConstantinopleFix'), # noqa: E501
('GeneralStateTests/stSStoreTest/InitCollision_d3g0v0.json', 'InitCollision_d3g0v0_ConstantinopleFix'), # noqa: E501
}

RPC_STATE_NORMALIZERS = {
Expand Down Expand Up @@ -178,7 +195,7 @@ def fixture_transaction_in_rpc_format(state):
}


def blockchain_fixture_mark_fn(fixture_path, fixture_name):
def blockchain_fixture_mark_fn(fixture_path, fixture_name, fixture_fork):
for slow_test in SLOW_TESTS:
if slow_test in fixture_path or slow_test in fixture_name:
if not should_run_slow_tests():
Expand All @@ -188,13 +205,33 @@ def blockchain_fixture_mark_fn(fixture_path, fixture_name):
return pytest.mark.xfail(reason="Listed in INCORRECT_UPSTREAM_TESTS.")


def generate_ignore_fn_for_fork(passed_fork):
if passed_fork:
normalized_fork = passed_fork.lower()

def ignore_fn(fixture_path, fixture_key, fixture_fork):
return fixture_fork.lower() != normalized_fork

return ignore_fn


@to_tuple
def expand_fixtures_forks(all_fixtures):
for fixture_path, fixture_key in all_fixtures:
fixture = load_fixture(fixture_path, fixture_key)
yield fixture_path, fixture_key, fixture['network']


def pytest_generate_tests(metafunc):
ignore_fn = generate_ignore_fn_for_fork(metafunc.config.getoption('fork'))
generate_fixture_tests(
metafunc=metafunc,
base_fixture_path=BASE_FIXTURE_PATH,
preprocess_fn=expand_fixtures_forks,
filter_fn=filter_fixtures(
fixtures_base_dir=BASE_FIXTURE_PATH,
mark_fn=blockchain_fixture_mark_fn,
ignore_fn=ignore_fn
),
)

Expand Down Expand Up @@ -375,9 +412,10 @@ async def validate_uncles(rpc, block_fixture, at_block):

@pytest.fixture
def chain_fixture(fixture_data):
fixture = load_fixture(*fixture_data)
if fixture['network'] == 'Constantinople':
pytest.skip('Constantinople VM rules not yet supported')
fixture_path, fixture_key, fixture_fork = fixture_data
fixture = load_fixture(fixture_path, fixture_key)
if fixture_fork == 'Istanbul':
pytest.skip('Istanbul VM rules not yet supported')
return fixture


Expand Down
24 changes: 15 additions & 9 deletions tox.ini
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
[tox]
envlist=
py{36,37}-{core,p2p,integration,lightchain_integration,eth2-core,eth2-fixtures,eth2-integration,plugins}
py{36}-long_run_integration
py{36}-rpc-state-{frontier,homestead,tangerine_whistle,spurious_dragon,byzantium,constantinople,quadratic}
py{36}-rpc-blockchain
py36-long_run_integration
py36-rpc-blockchain
py36-rpc-state-{frontier,homestead,tangerine_whistle,spurious_dragon,byzantium,constantinople,petersburg}
py37-rpc-state-{quadratic,sstore,zero_knowledge}
py{36,37}-libp2p
py{36,37}-lint
py{36,37}-wheel-cli
Expand All @@ -26,14 +27,19 @@ commands=
p2p: pytest -n 4 {posargs:tests/p2p}
plugins: pytest -n 4 {posargs:tests/plugins/}
rpc-blockchain: pytest -n 4 {posargs:tests/json-fixtures-over-rpc/test_rpc_fixtures.py -k 'not GeneralStateTests'}
rpc-state-frontier: pytest -n 4 {posargs:tests/json-fixtures-over-rpc/test_rpc_fixtures.py -k 'GeneralStateTests and not stQuadraticComplexityTest and Frontier'}
rpc-state-homestead: pytest -n 4 {posargs:tests/json-fixtures-over-rpc/test_rpc_fixtures.py -k 'GeneralStateTests and not stQuadraticComplexityTest and Homestead'}
rpc-state-tangerine_whistle: pytest -n 4 {posargs:tests/json-fixtures-over-rpc/test_rpc_fixtures.py -k 'GeneralStateTests and not stQuadraticComplexityTest and EIP150'}
rpc-state-spurious_dragon: pytest -n 4 {posargs:tests/json-fixtures-over-rpc/test_rpc_fixtures.py -k 'GeneralStateTests and not stQuadraticComplexityTest and EIP158'}
# Fork/VM-specific state transition tests; long-running categories run separately!
rpc-state-frontier: pytest -n 4 {posargs:tests/json-fixtures-over-rpc/test_rpc_fixtures.py --fork Frontier -k 'GeneralStateTests and not stQuadraticComplexityTest and not stSStoreTest and not stZeroKnowledge'}
rpc-state-homestead: pytest -n 4 {posargs:tests/json-fixtures-over-rpc/test_rpc_fixtures.py --fork Homestead -k 'GeneralStateTests and not stQuadraticComplexityTest and not stSStoreTest and not stZeroKnowledge'}
rpc-state-tangerine_whistle: pytest -n 4 {posargs:tests/json-fixtures-over-rpc/test_rpc_fixtures.py --fork EIP150 -k 'GeneralStateTests and not stQuadraticComplexityTest and not stSStoreTest and not stZeroKnowledge'}
rpc-state-spurious_dragon: pytest -n 4 {posargs:tests/json-fixtures-over-rpc/test_rpc_fixtures.py --fork EIP158 -k 'GeneralStateTests and not stQuadraticComplexityTest and not stSStoreTest and not stZeroKnowledge'}
# The following test seems to consume a lot of memory. Restricting to 3 processes reduces crashes
rpc-state-byzantium: pytest -n 3 {posargs:tests/json-fixtures-over-rpc/test_rpc_fixtures.py -k 'GeneralStateTests and not stQuadraticComplexityTest and Byzantium'}
rpc-state-constantinople: pytest -n 3 {posargs:tests/json-fixtures-over-rpc/test_rpc_fixtures.py -k 'GeneralStateTests and not stQuadraticComplexityTest and Constantinople'}
rpc-state-byzantium: pytest -n 3 {posargs:tests/json-fixtures-over-rpc/test_rpc_fixtures.py --fork Byzantium -k 'GeneralStateTests and not stQuadraticComplexityTest and not stSStoreTest and not stZeroKnowledge'}
rpc-state-constantinople: pytest -n 3 {posargs:tests/json-fixtures-over-rpc/test_rpc_fixtures.py --fork Constantinople -k 'GeneralStateTests and not stQuadraticComplexityTest and not stSStoreTest and not stZeroKnowledge'}
rpc-state-petersburg: pytest -n 3 {posargs:tests/json-fixtures-over-rpc/test_rpc_fixtures.py --fork ConstantinopleFix -k 'GeneralStateTests and not stQuadraticComplexityTest and not stSStoreTest and not stZeroKnowledge'}
# Long-running categories.
rpc-state-quadratic: pytest -n 4 {posargs:tests/json-fixtures-over-rpc/test_rpc_fixtures.py -k 'GeneralStateTests and stQuadraticComplexityTest'}
rpc-state-sstore: pytest -n 4 {posargs:tests/json-fixtures-over-rpc/test_rpc_fixtures.py -k 'GeneralStateTests and stSStoreTest'}
rpc-state-zero_knowledge: pytest -n 4 {posargs:tests/json-fixtures-over-rpc/test_rpc_fixtures.py -k 'GeneralStateTests and stZeroKnowledge'}
lightchain_integration: pytest --integration {posargs:tests/integration/test_lightchain_integration.py}

deps = .[p2p,trinity,eth2,test]
Expand Down