Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(plugins/forks): Fix covariant markers, add selector/marks parameters #762

Merged
merged 14 commits into from
Aug 30, 2024

Conversation

marioevz
Copy link
Member

🗒️ Description

  • Fix case when the fork method returns a list or tuple and the marker selector only allows a single parameter.
  • Disallow positional parameters on all with_all markers
  • Fix parametrization when combining with_all markers that share a common parameter, e.g. with_all_call_opcodes and with_all_create_opcodes share evm_code_type, therefore when used together, only test cases where the evm_code_type is compatible with each marker (e.g Op.CALL + Op.CREATE or Op.EXTCALL + Op.EOFCREATE, but not Op.EXTCALL + Op.CREATE)
  • Add support for markers by allowing marks and marks_selector keyword parameter in all with_all markers, this method has the disadvantage that the same set of markers needs to be used in all test cases where the marker is applied.

🔗 Related Issues

None

✅ Checklist

  • All: Set appropriate labels for the changes.
  • All: Considered squashing commits to improve commit history.
  • All: Added an entry to CHANGELOG.md.
  • All: Considered updating the online docs in the ./docs/ directory.
  • Tests: All converted JSON/YML tests from ethereum/tests have been added to converted-ethereum-tests.txt.
  • Tests: A PR with removal of converted JSON/YML tests from ethereum/tests have been opened.
  • Tests: Included the type and version of evm t8n tool used to locally execute test cases: e.g., ref with commit hash or geth 1.13.1-stable-3f40e65.
  • Tests: Ran mkdocs serve locally and verified the auto-generated docs for new tests in the Test Case Reference are correctly formatted.

@marioevz marioevz added type:feat type: Feature scope:fw Scope: Framework (evm|tools|forks|pytest) labels Aug 23, 2024
@marioevz marioevz requested a review from danceratopz August 23, 2024 21:56
Copy link
Member

@danceratopz danceratopz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey LGTM! Few small comments below. It's pretty heavy stuff 😆 but 💯 worth the overhead!

I like it (and think we can merge this as-is), but I'm not entirely sure we have the right interface for applying marks yet (these don't seem to used anywhere...yet).

For example, it's entirely possible to want to apply multiple marks to multiple parameters and, if I understand correctly, it's currently only possible to apply multiple marks to one of the parameters?

What I mean is something like either:

  1.  @pytest.mark.with_all_tx_types(
                 marks=[[pytest.mark.xfail], [pytest.mark.skip]],
                 marks_selector=[lambda tx_type: tx_type == 1, lambda tx_type: tx_type == 2],
             )
    or,
  2.  def tx_types_marks(tx_type: int):
         if tx_type == 1:
             return [pytest.mark.xfail, pytest.mark.slow]
         if tx_type == 2:
             return [pytest.mark.skipif(...)]
         return []
     @pytest.mark.with_all_tx_types(marks=tx_types_marks)
    i.e., Allow the user to define a function that returns the marks to be applied for each param.

But neither of these solve the case for marking with_all_txtypes and with_all_precompiles and wanting to mark if tx_type == 2 and precompile == 1? 😆

I pushed some additions to the tests, which I mainly made to help me get into the topic. In particular, I added a couple of test cases for marks and marks_selector cf f17a383.

Also, not necessary for this PR, but what do you think about moving these fork helper functions get_from_until_fork_set, get_forks_with_no_parents, get_forks_with_no_descendants and get_last_descendants from pytest_plugins.forks.forks to ethereum_test_forks.helpers? Or do you consider them too specific?

src/pytest_plugins/forks/forks.py Outdated Show resolved Hide resolved
src/pytest_plugins/forks/forks.py Outdated Show resolved Hide resolved
src/pytest_plugins/forks/forks.py Outdated Show resolved Hide resolved
@marioevz
Copy link
Member Author

2. ```python
    def tx_types_marks(tx_type: int):
        if tx_type == 1:
            return [pytest.mark.xfail, pytest.mark.slow]
        if tx_type == 2:
            return [pytest.mark.skipif(...)]
        return []
    @pytest.mark.with_all_tx_types(marks=tx_types_marks)
   ```
   i.e., Allow the user to define a function that returns the marks to be applied for each param.

I really like number 2, I think it's better than the selector implemented in this PR. I'm gonna give this a try and see if I can implement it in this PR.

But neither of these solve the case for marking with_all_txtypes and with_all_precompiles and wanting to mark if tx_type == 2 and precompile == 1? 😆

We'll think about something when we need this :P

Also, not necessary for this PR, but what do you think about moving these fork helper functions get_from_until_fork_set, get_forks_with_no_parents, get_forks_with_no_descendants and get_last_descendants from pytest_plugins.forks.forks to ethereum_test_forks.helpers? Or do you consider them too specific?

Sounds good, will do it in this PR 👍

@marioevz marioevz force-pushed the fix-covariant-marker branch from 7f85321 to e8eedeb Compare August 30, 2024 16:47
@marioevz marioevz changed the title feat(plugins/forks): Fix covariant markers feat(plugins/forks): Fix covariant markers, add selector/marks parameters Aug 30, 2024
@marioevz marioevz merged commit 0a42a18 into main Aug 30, 2024
7 checks passed
@marioevz marioevz deleted the fix-covariant-marker branch August 30, 2024 17:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
scope:fw Scope: Framework (evm|tools|forks|pytest) type:feat type: Feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants