Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

moving test selection to pytest markers #41

Merged
merged 5 commits into from
Jan 13, 2018
Merged

moving test selection to pytest markers #41

merged 5 commits into from
Jan 13, 2018

Conversation

mattsb42-aws
Copy link
Member

This change moves our test selection logic from using environment variables to filter out tests run in a large batch to instead using pytest markers to select specific tests. In addition to being simpler to maintain, this both allows each test environment to run fewer tests and makes it much clearer in our CI which types of tests fail (ex: if credentials are not found, only integration, acceptance, and examples tests will fail).

This adds six markers:

  1. local: superset of unit and functional (does not require network access)
  2. unit: mark test as a unit test (does not require network access)
  3. functional: mark test as a functional test (does not require network access)
  4. integ: mark a test as an integration test (requires network access)
  5. accept: mark a test as an acceptance test (requires network access)
  6. examples: mark a test as an examples test (requires network access)

Validating that all tests are caught, the counts for tests run and skipped when filtering for each marker:

marker run skipped total
all 2151 0 2151
local 861 1290 2151
unit 598 1553 2151
functional 263 1888 2151
integ 26 2125 2151
accept 1260 891 2151
examples 4 2147 2151
local = unit + functional
861   = 598  + 263

all  = local + integ + accept + examples
2151 = 861   + 26    + 1260   + 4

@mattsb42-aws mattsb42-aws requested a review from a team January 13, 2018 00:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants