Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI testing if tests run #8

Open
kjetilk opened this issue Aug 5, 2021 · 2 comments
Open

CI testing if tests run #8

kjetilk opened this issue Aug 5, 2021 · 2 comments
Assignees

Comments

@kjetilk
Copy link
Collaborator

kjetilk commented Aug 5, 2021

Do you think it would be possible to have a CI running on PRs to check that the tests can actually run, so that we don't merge broken tests to main?

@edwardsph
Copy link
Collaborator

I like the idea but not sure how to measure success. Without a fully conformant server we can measure how many tests fail. In the example where we had syntax wrong in the Karate, the test just failed in the same way as a non conforming server. If the harness can detect Karate failures in a different way this would work well so I'll investigate. I'll put an issue in that repo.

@kjetilk
Copy link
Collaborator Author

kjetilk commented Dec 30, 2021

Would it be possible to basically do what I do on the command line now: I use the --filter option to run only the tests modified by the PR?

That is the most useful thing here, just to see if a certain PR is sane. If all implementations pass, then that's very useful to know, because then we can fast-track that PR to approved state. If all fail, then it is likely something wrong with the test or the harness, or some contentious stuff in the spec, which is also good to know. If some implementations pass, it is also interesting, because then the reviewer has more to go on while they review.

If we get large feature files that are modified and tests that fail for longer periods of time on this, the usefulness of this might diminish, but I think that is likely to be not so much of a problem in the short term, and hopeful not in the long term either.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants