You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I like the idea but not sure how to measure success. Without a fully conformant server we can measure how many tests fail. In the example where we had syntax wrong in the Karate, the test just failed in the same way as a non conforming server. If the harness can detect Karate failures in a different way this would work well so I'll investigate. I'll put an issue in that repo.
Would it be possible to basically do what I do on the command line now: I use the --filter option to run only the tests modified by the PR?
That is the most useful thing here, just to see if a certain PR is sane. If all implementations pass, then that's very useful to know, because then we can fast-track that PR to approved state. If all fail, then it is likely something wrong with the test or the harness, or some contentious stuff in the spec, which is also good to know. If some implementations pass, it is also interesting, because then the reviewer has more to go on while they review.
If we get large feature files that are modified and tests that fail for longer periods of time on this, the usefulness of this might diminish, but I think that is likely to be not so much of a problem in the short term, and hopeful not in the long term either.
Do you think it would be possible to have a CI running on PRs to check that the tests can actually run, so that we don't merge broken tests to main?
The text was updated successfully, but these errors were encountered: