runTestWithVersions() in CONTRIBUTING.md #362
Conversation
There was a problem hiding this comment.
I think that a lot of the confusion in tests generally comes from all the abstraction layers which lead to us hard-coding versions we test against, instead of managing versions entirely outside of tests and skipping any tests which we know aren't relevant to run for the given version.
TL;DR I think we should just drop runTestWithVersions entirely - but of course that won't be a single LOC change 😅
Having it correctly documented in the meantime makes sense!
There was a problem hiding this comment.
Ah, I missed these in #359! 🙌🏻
To @radeksimko's point, I briefly went down a rabbit hole of trying to write a helper that would allow specifying a version constraint instead of a list of versions, so someone could express something like "run this test against minor versions greater than 1.0" or "run this test on all minor versions between 0.12 and 0.15" without listing out the versions manually. Maybe it would be an interesting path to explore in the future?
The
runTestVersions()function has been effectively renamed torunTestWithVersions()and its behaviour modified by #354 (e2etest/util_test.go) and #359 (e2etest/util_test.go) since the contributing docs were written.Is CONTRIBUTING.md still correct?