[ignoreme] experiment with go test -json
#30406
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This change is |
tbg
force-pushed
the
testing/failsuite
branch
from
0f70c00 to
10c7514
Compare
tbg
added a commit
to tbg/go-test-teamcity
that referenced
this pull request
Sep 19, 2018
This executes on the plan outlined in: https://github.com/cockroachdb/go-test-teamcity/issues/5#issuecomment-422494665 by making go-test-teamcity take test2json input. More precisely, and unfortunately, it realistically has to be taking the input of make test TESTFLAGS='-v -json' which isn't pure JSON but contains a bunch of `make` and `go build` output (requiring a bit of massaging to pluck the relevant JSON from the input stream). I retained the old functionality and introduced a new `-json` flag that triggers the new code. This important because our CI scripts are versioned, and so we need the updated code here to work with the non-updated CI pipeline. I initially tried to jsonify the old test cases, but they're basically garbage and don't correspond to real test output in some cases. Instead, I generated a new corpus from actual CI runs in the main repo from which I copied the artifacts file: cockroachdb/cockroach#30406 I think it's worth checking in that PR in suitable form because it'll allow us to iterate on this parser and extend the corpus much more easily in the future. Two caveats: 1. the code isn't creating TeamCity Test Suite tags. I was never married to those and have found it unfortunate that they mess up TeamCity's test duration counting (as far as I can remember). Adding them is probably possible, but with parallel tests (which we don't use heavily) and such I'm pretty sure that any naive implementation would be incorrect. 2. Tests which aren't explicitly ended (due to premature exit/crash of the package test binary) are now output at the very end of the log. Since the input contains *all tests*, this can be a little unintuitive. However, these final tests contain basically no information anyway, so that seems fine. This could be improved by flushing out the tests whenever a package-level PASS/FAIL event is encountered, but I'm not sure it's worth doing so. The next step, assuming the code is in good order, is to merge it and update the CI pipeline in the main repo to pass the `-json` flags both to `go test` and `go-test-teamcity`.
This package serves to make it easy to improve test coverage for go-test-teamcity in the future as problems arise, by opening a PR that edits the CI scripts to include the COCKROACH_FAILSUITE environment variable, triggering various interesting test failures and data races in these newly introduced packages. The created artifacts can then be incorporated into the test suite for go-test-teamcity (as was already done in cockroachdb/go-test-teamcity#9). Release note: None
Release note: None
tbg
force-pushed
the
testing/failsuite
branch
from
3baa3d9 to
3dd0b69
Compare
|
@tbg this is dead right? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
See https://github.com/cockroachdb/go-test-teamcity/issues/5 for context.