proposal: cmd/go: change go test to FAIL if -run is specified but no tests match

[hyangah]

Proposal Details

If -run is provided but there is no test matching the pattern, go test currently considers the test run PASS and exits with 0.

$ go test -run=Test_sum/1_+_1
testing: warning: no tests to run
PASS
ok      example/test    0.133s

Even though there is a warning like "testing: warning: no tests to run", this is easy to miss or dropped when the test binary runs by other runners (ide plugins, dlv, ..). I think when users set -run and if there is no matching test, that's likely not what users wanted, and treating it like misuse of command is more appropriate.

[dmitshur]

CC @bcmills, @matloob.

Slightly related, but in the opposite direction: go test and go vet currently print a warning and also exit with a fatal error when an import path pattern matches 0 packages (#31271, #52287), which can create overhead for a CI system wishing to permit modules that are empty or have packages for a subset of Go's ports, since they have no choice but need to convert such false-positive failures to successes on their own.

[bcmills]

• See also cmd/go: test with -run /something prints "no tests to run" inconsistently #48851.

[navytux]

For the reference: I'm using go test -run ø -bench . to only run benchmarks without running tests. If -run ø is changed to result in FAIL, please add another way to only run the benchmarks. Thanks.

[dmitshur]

@navytux That's a good point. In the original proposal here, there was only the -run flag specified, without -bench, so it likely shouldn't apply if something else, like benchmarks, does match and run.

As a side note, there is a possibility of that invocation matching some tests if they happen to have that character in their name. Since -run flag accepts a regular expression, consider doing go test -run=^$ -bench=. to ensure only benchmarks are run. The "^$" value is fairly explicit about the intention to match no tests/examples/fuzz tests.

[navytux]

@dmitshur, thanks for feedback. And I agree that ^$ is generally more strict and correct in limiting the set of allowed tests to run to be empty.

[rsc]

Will add to proposal minutes next week, but I don't see how to do this in a defensible way.

• It seems very strange to me that go test -run=XXX should fail but go test -run=XXX -bench=. should succeed.
• It also seems strange to say that go test -run=XXX should fail but go test -run=^$ should succeed.
• And what if I do go test -run=Foo -skip=Foo? Should that fail too?
• And right now go test notests succeeds if notests has no tests (and compiles and passes vet). Should it start failing? If not, what if I use go test notests -run=.*? Should that start failing?

All these strange contradictions go away if go test -run=XXX continues to succeed as it does today but also continues to print a message about not having any tests to run.

The original design of this warning in #15211 explicitly left the exit code unchanged to avoid all these problems. I think we should continue to leave it unchanged.

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[aclements]

Given the tricky contradictions in this proposal, I don't think we can do this, unfortunately. If someone finds a way to thread that needle, we can of course revisit. :)

[aclements]

Based on the discussion above, this proposal seems like a likely decline.

[aclements]

No change in consensus, so declined.