proposal: cmd/go: go test: allow -run and -skip flags to specify a package

[KevinFairise2]

Proposal Details

This proposal was initially mentioned in the issue for the addition of the -skip flag: #41583 (comment)

It would be very convenient to be able to specify the package in which the test lives when skipping or running a test. The command would look like that: go test -run "path/to/pkgA.TestToto" ./... to run only the test TestToto that lives in pkgA or go test -skip "path/to/pkgA.TestToto" ./... to skip only this test. We would keep the default behavior if no packages are specified

The current implementation has some limits because you cannot easily deal with tests that have the same name across packages. For example if I have a test named TestToto in pkgA and a test with the same name in pkgB. I cannot skip only TestToto in pkgA with the command go test -skip TestToto ./...

It would be useful for us to skip only some flaky tests in specific packages.

[gabyhelp]

Related Issues and Documentation

• cmd/go: add go test -skip to skip specific tests #41583 (closed)
• proposal: cmd/go: add "go test -compile" option to just compile tests, not link or run them #46712 (closed)
• proposal: testing: add a flag to detect unnecessary skips #25951
• proposal: cmd/go: change go test to FAIL if `-run` is specified but no tests match #64500
• cmd/go: go test quietly skips test if custom flag is defined #36527 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[rittneje]

How would it distinguish package paths from subtests? Also how specifically would this interact with existing support for arbitrary regexes?

[KevinFairise2]

I think -run and -skip flags could accept a string with the following format path/to/pkg.Test(Toto|Tata)/Subtest/Subsubtest
The . would be separator between the pkg we want to target and the test name.
The substring before the point should be a package path.
Everything after the point should have the same format as the currently accepted string for -skip and -run, that mean we'd keep the support for arbitrary regexes and subtests using /.
If the string contains no . then the behavior of the flags does not change.

[rittneje]

But . currently means "any character" since it is a regex. Plus a literal . could legally appear in a sub-test name. Thus I don't think we can use its presence alone to signal the new semantics.

[matloob]

I think the issues @rittneje mentioned make this complicated to address.

I think we haven't really needed this because it's not common to have many different packages with tests that are named the same. I'm curious what your use case looks like

[pducolin]

We do have several tests currently with the same name across different packages in our e2e tests module. One package corresponds to one feature, one test corresponds to one target remote infrastructure - so we have TestVM, TestKind etc across different features.

I agree we should rather rename tests to be unique, thinking about having a custom linter that ensure no homonym across this module specifically.

[matloob]

@pducolin Thanks for the example. Have you needed to run or skip tests in packages in a way that can't be done using the currently available options?

[gopherbot]

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

[pducolin]

@matloob we do have a use case where we need to skip a single test in packages that contain homonyms, in our e2e tests module.

As an example we have

• TestVMSuite in secagentfunctional package
• TestVMSuite in sysprobefunctional package

Both are executed within the same job on the CI.

We have an internal tool to mark a single test as flaky, either in code or in a file. We want to ensure the flakes marker behaviour is the same. This allows to skip a known flaky test on a dev pipeline. It still runs on main pipelines.

Today the in code marker allows to skip a single test context

We don't have the same support with the file based marker, as we cannot isolate a test in calls to go test within a module based on their path or package.

We need the file based marker as some test names are dynamic