Provide a way to uniquely identify a test?

[matklad]

Currently Cargo + test harness allow to easily identify a test inside the crate by specifying a :: separated path. However this does not uniquely identify a test, because it may live in different crates.

Today, cargo provides --lib, --bin name, --example name etc flags to select a test binary. I think this is a bit suboptimal:

• there are different flags to do essentially the same
• this is not machine friendly: we'd needed a bit of boilerplate in IntelliJ Rust to sort the flags out (and the initial version contained a bug, because I didn't noticed that you must not provide a package name with --lib)
• this is not UI friendly, it's difficult to provide a test name which can be just copy pasted into the shell
• it is not clear how this should be extended to workspaces, where you'd need to identify a package as well.

So perhaps we need to invent some kind of fully-qualified test name? Something along the lines of ::package_name::target_withing_a_package::harness_specific_name?

[alexcrichton]

I'd be fine doing this, yeah, but wouldn't it amount to basically the same thing? You'd basically end up doing:

cargo test --test ::bin::bin_name::argument_to_the_binar

Cargo doesn't actually know anything about the names of tests (they're all part of the test harness, which Cargo is opaque to), so it may be somewhat difficult to pass that kind of information down or parse it out as well.

[matklad]

but wouldn't it amount to basically the same thing?

Yeah, this is exactly the idea: consolidate different pieces of information about test location into the single string. It just doesn't feel right to me that you need more than one flag to specify the test. But the main point of the issue is to raise a concern that running a single test is a bit difficult at the moment, and I sincerely don't know what is the right solution here, the test harness and cargo are indeed independent :)

When workspaces are fully implemented, how one would run a single test from some target from some package of a workspace?

[alexcrichton]

I guess yeah we could benefit from baking this into Cargo itself. I'd suspect that IDEs could have their own layer and the in the translation step of "take this string and call Cargo" it'd be hairy, but not too bad logic. The downside there though is each IDE may make its own syntax, and that'd be bad!

Right now the idea to run tests anywhere would be:

cargo test -p package --test test-name test-argument

[alexcrichton]

Note that Cargo being so disconnected from the test harness has come up repeatedly in many many issues, so it may just be high time to wed the two together.

[jonhoo]

Another issue that I believe could be filed under this one is that it's currently impossible to uniquely identify a test whose name is a prefix of another test. For example, if I have the tests test_foo and test_foo_with_bar, there is no way for me to tell Cargo to only run test_foo. cargo test test_foo will run both. Unless I'm missing something?

In Go, this is solved by making the test name pattern a regex. Perhaps something similar could be done for Cargo?

[alexcrichton]

@jonhoo that's a feature of the test harness, not cargo itself. More recent versions of Rust have a --exact flag as well for that.

[jonhoo]

Well, yes and no — it's necessary for the test harness to support such a feature, but Cargo should also expose it for it to be useful. You're right that something like regex matching would need to be added to rustc first for Cargo to support it, but the ability to do exact matching should at least be made available directly through cargo test, no?

[matklad]

should at least be made available directly through cargo test, no?

It is available. You can pass arguments directly to the test harness after --:

cargo test --test foo::bar -- --exact

[jonhoo]

@matklad that seems like a rather roundabout way of getting at a fairly commonplace feature..?

[dwijnand]

@jonhoo

that seems like a rather roundabout way of getting at a fairly commonplace feature..?

I'd argue the more commonplace feature is just being able to somewhat filter down the tests, for which cargo has explicit support for: https://github.com/rust-lang/cargo/blob/0.33.0/src/bin/cargo/commands/test.rs#L126

@alexcrichton

Note that Cargo being so disconnected from the test harness has come up repeatedly in many many issues, so it may just be high time to wed the two together.

How high time is it, compared to the competing priorities for 2019-2021? Should we add it to next meeting's agenda?

@matklad

So perhaps we need to invent some kind of fully-qualified test name? Something along the lines of ::package_name::target_withing_a_package::harness_specific_name?

I prefer cargo test -p package_name --test target_withing_a_package harness_specific_name to that.

I was about to mention that we've started discussing the extraction of some stable reusable modules for cargo (first one being discussed in oli-obk/cargo_metadata#63), but I just realise that doesn't help you in IntelliJ Rust, seeing as it's in Kotlin.

[dwijnand]

Btw, if anyone wants to join me, I've kind of being on a little bit of a campain for a feature in libtest: rust-lang/rust#30422

[matklad]

FWIW, I am inclined to close this:

UI-wise, I think -p, --lib/bin/expample is pretty good, despite (or even because) being non-uniform: you get completions for -p, for example (at least in IntelliJ), and it's easier for a human to parse due to verbose syntax.

As for machine friendlies, yeah, it's quite a boilerplate to specify all the various flags, but you write it once and forget about it.