refactor: split integration tests from CLI (part 1) #22308
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mmastrac
force-pushed
the
test_code_reorg
branch
from
0d69bbb to
a417772
Compare
mmastrac
changed the title
mmastrac
force-pushed
the
test_code_reorg
branch
from
a8d4c3b to
91f8787
Compare
mmastrac
force-pushed
the
test_code_reorg
branch
from
fc7df72 to
e08e324
Compare
littledivy
pushed a commit
that referenced
this pull request
Feb 15, 2024
This PR separates integration tests from CLI tests into a new project named `cli_tests`. This is a prerequisite for an integration test runner that can work with either the CLI binary in the current project, or one that is built ahead of time. ## Background Rust does not have the concept of artifact dependencies yet (rust-lang/cargo#9096). Because of this, the only way we can ensure a binary is built before running associated tests is by hanging tests off the crate with the binary itself. Unfortunately this means that to run those tests, you _must_ build the binary and in the case of the deno executable that might be a 10 minute wait in release mode. ## Implementation To allow for tests to run with and without the requirement that the binary is up-to-date, we split the integration tests into a project of their own. As these tests would not require the binary to build itself before being run as-is, we add a stub integration `[[test]]` target in the `cli` project that invokes these tests using `cargo test`. The stub test runner we add has `harness = false` so that we can get access to a `main` function. This `main` function's sole job is to `execvp` the command `cargo test -p deno_cli`, effectively "calling" another cargo target. This ensures that the deno executable is always correctly rebuilt before running the stub test runner from `cli`, and gets us closer to be able to run the entire integration test suite on arbitrary deno executables (and therefore split the build into multiple phases). The new `cli_tests` project lives within `cli` to avoid a large PR. In later PRs, the test data will be split from the `cli` project. As there are a few thousand files, it'll be better to do this as a completely separate PR to avoid noise.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR separates integration tests from CLI tests into a new project named
cli_tests. This is a prerequisite for an integration test runner that can work with either the CLI binary in the current project, or one that is built ahead of time.Background
Rust does not have the concept of artifact dependencies yet (rust-lang/cargo#9096). Because of this, the only way we can ensure a binary is built before running associated tests is by hanging tests off the crate with the binary itself.
Unfortunately this means that to run those tests, you must build the binary and in the case of the deno executable that might be a 10 minute wait in release mode.
Implementation
To allow for tests to run with and without the requirement that the binary is up-to-date, we split the integration tests into a project of their own. As these tests would not require the binary to build itself before being run as-is, we add a stub integration
[[test]]target in thecliproject that invokes these tests usingcargo test.The stub test runner we add has
harness = falseso that we can get access to amainfunction. Thismainfunction's sole job is toexecvpthe commandcargo test -p deno_cli, effectively "calling" another cargo target.This ensures that the deno executable is always correctly rebuilt before running the stub test runner from
cli, and gets us closer to be able to run the entire integration test suite on arbitrary deno executables (and therefore split the build into multiple phases).The new
cli_testsproject lives withinclito avoid a large PR. In later PRs, the test data will be split from thecliproject. As there are a few thousand files, it'll be better to do this as a completely separate PR to avoid noise.