Strategy for Tests

[mitsuhiko]

Rye has basically no tests and that is becoming a growing issue. I have originally tried to build something on insta-cmd but that alone is not good enough as there is just too much that needs to be built around it.

In particular it's quite dependent on the state of the package index so it might be wise to serve up a dummy index with well known packages and metadata.

[cnpryer]

I looked around a bit.

Maybe we can put together some resources to get started. For #265 I'd want a bunch of "mock" projects modeled after popular Python projects that still use setup.py or setup.cfg. I'd add that to the PR if you'd like.

For tests with a package index we could have distribution files for different repository types. So legacy, pypi-latest, etc. I haven't built a testsuite that uses a dummy package index, but it'd be nice to just serve it with those files for any tests.

It doesn't have to just be popular projects we have testing for. The same testsuite structure could be used for any kind of project/repository target.

[bluss]

Given #346 it seems even something super simple that does not use pypi would be valuable. test rye init, setup local projects, workspace, rye add path dependencies and sync, then assert certain files exist or assert rye show understands the workspace correctly.

[mitsuhiko]

I want to give this some priority. I think for now I will focus mostly on getting it running in CI somehow. That is

1. use RYE_TOOLCHAIN to register the toolchain from CI for tests
2. point RYE_HOME to a temporary folder
3. make basic assertions of it's outputs, with string replace on the temporary folder before taking snapshots

I ran into some issues that the output of pip and tools it invokes are highly erratic and non snapshottable, maybe the solution would be for now to mostly snapshot in quiet mode.

[anti-social]

In one project we used bats to test CLI tool which worked a lot with files. An example how our tests look like: tests

IMHO bats tests look pretty nice and concise.

[anti-social]

This is how CLI tests could look for rye: rye tests

If you feel ok with such approach I could complete a PR.

[cnpryer]

I want to give this some priority. I think for now I will focus mostly on getting it running in CI somehow. That is

1. use `RYE_TOOLCHAIN` to register the toolchain from CI for tests

2. point `RYE_HOME` to a temporary folder

3. make basic assertions of it's outputs, with string replace on the temporary folder before taking snapshots

I ran into some issues that the output of pip and tools it invokes are highly erratic and non snapshottable, maybe the solution would be for now to mostly snapshot in quiet mode.

Is this still what you're thinking?

[mitsuhiko]

I believe so, yes.