How to catch dev dependencies that actually need to be included in the distributed presentations of Meltano?

[edgarrmondragon]

Just got bit by this with packaging in #6276:

It's a transitive dependency required by tox, but tox is only a dev dependency. So, in CI tests, things look good because dev dependencies are installed by poetry install. But, when meltano is installed from an upstream source or by another tool like pipx, which is the case in end-to-end tests here (pipx install ..), then the transitive dev dependency packaging is not installed and errors are encountered: https://gitlab.com/meltano/legacy-ci/meltano/-/jobs/2632994539#L1237.

$ meltano add orchestrator airflow
2022-06-23T19:10:30.061252Z [info     ] Environment 'dev' is active
No module named 'packaging'

I guess one easy way to ensure we don't fall in that scenario is to have tests that don't rely on Poetry to install Meltano.

[z3z1ma]

Export the lockfile from from poetry to pin deps in a requirements.txt as part of the release process? 🤔

[edgarrmondragon]

Export the lockfile from from poetry to pin deps in a requirements.txt as part of the release process? 🤔

@z3z1ma Yeah, I agree that poetry export is the way to go but it probably has to be done only during our testing stage, otherwise we'd have to do it with poetry export --dev for sdist and wheels, which is not ideal because that would include all dev dependencies.

I particularly like the nox + poetry export --dev approach where the exported requirements.txt is used as a pip constraints file to install only dev dependencies specifically required by the dev task.

[aaronsteers]

@pandemicsyn and @edgarrmondragon - What do you think of this being a part of

• New process and guidelines for integration tests (replaces demo-project and homepage_sample_code tests in GitLab CI #6218?

At least when running in CI, we could use a pipx-installed version of Meltano instead of the Poetry-installed one.

[pandemicsyn]

At least when running in CI, we could use a pipx-installed version of Meltano instead of the Poetry-installed one.

Probably a better discussion for #6218, but sounds fine - at some point we'd probably want a test mirroring the contributor guide which might end up using poetry.

[edgarrmondragon]

The SDK had this exact issue in the v0.6.0 release so I went ahead and started a PR with nox-poetry, which runs tasks against the installed form of the package using the wheel.

meltano/sdk#780