-
Notifications
You must be signed in to change notification settings - Fork 105
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Configure private PyPi repositories in the environment #471
Changes from all commits
34a5b62
134bd76
84d13d8
90981e6
57391f3
effce6b
6b0a631
052b256
4b74246
c2083e5
665e8c8
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -154,13 +154,28 @@ The default category is `main`. | |
[environment.yml][envyaml], using a vendored copy of [Poetry's][poetry] dependency solver. | ||
|
||
### private pip repositories | ||
Right now `conda-lock` only supports [legacy](https://warehouse.pypa.io/api-reference/legacy.html) pypi repos with basic auth. Most self-hosted repositories like Nexus, Artifactory etc. use this. To use this feature, add your private repo into Poetry's config _including_ the basic auth in the url: | ||
Right now `conda-lock` only supports [legacy](https://warehouse.pypa.io/api-reference/legacy.html) pypi repos with basic auth. Most self-hosted repositories like Nexus, Artifactory etc. use this. You can configure private repositories by setting the `CONDA_LOCK_PYPI_REPOSITORY_<custom-repository>` environment variable to the URL of the private repo, _including_ any basic auth. For example: | ||
|
||
```bash | ||
poetry config repositories.foo https://username:password@foo.repo/simple/ | ||
export CONDA_LOCK_PYPI_REPOSITORY_EXAMPLE="https://username:password@example.repo/simple" | ||
``` | ||
|
||
The private repo will be used in addition to `pypi.org`. For projects using `pyproject.toml`, it is possible to [disable `pypi.org` entirely](#disabling-pypiorg). | ||
Alternatively, you can use the `poetry` configuration file format to configure private PyPi repositories. The configuration file should be named `config.toml` and have the following format: | ||
|
||
```toml | ||
[repositories.example] | ||
url = "https://username:[email protected]/simple" | ||
``` | ||
|
||
The discoverable location of this file depends on the operating system: | ||
|
||
- **Linux**: `$XDG_CONFIG_HOME/pypoetry-conda-lock` or `~/.config/pypoetry-conda-lock` | ||
- **OSX**: `~/Library/Application Support/pypoetry-conda-lock` | ||
- **Windows**: Check output of `python -c 'from conda_lock._vendor.poetry.locations import CONFIG_DIR;print(CONFIG_DIR)'` | ||
|
||
|
||
Private repositories will be used in addition to `pypi.org`. For projects using `pyproject.toml`, it is possible to [disable `pypi.org` entirely](#disabling-pypiorg). | ||
|
||
|
||
### --dev-dependencies/--no-dev-dependencies | ||
|
||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,6 +1,8 @@ | ||
import os | ||
import re | ||
import sys | ||
|
||
from functools import lru_cache | ||
from pathlib import Path | ||
from typing import TYPE_CHECKING, Dict, List, Optional | ||
from urllib.parse import urldefrag, urlsplit, urlunsplit | ||
|
@@ -373,6 +375,9 @@ def _prepare_repositories_pool(allow_pypi_requests: bool) -> Pool: | |
factory = Factory() | ||
config = factory.create_config() | ||
repos = [ | ||
factory.create_legacy_repository({"name": name, "url": url}, config) | ||
for name, url in _parse_repositories_from_environment().items() | ||
] + [ | ||
Comment on lines
+378
to
+380
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Could you please write a test for |
||
factory.create_legacy_repository( | ||
{"name": source[0], "url": source[1]["url"]}, config | ||
) | ||
Comment on lines
381
to
383
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I've left the previous method here so this is not a breaking change. However since this is exposing an implementation detail as an interface, it might be worth adding a deprecation warning? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm wondering if we're ready yet to deprecate it. I'm very eager to ditch anything Poetry-related, but setting up environment variables can be a pain, and until we get something better in place, some people may prefer the Poetry interface. |
||
|
@@ -383,6 +388,16 @@ def _prepare_repositories_pool(allow_pypi_requests: bool) -> Pool: | |
return Pool(repositories=[*repos]) | ||
|
||
|
||
@lru_cache(maxsize=1) | ||
def _parse_repositories_from_environment() -> Dict[str, str]: | ||
env_prefix = "CONDA_LOCK_PYPI_REPOSITORY_" | ||
return { | ||
key[len(env_prefix) :].lower(): value | ||
for key, value in os.environ.items() | ||
if key.startswith(env_prefix) | ||
} | ||
|
||
|
||
def _strip_auth(url: str) -> str: | ||
"""Strip HTTP Basic authentication from a URL.""" | ||
parts = urlsplit(url, allow_fragments=True) | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -60,7 +60,13 @@ | |
) | ||
from conda_lock.models.channel import Channel | ||
from conda_lock.models.lock_spec import VCSDependency, VersionedDependency | ||
from conda_lock.pypi_solver import _strip_auth, parse_pip_requirement, solve_pypi | ||
from conda_lock.pypi_solver import ( | ||
_parse_repositories_from_environment, | ||
_prepare_repositories_pool, | ||
_strip_auth, | ||
parse_pip_requirement, | ||
solve_pypi, | ||
) | ||
from conda_lock.src_parser import ( | ||
DEFAULT_PLATFORMS, | ||
LockSpecification, | ||
|
@@ -1673,6 +1679,24 @@ def test__extract_domain(line: str, stripped: str): | |
assert _extract_domain(line) == stripped | ||
|
||
|
||
def test_parse_repositories_from_environment(): | ||
# Given a private repository is configured in the environment | ||
try: | ||
_parse_repositories_from_environment.cache_clear() | ||
repository_url = "https://username:[email protected]/simple" | ||
os.environ["CONDA_LOCK_PYPI_REPOSITORY_EXAMPLE"] = repository_url | ||
# When I prepare the repositories pool | ||
pool = _prepare_repositories_pool(allow_pypi_requests=False) | ||
# Then the private repository is included in the pool | ||
assert pool.repositories, "No repositories were detected" | ||
assert ( | ||
pool.repositories[0].url == repository_url | ||
), "Detected repository has incorrect URL" | ||
finally: | ||
_parse_repositories_from_environment.cache_clear() | ||
del os.environ["CONDA_LOCK_PYPI_REPOSITORY_EXAMPLE"] | ||
|
||
|
||
def _read_file(filepath: "str | Path") -> str: | ||
with open(filepath, mode="r") as file_pointer: | ||
return file_pointer.read() | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@maresb I have restored documentation for the existing repository configuration method here.
Since the previous documentation was incorrect, I have changed the documented process for configuring it this way. This is because the
poetry config repositories.foo
will write configuration to the wrong appdirs directory forconda-lock
to discover (~/.config/pypoetry
vs~/.config/pypoetry-conda-lock
).I have also written it in a way which avoids users needing to install the exact version of
poetry
vendored byconda-lock
.Also note: I couldn't decipher which directory to recommend for windows, as I suspect its platform-dependent, so that's the best I could do. The old documentation didn't have this problem for the same reason it didn't work.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks a lot for catching this! Seems like we broke this in #342.