Ability to specify the directory to run poetry in instead of always using current directory

[svermeulen]

• [x ] I have searched the issues of this repo and believe that this is not a duplicate.
• [ x] I have searched the documentation and believe that my question is not covered.

Feature Request

I'd like to be able to run poetry run X while also setting the current directory to whatever I want. Right now, this appears to be impossible, since I have to set the current directory to wherever my pyproject.toml file is located instead. Can there be a command line option to override this default?

[svermeulen]

I did find a workaround though. I created the following helper script:

#!/bin/sh
STARTDIR=`pwd`
cd TOML_DIR
poetry run bash -c "cd $STARTDIR && python -m foo $*"

Where TOML_DIR is the path to the directory with your pyproject.toml

[bphunter1972]

This works well if your module has an if __name__ == '__main__', which isn't precisely what will get created with a [tool.poetry.scripts] specification.

[noamraph]

Just in case it helps someone, here's the script I used. It finds TOML_DIR from the location of the executable, so it doesn't need to be hardcoded. It also uses a function, like [tool.poetry.scripts].

#!/bin/sh
STARTDIR=$(pwd)
TOML_DIR=$(dirname "$0")
cd "$TOML_DIR" || exit
poetry run bash -c "cd $STARTDIR && python -c 'from foo.foo import main; main()' $*"

[abn]

I am a bit unclear on the issue here. Poetry, similar to git, will detect the pyproject.toml as long as the current directory is a child directory.

Here is an example where poetry run is executed within the src directory in the project.

$ tree .
.
├── poetry.lock
├── pyproject.toml
├── README.rst
├── src
│   ├── poetry
│   │   └── run
│   │       └── context
│   │           └── __init__.py
│   └── poetry_run_context.egg-info
│       ├── dependency_links.txt
│       ├── PKG-INFO
│       ├── SOURCES.txt
│       └── top_level.txt
└── tests
    └── __init__.py

6 directories, 9 files

$ type pytest
bash: type: pytest: not found

$ cd src/

$ poetry run type pytest
pytest is /tmp/poetry-run-context/.venv/bin/pytest

[abn]

@noamraph something like this will also work.

[tool.poetry.scripts]
foo = "foo.bar.main:main"

And then execute the follwoing within the project context (any sub directory):

poetry run foo

Is the requirement here for you to be able to run foo in a directory outside the project context? Something like this? (Possible proposal.)

poetry --project=/path/to/project-foo run foo

[svermeulen]

@abn yes, a --project argument would make our lives much easier in these cases

[noamraph]

Yes, exactly. It would be like git's -C option, that allows you to run commands on a repository regardless of your working directory.

[noamraph]

@abn I want to describe my use case a bit more. I'm developing a command line script that depends on the working directory. It's actually a tool for working with git repositories, so it detects the repository using the working directory (and it also has a -C option for specifying the repository directory manually). So in order to test the script, I have to be able to run it from different working directories.

[hamedhsn]

@abn I have the same sort of requirement, I am developing a CLI tool for which one of the functionalities is to build docker images. so for that, my working directory should be dir which I want to build my image and I want to remotely run my cli(managed by poetry). --project argument makes things much easier.

[abn]

The reason why I am not a 100% sold on the --project option is that this does not seem, atleast not obviously, useful for commands other than run. But even then the better solution might be to install scripts to alternative location (user/system site) judging by the above use cases.

This might not be the exact solution for the CLI development case, but here are a few options that might help till a better solution exists. All examples assume snippet from #2179 (comment).

1. Using the poetry managed virtual environment using one of the following.

poetry shell
source /path/to/project/.venv/bin/activate
source $(poetry env info --path)/bin/activate

You can then do something like this.

$ poetry shell
(.venv) $ cd ~/
(.venv) $ foo --help

2. Using an alias (you can add this to your .bashrc.

alias foo=/path/to/project/.venv/bin/foo

3. Install as an editable install (YMMV). This is not officially supported by pip since editable installs via PEP 517 is not supported. For this you need to add a setup.py shim.

$ echo "import setuptools; setuptools.setup()" > /path/to/project/setup.py
$ pip install --user /path/to/project

[bphunter1972]

@abn The issue I will have with the 'poetry shell' method is that my command-line script needs other environment variables outside of poetry's control to launch the tools that my script will use. As soon as I switch to poetry's virtual env, those will get cleared out.

However, running 'poetry install' and then using @noamraph 's solution is ideal.

[abn]

@bphunter1972 option 1 (using source) or option 2 (usimg alias) above should work in your case, ie. not clear existing variables.

[bphunter1972]

@abn You're right! I don't know how I missed it before, but after 'poetry install' running '/path/to/project/.venv/bin/foo' just works from any directory!

I feel like this should be more clearly documented, but I'm not complaining. Just happy to have resolved this.

[ericriff]

Pipenv allows you to do this by exporting PIPENV_PIPFILE=path-to-pipfile before running pipenv run. I would lovo to have the same functionality on poetry.

[ericriff]

Having the ability to point to the path of the pyproject provides many advantages.

One example would be to be able to run your scripts from outside the project folder. Lets say we have something like
~/projects/poetryproject which contains script.py and all the poetry files.
I would love to do poetry run ~/projects/poetryproject/script.py, but I cant since I have no way to tell poetry where to find the files it needs.
And this would allow us to have something like pipenv-shebang which is a game changer if you ask me. You can simply add a shebang to your scripts and forget about it being called from and virtual environment.
https://github.com/laktak/pipenv-shebang/blob/master/bash/pipenv-shebang

[ericriff]

The reason why I am not a 100% sold on the --project option is that this does not seem, atleast not obviously, useful for commands other than run. But even then the better solution might be to install scripts to alternative location (user/system site) judging by the above use cases.

This might not be the exact solution for the CLI development case, but here are a few options that might help till a better solution exists. All examples assume snippet from #2179 (comment).

1. Using the poetry managed virtual environment using one of the following.

poetry shell
source /path/to/project/.venv/bin/activate
source $(poetry env info --path)/bin/activate

You can then do something like this.

$ poetry shell
(.venv) $ cd ~/
(.venv) $ foo --help

1. Using an alias (you can add this to your .bashrc.

alias foo=/path/to/project/.venv/bin/foo

1. Install as an editable install (YMMV). This is not officially supported by pip since editable installs via PEP 517 is not supported. For this you need to add a setup.py shim.

$ echo "import setuptools; setuptools.setup()" > /path/to/project/setup.py
$ pip install --user /path/to/project

This source $(poetry env info --path)/bin/activate is broken if you use python 3.5 or python 2, because of the warning about those versions of python being deprecated.
This also fails if python defaults to python2 and you installed poetry with get-poetry.py, since the executable has python in its shebang.

[kshcherban]

Even cd project/path && $(poetry env info -p)/bin/python foo.py works but changing directories all the time is not convenient.
Also imagine such use case: you work in a monorepo and you need to start multiple projects from the same shell script.

[fn-reflection]

I want to call IPython in virtual env that is managed by poetry.
So I write some shell script function as below.

# I use zsh shell, therefore I wrote this in ~/.zshrc
function ipy() {
    cd ~/pj/ipython_env # go to poetry project path that manage ipython's env
    poetry run python -m IPython # call IPython in virtual env
    cd - > /dev/null # return to previous path
}

This function almost achieve my purpose.

[sinoroc]

If you know the path to the virtual environment, then something like the following should work:

path/to/venv/bin/somescript

No need to call poetry run somescript or call poetry shell or activate the virtual environment. Won't cover all the use cases, but maybe some.

[cstruct]

I have a use case other than run that would benefit from --project, if that is whats blocking that flag. I usually run poetry update, poetry install or poetry lock --no-update across multiple packages with find, if I could pass --target I would no longer need to spawn a shell and run cd. @abn

i.e.

find . -name 'pyproject.toml' -exec bash -c 'cd "$(dirname {})" && poetry update' \;

Would become:

find . -name 'pyproject.toml' -exec poetry update --target {} \;

[gvoysey]

Perhaps tangentially, but germane to the issue at hand of "doing poetry things outside of the directory pyproject.toml is in": Would it make sense for poetry env info to gain an optional project directory argument that would default to pwd?

The immediate use case i can think of would be having a crontab entry of the form

0 * * * *  $(poetry env info -p /home/user/project/poetry-managed-project)/bin/entry_point --arg foo 

but there are many other cases where activation or spawning a shell are less than desirable, but you still want to reach out and hit an entry point.

[k0t3n]

Extremely needed feature for multirepos!

[DanielKusyDev]

GitHub Actions does not support setting WORKDIR in Dockerfile which means that at the moment it's not possible to use poetry in other directories than the main one without tricks. Explanation on Dockerfile support for Github Actions. This feature would make life much easier.

[wolph]

The reason why I am not a 100% sold on the --project option is that this does not seem, atleast not obviously, useful for commands other than run. But even then the better solution might be to install scripts to alternative location (user/system site) judging by the above use cases.

For the case of a cron job the --project option which changes the working directory and active poetry project would be the most ideal.

Installing the scripts in a different location would require you to add both of those paths to the crontab which makes it another thing that can break. For example, when creating a new virtualenv in a different location your script location in the crontab might be out of date. Having poetry installed in /usr/local/bin/poetry would at least result in a constant path so you could do something like this:

0 1 2 3 4 5 /usr/local/bin/poetry --project /some/project/directory run python3 some_script.py

Or if --project would be part of the run command:

0 1 2 3 4 5 /usr/local/bin/poetry run --project /some/project/directory python3 some_script.py

Having to use a subshell to get the python path is inconvenient and prone to breaking. If some DeprecationWarning or something else pops up it can break.

[dgurns]

Being able to pass a --project flag (or similar) would be helpful for our use case. We have a Go script which calls a Python project in a child directory. Like:

go-project/
  python-project/
    pyproject.toml

Would love to do:

cd go-project
poetry run --project python-project [...run Go command...]

Otherwise we currently have to do this, which is inconvenient:

cd go-project/python-project
poetry shell
cd ..
[...run Go command...]

[flickerfly]

I want to setup a pre-commit hook that runs poetry run commands against the python code in the specific directory because that is where the python project is in the context of a bigger project.

-   repo: local
    hooks:
    -   id: flake8
        name: flake8
        entry: poetry run flake8
        language: system
        types: [python]
        files: cli/

[mohkale]

@flickerfly

There's been a few example wrapper scripts shared in this issue. Here's mine for reference. Simply pass it the path to the directory containing the poetry project and then the command to run. It'll change back to the cwd within the poetry shell before running the command.

[flickerfly]

I messed with that a bit, but ran into a cross-platform support issue. I need it to work on windows and linux. Seems like in Windows it was running in cmd or Powershell.

[mohkale]

@flickerfly

Can't really help there. You could probably reimplement it in python, but my use case was linux so I just kept it in sh. Hopefully someone else has a similair cross platform script available.

[flickerfly]

Understood, I'm more adding to the case that having the ability to identify the context the command is run in would simplify the user experience.

[charmoniumQ]

There's also env -C /path/to/pyproject poetry run env -C ${PWD} cmd which has the advantage of being one command (no cd ... && ...). Being one command works better for subprocess.run([...], shell=False). But the best would still be something like: poetry -C /path/to/pyproject run cmd.

Note that -C/--chdir is the name used in make, CMake, Git, ninja, env, and I'm sure others.

[neozenith]

Context and Prior Art

Coming from working in Typescript for a few years I have become accustom to multi-project monorepos where we need to manage and deploy multiple projects in a very similar way, but within the same repo so we have a transaction of system wide changes that impact refactorings across all projects.

The tooling we used there was:

• https://github.com/lerna/lerna
• but we later migrated to https://docs.npmjs.com/cli/v7/using-npm/workspaces

Desired future

Trying to find similar tooling I have a combination of poetry and invoke.

• poetry is amazing at managing the virtualenv and dependency management.
• invoke has a flask like ergonomics for creating workflow automations and discoverable tasks.

I even have a poetry-pyinvoke-plugin ready for when poetry launches v1.2.+ so poetry run inv -l simply becomes poetry inv -l

Sub Projects

Each sub-project has a tasks.py file so I can run poetry run invoke --list or the shorthand poetry run inv -l.

Top level project

So ideally in my root project, I am using poetry to manage this top level tooling and virtualenv.
Then I can run something like:

Single project
poetry run invoke manage --project project1 --command 'lint test build deploy'
or the shorthand:
poetry run inv m -p project1 -c 'lint test build deploy'

All projects
poetry run inv m -c 'lint test build deploy'

Where this runs the same command over a cleaned up version of

all_projects = [
  p for p in os.listdir('.') 
  if os.path.is_dir(os.path.join('.', p)) and not p.startswith('.')
]

Problem

If I have started a poetry run to start using the tooling out of the top level venv, it seems impossible to get a secondary command to run out of the sub-project venv.

• I have tried all of the above "workarounds" with no success.
• I have even tried activating the root venv without poetry just using . .venv/bin/activate and then running the root invoke manage -c 'lint test build deploy' which will cd subproject && poetry run inv lint test build deploy.
• I have tried not using the from invoke import run, but instead using the from subprocess import run.
• I have tried run(..., env={"VIRTUAL_ENV": "...", "PATH": "..."}) where I manually construct the PATH and VIRTUAL_ENV so poetry detects the venv I am trying to coerce it to detect
• I have tried scouring through the poetry source code to figure out how EnvManager creates an Env for the EnvCommand that eventually gets inherited for use by RunCommand and for the life of me, I could not figure out what I needed to set to coerce it to detect the subproject virtualenv instead.
• I have even tried, poetry shell from my root, cd into a subproject, then poetry shell and it won't activate the subproject because Virtual environment already activated.

The only way around it so far is getting this manage tasks to run something like:
cd subproject && .venv/bin/python3 -m invoke {my_command_here}

This completely avoids, poetry and virtualenv altering the environment variables and explicitly runs a targeted python and the relative libraries.

Conclusion / Summary

1. Could I get some guidance on how I could get poetry to detect the virtualenv based on the current directory / targetted pyproject.toml when another virtualenv is already active?
2. Or some guidance on if a PR could be opened to add this feature?

[sinoroc]

@neozenith You might have more luck here: #2270

[neozenith]

@sinoroc Brilliant! Thank you so much 🙏 That thread of discussion looks promising.

[neozenith]

Actually after some experimentation I did wind up coercing a poetry run in a top level project to perform a poetry run in a subproject.

Can thank this stackoverflow answer for pointing out subprocess.run has a start_new_session option.

But also after some manipulation, it seems:

• I needed to set VIRTUAL_ENV for the subproject and
• I need to update PATH as well so that the right python3 binary was discovered
• Of course change the current working directory

With those three components in place, the subprocess has everything needed to run independently.

Below is a snippet from my root level tasks.py

import os
from pathlib import Path
from subprocess import run
from invoke import task

@task(aliases=["m"])
def manage(c, project=None, command="poetry run inv -l"):
    """Manage a task in a project or all projects."""

    # Collect list of dirs that are valid projects
    d = "."
    all_projects = [
        o
        for o in os.listdir(d)
        if os.path.isdir(os.path.join(d, o)) and not o.startswith(".")
    ]

    # Reconcile list of projects to apply command to
    if project is not None:
        if project in all_projects:
            projects = [project]
        else:
            raise ValueError(f"The project '{project}' is not one of {all_projects}")
    else:
        projects = all_projects

    for p in projects:
        target_venv_path = Path(os.environ["VIRTUAL_ENV"]).parent / p / ".venv"
        target_bin_path = str(target_venv_path / "bin")
        target_cwd = str(Path().cwd() / p)
        target_path = ":".join([target_bin_path] + os.environ["PATH"].split(":"))

        # Curate new ENV to coerce poetry into picking the subproject venv
        ENV = {"VIRTUAL_ENV": str(target_venv_path), "PATH": target_path}

        result = run(
            command,
            cwd=target_cwd,
            shell=True,
            capture_output=True,
            env=ENV,
        )

        print(result.stdout.decode("utf-8"))
        print(result.stderr.decode("utf-8"))

With this format, I now have the power to run any arbitrary task like:

poetry run inv manage -c 'poetry install'

[marcospgp]

Could someone let me know if this issue has been fixed or if there is a simple workaround? This thread has grown to become quite huge and hard to go through in its entirety!

[sinoroc]

@marcospgp global option -C or --directory https://python-poetry.org/docs/cli/#global-options

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.