dist/ in .gitignore makes wheel empty

[Andrew-Sheridan]

Hello build maintainers.

tl;dr As of build==0.5.0, if .gitignore has dist/ in it, then the wheel created will be mostly empty.

Below are steps to recreate

Create files

mkdir git-build-wheel
cd git-build-wheel
mkdir src
mkdir src/org
mkdir src/org/lib_name
touch src/org/lib_name/__init__.py
cat << EOF > pyproject.toml
[tool.poetry]
name = "org.lib-name"
version = "0.1.0"
description = ""
authors = [
    "Andrew Sheridan <[email protected]>",
]
packages = [{ include = "org", from = "src"}]

[tool.poetry.dependencies]
python = "^3.7.0"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

EOF

Init git / activate python / install build

git init
python3.7 -m venv venv
source venv/bin/activate
pip install build

Build (working)

python -m build
unzip dist/org.lib_name-0.1.0-py3-none-any.whl
ls org

prints lib_name, and ls org/lib_name prints __init__.py

Build (not working)

rm -rf org dist org.lib_name-0.1.0.dist-info
echo "dist/" >> .gitignore
python -m build
unzip dist/org.lib_name-0.1.0-py3-none-any.whl
ls org

prints ls: org: No such file or directory

And working with 0.4.0

pip install build==0.4.0
rm -rf org dist org.lib_name-0.1.0.dist-info
python -m build
ls org

prints lib_name, and ls org/lib_name prints __init__.py

🤷🏻‍♂️

[henryiii]

Poetry uses the .gitignore to decide what to include. When it's building from SDist, I have a feeling it's detecting that it's in a git repository? @FFY00 Why are we not building in an isolated temporary directory? In #304, they are being placed in <outdir>/<tmpdir>. It's in source-tree, Poetry (and other tools that use SCM, including setuptools plugins) will likely break with this scheme. Plus it's also possible tools that automatically detect the version from SCM will "work" even on a broken SDist because they are picking it up from git, not from the SDist. The extracted file should be fully isolated in a temporary directory.

As a temporary workaround, @Andrew-Sheridan, you can set --outdir to be something not inside the source directory.

[Andrew-Sheridan]

@henryiii Thanks for the response. Please doublecheck what I typed is actually accurate when you are able, it's possible I have made some simple error.

[FFY00]

Why are we not building in an isolated temporary directory?

The rationale was that I wanted to provide the source on the failed build for users to easily inspect and try to reproduce the issue. Given the issue presented here, I believe we should do it in an isolated temporary directory. I'd want build systems to be smart enough to notice they are being invoked inside the directory that is in the .gitignore and so they should probably ignore that, but that's probably a big ask 😅

[layday]

If we chdir into the unpacked sdist then why does Poetry try to read the git ignores from the project root? Does it mean that it'll do that for any Poetry sdist you happen to build inside your project, so that if you ignore "foo" in your project's repo and a downloaded sdist contains a package "foo", "foo" won't be packaged in the wheel? Poetry should be passing --git-dir=${PROJECT_ROOT}/.git to git, taking advantage of the fact that PEP 517 mandates that hooks are run from the project root. Please open an issue with Poetry as well.

[henryiii]

I think it's querying git, not .gitignore directly. If the project root was not at the top level (which is not uncommon), you still want to use git's ignore scheme (including the project root .gitignore) in seeing if anything is ignored. For example, if your package is in /python, and you put /python/dist in your main .gitignore, it should be still be ignored (which it currently is).

The problem is that everything is in /dist, and poetry asks git, and git says everything in dist is being ignored.

[FFY00]

Poetry should be passing --git-dir=${PROJECT_ROOT}/.git to git

That will break this functionality when the package is not at the root of the repo.

[layday]

I don't understand what will break? The "project root" here is the unpacked sdist folder.

[FFY00]

If I have a git repo with packages in subdirectories, hardcoding the .git path based on the python project root will break the vcs files detection because the git repo does not live there.

[layday]

Then they might want to consider disabling gitignore detection for sdists, e.g. if PKG-INFO exists in the directory or some other Poetry-specific marker file. That the gitignores apply to any Poetry sdist you happen to build in your project is probably unintentional and would merit discussion, even if it's to say the status quo is an acceptable compromise.

[FFY00]

Perhaps, but some other backends will inevitably run into this same issue again, so we can't do anything about it.
In my backend, I only put the relevant files in the sdist. If the user want to add something, they need to specifically request that.