Some wheels published fail to install with "BadZipFile - Bad CRC-32 for file"

[jselig-rigetti]

Found during rigetti/pyquil#1817

This may only affect some platforms and versions, but can be reproduced by:

docker run -it --platform="linux/x86_64" python:3.9 pip install qcs-sdk-python

Resulting in:

ERROR: Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 160, in exc_logging_wrapper
    status = run_func(*args)
  File "/usr/local/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 247, in wrapper
    return func(self, options, args)
  File "/usr/local/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 507, in run
    installed = install_given_reqs(
  File "/usr/local/lib/python3.9/site-packages/pip/_internal/req/__init__.py", line 73, in install_given_reqs
    requirement.install(
  File "/usr/local/lib/python3.9/site-packages/pip/_internal/req/req_install.py", line 796, in install
    install_wheel(
  File "/usr/local/lib/python3.9/site-packages/pip/_internal/operations/install/wheel.py", line 729, in install_wheel
    _install_wheel(
  File "/usr/local/lib/python3.9/site-packages/pip/_internal/operations/install/wheel.py", line 589, in _install_wheel
    file.save()
  File "/usr/local/lib/python3.9/site-packages/pip/_internal/operations/install/wheel.py", line 388, in save
    shutil.copyfileobj(f, dest)
  File "/usr/local/lib/python3.9/shutil.py", line 205, in copyfileobj
    buf = fsrc_read(length)
  File "/usr/local/lib/python3.9/zipfile.py", line 926, in read
    data = self._read1(n)
  File "/usr/local/lib/python3.9/zipfile.py", line 1016, in _read1
    self._update_crc(data)
  File "/usr/local/lib/python3.9/zipfile.py", line 944, in _update_crc
    raise BadZipFile("Bad CRC-32 for file %r" % self.name)
zipfile.BadZipFile: Bad CRC-32 for file 'qcs_sdk/qcs_sdk.cpython-39-x86_64-linux-gnu.so'

[jselig-rigetti]

Starting in the repo root directory, running either of the following

docker run -it --platform="linux/x86_64" --entrypoint /bin/bash -v $(pwd):/code quay.io/pypa/manylinux_2_34_x86_64:latest
docker run -it --platform="linux/aarch64" --entrypoint /bin/bash -v $(pwd):/code quay.io/pypa/manylinux_2_34_aarch64:latest

Then this setup

(curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y) && . "$HOME/.cargo/env" && cargo install cargo-make
python3.9 -m venv /.venv && source /.venv/bin/activate

cd /code/crates/python
cargo make install-deps

cd /code
maturin build -i 3.9 --release --manifest-path crates/python/Cargo.toml --out /dist

Makes this complete successfully

pip install /dist/*.whl
python -c "import qcs_sdk; print(qcs_sdk.__version__)"

... since this is the case, maybe we can override the inputs.container

[pavoljuhas]

+1 - there seems to be the same issue for the recently released
qcs_sdk_python-0.21.9-cp312-cp312-win_amd64.whl - https://github.com/quantumlib/Cirq/actions/runs/13122857922/job/36614926452#step:5:832

Can you please yank out the broken files from the PyPI release?

[pavoljuhas]

This seems to be happening for the recent release 0.21.11 as well:

$ wget 'https://files.pythonhosted.org/packages/a3/f3/8c346e87159ca2216dfb8fd8fcec19bd2247f1ca23ab36af086bba24b25b/qcs_sdk_python-0.21.11-cp312-cp312-manylinux_2_28_x86_64.whl'

$ unzip qcs_sdk_python-0.21.11-cp312-cp312-manylinux_2_28_x86_64.whl
Archive:  qcs_sdk_python-0.21.11-cp312-cp312-manylinux_2_28_x86_64.whl
...
  inflating: qcs_sdk/qcs_sdk.cpython-312-x86_64-linux-gnu.so
  error:  invalid compressed data to inflate
  inflating: qcs_sdk_python-0.21.11.dist-info/RECORD

Can you please add a zipfile integrity test to the publishing process?

[jselig-rigetti]

@pavoljuhas https://pypi.org/project/qcs-sdk-python/0.21.12/ should have fixed most of these errors, although some still remain only for the linux python3.9 versions. Can you try to update?

Particularly, https://files.pythonhosted.org/packages/ae/8b/1009f78b20b0fd709ab170395daab64329d8cfeed5993d6bd6bba08b4384/qcs_sdk_python-0.21.12-cp312-cp312-manylinux_2_28_x86_64.whl should be fixed.

[pavoljuhas]

We do not support Python 3.9 so that is not an issue.
The latest release of qcs-sdk-python works, thank you for the fix!

[pavoljuhas]

@jselig-rigetti - we just hit the same issue for the yesterday's release qcs-sdk-python-0.21.13 for Linux Python 3.10.
Specifically, after downloading https://files.pythonhosted.org/packages/1a/a5/09c8f9324f5015e2251148b62caeecddba086634ba67df265faecb495962/qcs_sdk_python-0.21.13-cp310-cp310-manylinux_2_28_x86_64.whl
the pip install fails on Bad CRC-32:

$ pip install --no-deps ./qcs_sdk_python-0.21.13-cp310-cp310-manylinux_2_28_x86_64.whl
...
zipfile.BadZipFile: Bad CRC-32 for file 'qcs_sdk/qcs_sdk.cpython-310-x86_64-linux-gnu.so'

Strangely, the wheel file passes unzip -t, but the CRC-32 checksum indeed does not agree for the .so file
(expected 11a9451c actual 6ebf899d):

$ unzip -t ./qcs_sdk_python-0.21.13-cp310-cp310-manylinux_2_28_x86_64.whl
...
    testing: qcs_sdk/qcs_sdk.cpython-310-x86_64-linux-gnu.so   OK
...
No errors detected in compressed data of ./qcs_sdk_python-0.21.13-cp310-cp310-manylinux_2_28_x86_64.whl.

$ unzip -ll ./qcs_sdk_python-0.21.13-cp310-cp310-manylinux_2_28_x86_64.whl "*.so"
Archive:  ./qcs_sdk_python-0.21.13-cp310-cp310-manylinux_2_28_x86_64.whl
 Length   Method    Size  Cmpr    Date    Time   CRC-32   Name
--------  ------  ------- ---- ---------- ----- --------  ----
15528680  Defl:N  6314527  59% 2025-03-04 22:56 11a9451c  qcs_sdk/qcs_sdk.cpython-310-x86_64-linux-gnu.so
--------          -------  ---                            -------
15528680          6314527  59%                            1 file

$ unzip ./qcs_sdk_python-0.21.13-cp310-cp310-manylinux_2_28_x86_64.whl "*.so"
$ crc32 qcs_sdk/qcs_sdk.cpython-310-x86_64-linux-gnu.so
6ebf899d

Can you please do a check-up pip-install of your wheels before publishing them on PyPI?
Apparently unzip -t wheelfile.whl cannot be relied on for testing a zip archive integrity, but pip install ./wheelfile.whl does catch the errors.

[jselig-rigetti]

@pavoljuhas apologies, looking at this now - this is has been a tricky one as we are already test-installing the wheels via https://github.com/rigetti/qcs-sdk-rust/blob/main/.github/action-common-python-release/action.yml#L68

[pavoljuhas]

Thank you for checking. The wheel file also fails integrity test as follows -

$ python -m zipfile --test qcs_sdk_python-0.21.13-cp310-cp310-manylinux_2_28_x86_64.whl
The following enclosed file is corrupted: 'qcs_sdk/qcs_sdk.cpython-310-x86_64-linux-gnu.so'
Done testing

Unfortunately, python -m zipfile --test does not set a nonzero exit code so the only way to check is to inspect the output.

[jselig-rigetti]

@pavoljuhas based on the similarity to https://bugs.python.org/issue32304 I suspect this may be an issue that is particular to uploading, I've opened PyO3/maturin#2499

[jselig-rigetti]

I believe this was a regression caused by https://github.com/rigetti/qcs-sdk-rust/pull/514/files, that is:

Both qcs-sdk-python and qcs-sdk-python-grpc-web were building wheels called qcs-sdk-python, and the publish job for the standard qcs-sdk-python was downloading artifacts from either package which would non-deterministically overwrite each-other.

That is to say, if you are publishing two packages with different names because they use different rust feature flags, make sure they are named correctly and their artifact names don't overlap!

[jselig-rigetti]

Closing as #531 (comment) is the expected fix.

Please re-open if this issue persists.