Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Python] Add version suffix to libarrow_python* libraries #44614

Closed
eltoder opened this issue Nov 1, 2024 · 2 comments
Closed

[Python] Add version suffix to libarrow_python* libraries #44614

eltoder opened this issue Nov 1, 2024 · 2 comments
Assignees
@raulcd
Labels
Component: Python Type: enhancement
Milestone
19.0.0

Comments

@eltoder
Copy link

eltoder commented Nov 1, 2024 •
edited
Loading

Describe the enhancement requested

pyarrow wheels ship a number of shared libraries. Most of these libraries have a version suffix, but the "python" libraries do not:

$ readelf -d .venv/lib/python3.10/site-packages/pyarrow/*.so* | grep "NEEDED.*arrow" | sort -u
 0x0000000000000001 (NEEDED)             Shared library: [libarrow_acero.so.1800]
 0x0000000000000001 (NEEDED)             Shared library: [libarrow_dataset.so.1800]
 0x0000000000000001 (NEEDED)             Shared library: [libarrow_flight.so.1800]
 0x0000000000000001 (NEEDED)             Shared library: [libarrow_python_flight.so]
 0x0000000000000001 (NEEDED)             Shared library: [libarrow_python_parquet_encryption.so]
 0x0000000000000001 (NEEDED)             Shared library: [libarrow_python.so]
 0x0000000000000001 (NEEDED)             Shared library: [libarrow.so.1800]
 0x0000000000000001 (NEEDED)             Shared library: [libarrow_substrait.so.1800]

Version suffixes are nice, because they prevent us from loading an incompatible version of the library if it somehow happens to be present in LD_LIBRARY_PATH. (Note that LD_LIBRARY_PATH takes precedence over RUNPATH entries added during linking.)

It would be good to add version suffixes for libarrow_python* libraries as well -- for the same reason. We currently have a situation where (for hard to fix reasons) we end up with LD_LIBRARY_PATH that contains an old install of pyarrow. This breaks installing newer pyarrow into venvs -- even though the correct version of python scripts and lib.cpython-310-x86_64-linux-gnu.so is used, the latter picks up the old libarrow_python.so and fails to resolve symbols.

Component(s)

Python
@kou kou changed the title Add version suffix to libarrow_python* libraries [Python] Add version suffix to libarrow_python* libraries Nov 3, 2024
@pitrou
Copy link
Member

pitrou commented Nov 12, 2024

cc @raulcd

raulcd added a commit to raulcd/arrow that referenced this issue Nov 12, 2024
@raulcd
apacheGH-44614: [Python][C++] Add version suffix to libarrow_python* …
07b1f4e 
…libraries
@github-actions github-actions bot mentioned this issue Nov 12, 2024
Merged
assignUser pushed a commit that referenced this issue Nov 13, 2024
@raulcd
GH-44614: [Python][C++] Add version suffix to libarrow_python* librar…
d7641dc 
…ies (#44702)

### Rationale for this change

We are currently not setting library version suffixes for arrow python C++ libraries but we do so for libarrow C++.

### What changes are included in this PR?

Add the same logic that we use for libarrow.

### Are these changes tested?

I've validated manually that the suffixes are generated.

```
tree | grep libarrow_python
│       │   ├── libarrow_python.so -> libarrow_python.so.1900
│       │   ├── libarrow_python.so.1900 -> libarrow_python.so.1900.0.0
│       │   ├── libarrow_python.so.1900.0.0
│   │   ├── libarrow_python.pxd
│   ├── libarrow_python.so -> libarrow_python.so.1900
│   ├── libarrow_python.so.1900 -> libarrow_python.so.1900.0.0
│   ├── libarrow_python.so.1900.0.0
```

### Are there any user-facing changes?

We will generate so libraries with version suffixes.
* GitHub Issue: #44614

Authored-by: Raúl Cumplido <[email protected]>
Signed-off-by: Jacob Wujciak-Jens <[email protected]>
@assignUser
Copy link
Member

Issue resolved by pull request 44702
#44702

@assignUser assignUser added this to the 19.0.0 milestone Nov 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@raulcd raulcd

Labels
Component: Python Type: enhancement
Projects
None yet
Milestone
19.0.0
Development

No branches or pull requests
4 participants
@raulcd @pitrou @eltoder @assignUser