Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

(t) upgrade psycopg2 within current constrains #2406

Closed
phillxnet opened this issue Oct 26, 2022 · 12 comments
Closed

(t) upgrade psycopg2 within current constrains #2406

phillxnet opened this issue Oct 26, 2022 · 12 comments

Comments

@phillxnet
Copy link
Member

phillxnet commented Oct 26, 2022 •
edited
Loading

As part of an ongoing effort to identify our update options, this issue is opened to identify the limits of our current psycopg2.
This currently fails compilation on our aarch64 build:

Getting distribution for 'psycopg2==2.7.4'.
Error: pg_config executable not found.
pg_config is required to build psycopg2 from source.
...

Where-as the same psycopg2 version has a wheel available and default to it in our x86 builds; resulting in a successful build.

From: https://www.psycopg.org/docs/#

Psycopg is the most popular PostgreSQL database adapter for the Python programming language.

Psycopg 2 is mostly implemented in C as a libpq wrapper, resulting in being both efficient and secure.

Looking to upgrade limits and constrains.

Constrains:
Django 1.11.29 (max psycopg2 version 2.7.7, >= Djanog 2.2.1 needed for >= 2.8 psycopg2)
https://pyup.io/packages/pypi/django/changelog
2.2.1
"* Added compatibility for psycopg2 2.8 (30331)."
https://code.djangoproject.com/ticket/30331

We may have the option to specify pre-build wheel options, if available, with "psycopg2-binary==2.7.7" which are recommended for venv use which we intent to migrate towards in the mid-term. No canonical reference yet for binary use within venv.

We also have a potential openSSL constraint here regarding psycopg2 version, most notably if using the wheel (binary) version:

Leap 15.3: - 1.1.1d-1.46
Leap 15.4: - 1.1.1l-150400.1.5

from: https://www.psycopg.org/docs/news.html#news
excerpts of note around our upgrade here:

2.7.4 Wheel packages compiled against PostgreSQL 10.1 libpq and OpenSSL 1.0.2n.
2.7.5 Wheel package compiled against PostgreSQL 10.4 libpq and OpenSSL 1.0.2o.
2.7.6 Wheel package compiled against PostgreSQL 10.5 libpq and OpenSSL 1.0.2p.
2.7.6.1 Wheel package compiled against PostgreSQL 11.1 libpq
2.7.7 Wheel package compiled against OpenSSL 1.0.2q.
2.8 (major upgrade)

  • Binary packages no longer installed by default. The ‘psycopg2-binary’ package must be used explicitly.
  • No longer use 2to3 during installation for Python 2 & 3 compatibility. All source files are now compatible with Python 2 & 3 as is.
  • Wheel package compiled against OpenSSL 1.0.2r and PostgreSQL 11.2 libpq.

2.8.2 Binary packages built with openssl 1.1.1b. Should fix concurrency problems (tickets 543, 836).

2.8.4

  • errorcodes map and errors classes updated to PostgreSQL 12.
  • Wheel package compiled against OpenSSL 1.1.1d and PostgreSQL at least 11.4.

2.8.6

  • errorcodes map and errors classes updated to PostgreSQL 13
  • Added wheel packages for ARM architecture (ticket 1125).
  • Wheel package compiled against OpenSSL 1.1.1g.
@phillxnet
Copy link
Member Author

The core problem currently is the compile failure on aarch64 that is not shared by the x86_64 build, holding up our dual architecture release.

@phillxnet
Copy link
Member Author

"psycopg2-binary==2.7.7" in setup.py results in the same failure on aarch64.

@phillxnet
Copy link
Member Author

An apparently successful source build on aarch64 in Leap 15.3 looks, on initial testing, to be working with this psycopg2 version specification removed entirely.

@phillxnet
Copy link
Member Author

The current proposal on this issue is to remove our version definition within setup.py as it has inconsistencies at least with the aarch64 architecture given our master and testing django-hack versions where we have a hard code for the x86_64 version.
Further investigation planned for the requirement of this library is needed but it seems likely we are resourcing, successfully in the most part, our system version. Even though it is a version known to be incompatible with our Django versions. It may well be we simply don't trigger this incompatibility with our limited use.

@phillxnet
Copy link
Member Author

Given in our x86_64 we are defaulting to a binary (pre 2.8 this was the default) but in our aarch64 we fail through to a source build of psycopg2 as the binary is not available for this arch, and then fail due to missing build requirements, namely from the original error message "pg_config" we look to the upstream advise re source/binary:

psycopg vs psycopg-binary

from: https://www.psycopg.org/docs/install.html#psycopg-vs-psycopg-binary

Warning The psycopg2 wheel package comes packaged, among the others, with its own libssl binary. This may create conflicts with other extension modules binding with libssl as well, for instance with the Python ssl module: in some cases, under concurrency, the interaction between the two libraries may result in a segfault. In case of doubts you are advised to use a package built from source.

Which would seem to advise us to resolve our missing dependency and have both architectures build psychopg2 from source, this way at least we can pick exactly our version, given our OS version is now already beyond the recommendation for our current (testing branch) django anyway, but appears to work on arm side. But later we will again want to exactly specify our versions so now moving here to resolve build deps for arm and attempt to force x86_64 to source build via a setup.py option hopefully; from above link we also have:

which can be specified in your requirements.txt files too, e.g. use:

psycopg2>=2.7,<2.8 --no-binary psycopg2

This path is consistent with our future requirement to run in a venv and build as much as possible. But inconsistent with the un-referenced advise to use the binary version when in a venv!

@phillxnet
Copy link
Member Author

Although we already have above more strict limits on our psycopg version, adding for future info here from the following:
https://www.psycopg.org/docs/news.html#news

What’s new in psycopg 2.9

  • Dropped support for Python 2.7, 3.4, 3.5 (tickets 1198, 1000, 1197).
  • Wheel package compiled against OpenSSL 1.1.1k and PostgreSQL 13.3 libpq.

What’s new in psycopg 2.9.2

@phillxnet
Copy link
Member Author

We appear to have yet-another-dependency issue related to our enabling the source build of psycopg2, against system postgresql/libpq, that of the removal of pg_config, in updates of our postgresql10 i.e.

rleap15-4:~ # 
zypper in command-not-found
cnf pg_config

The program 'pg_config' can be found in the following packages:
  * postgresql10-devel [ path: /usr/bin/pg_config, repository: zypp (repo-oss) ]
  * postgresql12-server-devel [ path: /usr/bin/pg_config, repository: zypp (repo-oss) ]
  * postgresql13-server-devel [ path: /usr/bin/pg_config, repository: zypp (repo-oss) ]
  * postgresql14-server-devel [ path: /usr/bin/pg_config, repository: zypp (repo-oss) ]
  * postgresql12-server-devel [ path: /usr/bin/pg_config, repository: zypp (repo-sle-update) ]
  * postgresql13-server-devel [ path: /usr/bin/pg_config, repository: zypp (repo-sle-update) ]
  * postgresql14-server-devel [ path: /usr/bin/pg_config, repository: zypp (repo-sle-update) ]

and the same for our Leap 15.3:

rleap15-3:~ # cnf pg_config
                           
The program 'pg_config' can be found in the following packages:
  * postgresql10-devel [ path: /usr/bin/pg_config, repository: zypp (Leap_15_3) ]
  * postgresql12-server-devel [ path: /usr/bin/pg_config, repository: zypp (Leap_15_3) ]
  * postgresql13-server-devel [ path: /usr/bin/pg_config, repository: zypp (Leap_15_3) ]
  * postgresql10-devel [ path: /usr/bin/pg_config, repository: zypp (repo-sle-update) ]
  * postgresql12-server-devel [ path: /usr/bin/pg_config, repository: zypp (repo-sle-update) ]
  * postgresql13-server-devel [ path: /usr/bin/pg_config, repository: zypp (repo-sle-update) ]
  * postgresql14-server-devel [ path: /usr/bin/pg_config, repository: zypp (repo-sle-update) ]
...

I.e. an attempted install of the legacy named postgresql10-devel, (differing from the future postgresql12-server-devel) resulting in a need to downgrade:

rleap15-3:~ # zypper in postgresql10-devel
...
Problem: the to be installed postgresql10-devel-10.17-8.35.1.x86_64 requires 'postgresql10-server = 10.17-8.35.1', but this requirement cannot be provided
  not installable providers: postgresql10-server-10.17-8.35.1.x86_64[repo-sle-update]
 Solution 1: Following actions will be done:
  downgrade of postgresql10-server-10.22-150100.8.50.1.x86_64 to postgresql10-server-10.17-8.35.1.x86_64
  downgrade of postgresql10-10.22-150100.8.50.1.x86_64 to postgresql10-10.17-8.35.1.x86_64
 Solution 2: do not install postgresql10-devel-10.17-8.35.1.x86_64
 Solution 3: break postgresql10-devel-10.17-8.35.1.x86_64 by ignoring some of its dependencies

with our currently installed and updated posgresql10-server having been updated from:

zypper info postgresql10-server | grep "Repository"
Repository     : Update repository with updates from SUSE Linux Enterprise 15
rpm -ql postgresql10-server-10.22-150100.8.50.1 | grep pg_config

but if we downgrade our 10 server, forgoing the updates in the newer, added by default via updates, SLE repo we gain our pg_config:

/usr/bin/pg_config

@phillxnet
Copy link
Member Author

rleap15-3:~ # rpm -qf /usr/bin/pg_config
postgresql10-devel-10.17-8.35.1.x86_64

And we are then stuck on this older version of postgresql10-server:

zypper up
The following 3 package updates will NOT be installed:
postgresql10 postgresql10-server python3-iniconfig

https://documentation.suse.com/sles/15-SP3/html/SLES-all/cha-update-alternative.html

rleap15-3:~ # ls -la /usr/bin/pg_config
lrwxrwxrwx 1 root root 27 Oct 27 13:46 /usr/bin/pg_config -> /etc/alternatives/pg_config
rleap15-3:~ # ls -la /etc/alternatives/pg_config
lrwxrwxrwx 1 root root 35 Oct 27 13:46 /etc/alternatives/pg_config -> /usr/lib/postgresql10/bin/pg_config

rleap15-3:~ # ls -la /usr/lib/postgresql10/bin/pg_config
-rwxr-xr-x 1 root root 31328 May 18 2021 /usr/lib/postgresql10/bin/pg_config

rleap15-3:~ # update-alternatives --display postgresql | grep "postgres|pg_config"
postgresql - auto mode
link best version is /usr/lib/postgresql10
link currently points to /usr/lib/postgresql10
link postgresql is /usr/lib/postgresql
slave pg_config is /usr/bin/pg_config
slave postgres is /usr/bin/postgres
/usr/lib/postgresql10 - priority 100
slave clusterdb: /usr/lib/postgresql10/bin/clusterdb
slave createdb: /usr/lib/postgresql10/bin/createdb
slave createuser: /usr/lib/postgresql10/bin/createuser
slave dropdb: /usr/lib/postgresql10/bin/dropdb
slave dropuser: /usr/lib/postgresql10/bin/dropuser
slave ecpg: /usr/lib/postgresql10/bin/ecpg
slave initdb: /usr/lib/postgresql10/bin/initdb
slave pg_basebackup: /usr/lib/postgresql10/bin/pg_basebackup
slave pg_config: /usr/lib/postgresql10/bin/pg_config
slave pg_controldata: /usr/lib/postgresql10/bin/pg_controldata
slave pg_ctl: /usr/lib/postgresql10/bin/pg_ctl
slave pg_dump: /usr/lib/postgresql10/bin/pg_dump
slave pg_dumpall: /usr/lib/postgresql10/bin/pg_dumpall
slave pg_isready: /usr/lib/postgresql10/bin/pg_isready
slave pg_receivewal: /usr/lib/postgresql10/bin/pg_receivewal
slave pg_recvlogical: /usr/lib/postgresql10/bin/pg_recvlogical
slave pg_resetwal: /usr/lib/postgresql10/bin/pg_resetwal
slave pg_restore: /usr/lib/postgresql10/bin/pg_restore
slave pg_rewind: /usr/lib/postgresql10/bin/pg_rewind
slave pg_waldump: /usr/lib/postgresql10/bin/pg_waldump
slave postgres: /usr/lib/postgresql10/bin/postgres
slave postmaster: /usr/lib/postgresql10/bin/postmaster
slave psql: /usr/lib/postgresql10/bin/psql
slave reindexdb: /usr/lib/postgresql10/bin/reindexdb
slave vacuumdb: /usr/lib/postgresql10/bin/vacuumdb

rleap15-3:~ # update-alternatives --config postgresql
There is only one alternative in link group postgresql (providing /usr/lib/postgresql): /usr/lib/postgresql10
Nothing to configure.

And removing the older dev package for postgresql10 with unique dependencies via:

zypper remove --clean-deps postgresql10-server-devel

and we no longer have a pg_config

which pg_config
which: no pg_config in (/sbin:/usr/sbin:/usr/local/sbin:/root/.local/bin:/root/bin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin)

But we have unblocked our postgresql10 server updates:

zypper up
...
The following 2 packages are going to be upgraded:
postgresql10 postgresql10-server

@phillxnet
Copy link
Member Author

Thanks to @FroggyFlox for digging out a prior instance of this when Tumbelweed moved to postgresql11 and dropped pg_config in that release, linking for context.
https://forum.rockstor.com/t/build-instructions/5946/9

@phillxnet
Copy link
Member Author

phillxnet commented Oct 27, 2022 •
edited
Loading

Since we pg_config is required for a compile and it has been removed and not replace in postgres10 in the sle update repo we may have not choice but to upgrade out postgresql to what is now current, incidentally from cnf it appears that postresql10-devel in Leap 15.4 has returned pg_config. But we still have 15.3 as our testing base, and currently used due to it's Python2 default inclusion.

zypper in postgresql-server-devel

defaults to
13 for the near end-of-life Leap 15.3
14 for the current 15.4 for which we are making changes such as this issue to catch up with,

So we need at least postgresql12 to get updates with pg_config in (hopefully) but the above defaults suggest we would be better not going for the lowest common denominator of 12. We have an apparent working aarch64 build using system psycopg2 of 2.8.4 and this, from the above changelog pick, includes only 12 error codes map & 2.8.6 adds postgresql13 errorcodes map. However we are caught between a multiple dependencies here and once we have moved to python 3 we can re-visit all. But the OS will still be our database hosts.

@phillxnet
Copy link
Member Author

phillxnet commented Oct 27, 2022 •
edited
Loading

zypper in --no-recommends postgresql13-server-devel

returns us our needed pg_config:

which pg_config
/usr/bin/pg_config

and updates our alternatives to use 13, as we still have a posgres10-server installed, to the new 13. Old enough for psycopg2 2.8.7 to have error maps but this psycopg2 is already a little too new for our current Django (testing branch) max of 2.7.7 but looks to be OK for our use with OS package of psychopg2 2.8.4 - rock-and-several-hard-place until we complete our Python 2 -> 3 move and re-do our build env to be venv based.

rleap15-3:~ # update-alternatives --config postgresql
There are 2 choices for the alternative postgresql (providing /usr/lib/postgresql).

Selection Path Priority Status

  • 0 /usr/lib/postgresql13 130 auto mode
    1 /usr/lib/postgresql10 100 manual mode
    2 /usr/lib/postgresql13 130 manual mode

Press to keep the current choice[*], or type selection number:

phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Oct 27, 2022
@phillxnet
(t) upgrade psycopg2 within current constrains rockstor#2406
a71454a 
Includes related & unrelated previously missing aarch64
path additions to our django environment launcher.
@phillxnet phillxnet mentioned this issue Oct 27, 2022
Merged
phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Oct 27, 2022
@phillxnet
(t) psycopg2 2.8.7 not published on pypi use 2.8.6 rockstor#2406
f589607 
From https://www.psycopg.org/docs/news.html#news we have
a 2.8.7 release with Postgresql 13 errorcodes, but PyPi
https://pypi.org/project/psycopg2/#history
has only 2.8.6 (>= 2.9 requires Python 3).
phillxnet added a commit that referenced this issue Oct 31, 2022
@phillxnet
Merge pull request #2407 from phillxnet/2406_(t)_upgrade_psycopg2_wit…
285534e 
…hin_current_constrains

(t) upgrade psycopg2 within current constrains #2406
@phillxnet
Copy link
Member Author

Closing as:
Fixed by #2407

@phillxnet phillxnet mentioned this issue Nov 7, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@phillxnet