Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disable unixodbc and related packages from Microsoft APT repo #29661

Closed

Conversation

Taragolis
Copy link
Contributor

@Taragolis Taragolis commented Feb 21, 2023

Disable unixodbc and related packages from https://packages.microsoft.com/ and use packages from Debian repo instead.

Update, forgot attach output from local built container

docker run --rm -ti ghcr.io/apache/airflow/main/ci/python3.7:latest

...

root@95333f230202:/opt/airflow# apt update
Get:1 http://deb.debian.org/debian bullseye InRelease [116 kB]
Get:2 http://deb.debian.org/debian-security bullseye-security InRelease [48.4 kB]           
Get:3 http://deb.debian.org/debian bullseye-updates InRelease [44.1 kB]                                                 
Get:4 http://deb.debian.org/debian bullseye/main arm64 Packages [8072 kB]                                               
Get:5 https://apt.postgresql.org/pub/repos/apt bullseye-pgdg InRelease [91.7 kB]                                        
Get:6 https://packages.microsoft.com/debian/11/prod bullseye InRelease [10.5 kB]                                        
Get:7 https://apt.postgresql.org/pub/repos/apt bullseye-pgdg/main arm64 Packages [262 kB]               
Get:8 https://packages.microsoft.com/debian/11/prod bullseye/main arm64 Packages [26.8 kB]
Get:9 https://packages.microsoft.com/debian/11/prod bullseye/main armhf Packages [27.2 kB]                       
Get:10 https://packages.microsoft.com/debian/11/prod bullseye/main amd64 Packages [117 kB]                      
Get:11 http://deb.debian.org/debian-security bullseye-security/main arm64 Packages [224 kB]         
Get:12 http://deb.debian.org/debian bullseye-updates/main arm64 Packages [12.0 kB]        
Fetched 9051 kB in 4s (2334 kB/s)                                                             
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
1 package can be upgraded. Run 'apt list --upgradable' to see it.

root@95333f230202:/opt/airflow# apt list --upgradable
Listing... Done
postgresql-client-common/bullseye-pgdg 247.pgdg110+1 all [upgradable from: 225]
N: There is 1 additional version. Please use the '-a' switch to see it

root@95333f230202:/opt/airflow# apt-cache policy
Package files:
 100 /var/lib/dpkg/status
     release a=now
 500 https://apt.postgresql.org/pub/repos/apt bullseye-pgdg/main arm64 Packages
     release o=apt.postgresql.org,a=bullseye-pgdg,n=bullseye-pgdg,l=PostgreSQL for Debian/Ubuntu repository,c=main,b=arm64
     origin apt.postgresql.org
 500 https://packages.microsoft.com/debian/11/prod bullseye/main arm64 Packages
     release o=microsoft-debian-bullseye-prod bullseye,a=bullseye,n=bullseye,l=microsoft-debian-bullseye-prod bullseye,c=main,b=arm64
     origin packages.microsoft.com
 500 https://packages.microsoft.com/debian/11/prod bullseye/main armhf Packages
     release o=microsoft-debian-bullseye-prod bullseye,a=bullseye,n=bullseye,l=microsoft-debian-bullseye-prod bullseye,c=main,b=armhf
     origin packages.microsoft.com
 500 https://packages.microsoft.com/debian/11/prod bullseye/main amd64 Packages
     release o=microsoft-debian-bullseye-prod bullseye,a=bullseye,n=bullseye,l=microsoft-debian-bullseye-prod bullseye,c=main,b=amd64
     origin packages.microsoft.com
 500 http://deb.debian.org/debian bullseye-updates/main arm64 Packages
     release v=11-updates,o=Debian,a=stable-updates,n=bullseye-updates,l=Debian,c=main,b=arm64
     origin deb.debian.org
 500 http://deb.debian.org/debian-security bullseye-security/main arm64 Packages
     release v=11,o=Debian,a=stable-security,n=bullseye-security,l=Debian-Security,c=main,b=arm64
     origin deb.debian.org
 500 http://deb.debian.org/debian bullseye/main arm64 Packages
     release v=11.6,o=Debian,a=stable,n=bullseye,l=Debian,c=main,b=arm64
     origin deb.debian.org
Pinned packages:
     libodbc1 -> 2.3.11 with priority -1
     odbcinst -> 2.3.11 with priority -1
     odbcinst1debian2 -> 2.3.11 with priority -1
     unixodbc-dev -> 2.3.11 with priority -1
     unixodbc -> 2.3.11 with priority -1

root@95333f230202:/opt/airflow# apt-cache policy unixodbc-dev
unixodbc-dev:
  Installed: 2.3.6-0.1+b1
  Candidate: 2.3.6-0.1+b1
  Version table:
     2.3.11 -1
        500 https://packages.microsoft.com/debian/11/prod bullseye/main arm64 Packages
 *** 2.3.6-0.1+b1 500
        500 http://deb.debian.org/debian bullseye/main arm64 Packages
        100 /var/lib/dpkg/status

@boring-cyborg boring-cyborg bot added area:dev-tools area:production-image Production image improvements and fixes labels Feb 21, 2023
@Taragolis Taragolis force-pushed the disable-unixodb-from-ms-repo branch from f438889 to d31e03f Compare February 21, 2023 18:07
@Taragolis
Copy link
Contributor Author

For comparison output from latest official Airflow Docker. By default PROD image also contains Debian's packages, however if user upgrade packages then unixodbc related packages will upgrade to packages from Microsoft repo

docker run -it --rm --user root apache/airflow:2.5.1-python3.9 bash
The container is run as root user. For security, consider using a regular user account.

root@ed86d0174f98:/opt/airflow# dpkg --list | grep -E "(libodbc1|odbcinst1debian2|odbcinst|unixodbc-dev|unixodbc)"
ii  libodbc1:arm64             2.3.6-0.1+b1                   arm64        ODBC library for Unix
ii  odbcinst                   2.3.6-0.1+b1                   arm64        Helper program for accessing odbc ini files
ii  odbcinst1debian2:arm64     2.3.6-0.1+b1                   arm64        Support library for accessing odbc ini files
ii  unixodbc                   2.3.6-0.1+b1                   arm64        Basic ODBC tools

root@ed86d0174f98:/opt/airflow# apt update
...                          
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
16 packages can be upgraded. Run 'apt list --upgradable' to see them.

root@ed86d0174f98:/opt/airflow# apt list --upgradable
Listing... Done
bind9-host/stable-security 1:9.16.37-1~deb11u1 arm64 [upgradable from: 1:9.16.33-1~deb11u1]
bind9-libs/stable-security 1:9.16.37-1~deb11u1 arm64 [upgradable from: 1:9.16.33-1~deb11u1]
curl/stable-security 7.74.0-1.3+deb11u5 arm64 [upgradable from: 7.74.0-1.3+deb11u3]
libcurl4/stable-security 7.74.0-1.3+deb11u5 arm64 [upgradable from: 7.74.0-1.3+deb11u3]
libgnutls30/stable-security 3.7.1-5+deb11u3 arm64 [upgradable from: 3.7.1-5+deb11u2]
libodbc1/bullseye 2.3.11 arm64 [upgradable from: 2.3.6-0.1+b1]
libpq5/bullseye-pgdg 15.2-1.pgdg110+1 arm64 [upgradable from: 15.1-1.pgdg110+1]
libssl1.1/stable-security 1.1.1n-0+deb11u4 arm64 [upgradable from: 1.1.1n-0+deb11u3]
msodbcsql18/bullseye 18.2.1.1-1 arm64 [upgradable from: 18.1.2.1-1]
odbcinst1debian2/bullseye 2.3.11 arm64 [upgradable from: 2.3.6-0.1+b1]
odbcinst/bullseye 2.3.11 arm64 [upgradable from: 2.3.6-0.1+b1]
openssl/stable-security 1.1.1n-0+deb11u4 arm64 [upgradable from: 1.1.1n-0+deb11u3]
postgresql-client-15/bullseye-pgdg 15.2-1.pgdg110+1 arm64 [upgradable from: 15.1-1.pgdg110+1]
postgresql-client-common/bullseye-pgdg 247.pgdg110+1 all [upgradable from: 246.pgdg110+1]
postgresql-client/bullseye-pgdg 15+247.pgdg110+1 all [upgradable from: 15+246.pgdg110+1]
unixodbc/bullseye 2.3.11 arm64 [upgradable from: 2.3.6-0.1+b1]

root@ed86d0174f98:/opt/airflow# apt-cache policy unixodbc
unixodbc:
  Installed: 2.3.6-0.1+b1
  Candidate: 2.3.11
  Version table:
     2.3.11 500
        500 https://packages.microsoft.com/debian/11/prod bullseye/main arm64 Packages
 *** 2.3.6-0.1+b1 500
        500 http://deb.debian.org/debian bullseye/main arm64 Packages
        100 /var/lib/dpkg/status

@Taragolis
Copy link
Contributor Author

At the moment when you decide not to wait anymore

microsoft/linux-package-repositories#36 (comment) 🤣 🤣

Closing as the original issue (unixodbc.h) has been resolved.
For issues with libodbc1 vs libodbc2, refer to microsoft/linux-package-repositories#39.

@potiuk
Copy link
Member

potiuk commented Feb 22, 2023

At the moment when you decide not to wait anymore

Always. No exceptions.

@Taragolis
Copy link
Contributor Author

I don't know potentially this is PR still relevant due to libodbc1 vs libodbc2.
Or we could close this PR and return when (if) the error happen and we will require libodbc2 in Airflow Docker Image

@Taragolis
Copy link
Contributor Author

Let's close until it really needed

@Taragolis Taragolis closed this Feb 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:dev-tools area:production-image Production image improvements and fixes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants