Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

control-service: install necessary dependencies to job builder secure #2472

Conversation

mivanov1988
Copy link
Collaborator

@mivanov1988 mivanov1988 commented Jul 25, 2023

Why
Right now, the base image is lacking many libraries necessary for installing Python libraries, such as gcc.

What
After installing Python libraries, most of these extra libraries are not required anymore, so we deleted them once the installation was done. Also, move all commands in a single Docker layer in order to reduce the image size (~150 MB uncompressed). This helped make the image smaller, and we also installed the Oracle library only when needed for jobs that had it listed in their requirements.txt file.

Removed the docker-slim since it does not fit in the new building strategy. We will improve the image cleanup step in the future.

Testing Done:
Built images locally.

Command:

docker buildx build -f versatile-data-kit/projects/control-service/projects/job-builder-secure/Dockerfile.python.vdk -t test-image-tag . --build-arg job_githash=localhsh1 --build-arg base_image=registry.hub.docker.com/versatiledatakit/data-job-base-python-3.8-secure:latest --build-arg job_name="data_job_name" --progress=plain

Output:

#8 4.754 Installing:
#8 4.754 openssl          x86_64       3.0.9-1.ph5      photon-updates 964.55k   342.59k
#8 4.754 pcre2-libs       x86_64       10.40-3.ph5      photon-updates   1.06M   285.92k
#8 4.754 libselinux       x86_64       3.4-3.ph5        photon-updates 183.63k    82.48k
#8 4.754 gdbm             x86_64       1.23-2.ph5       photon-updates 322.16k   122.51k
#8 4.754 pkg-config       x86_64       0.29.2-3.ph5     photon-updates 614.31k   231.85k
#8 4.754 libffi           x86_64       3.4.2-1.ph5      photon-updates  59.92k    29.36k
#8 4.754 shadow-tools     x86_64       4.13-3.ph5       photon-updates 170.88k    59.31k
#8 4.754 shadow-libs      x86_64       4.13-3.ph5       photon-updates 154.32k    68.57k
#8 4.754 libpwquality     x86_64       1.4.4-3.ph5      photon-updates 379.36k   100.21k
#8 4.754 cracklib         x86_64       2.9.8-1.ph5      photon-updates  44.62k    25.07k
#8 4.754 Linux-PAM        x86_64       1.5.3-1.ph5      photon-updates   1.16M   370.49k
#8 4.754 libffi-devel     x86_64       3.4.2-1.ph5      photon-updates  22.46k    16.76k
#8 4.754 shadow           x86_64       4.13-3.ph5       photon-updates   1.87M   367.96k
#8 4.754 
#8 4.754 Total installed size:   6.93M
#8 4.754 Total download size:   2.05M
#8 5.128 
#8 5.481 
#8 5.742 
#8 6.032 
#8 6.373 
#8 6.589 
#8 6.829 
#8 7.092 
#8 7.375 
#8 7.581 
#8 7.980 
#8 8.210 
#8 8.587 
#8 8.657 Testing transaction
#8 8.679 Running transaction
#8 8.699 Installing/Updating: cracklib-2.9.8-1.ph5.x86_64
#8 8.742 using empty dict to provide pw_dict
#8 8.749 Installing/Updating: libffi-3.4.2-1.ph5.x86_64
#8 8.760 Installing/Updating: pkg-config-0.29.2-3.ph5.x86_64
#8 8.772 Installing/Updating: gdbm-1.23-2.ph5.x86_64
#8 8.785 Installing/Updating: pcre2-libs-10.40-3.ph5.x86_64
#8 8.819 Installing/Updating: libselinux-3.4-3.ph5.x86_64
#8 8.851 Installing/Updating: Linux-PAM-1.5.3-1.ph5.x86_64
#8 8.888 Installing/Updating: shadow-libs-4.13-3.ph5.x86_64
#8 8.900 Installing/Updating: libpwquality-1.4.4-3.ph5.x86_64
#8 8.942 Installing/Updating: openssl-3.0.9-1.ph5.x86_64
#8 8.956 Installing/Updating: shadow-tools-4.13-3.ph5.x86_64
#8 8.967 Installing/Updating: shadow-4.13-3.ph5.x86_64
#8 9.134 Installing/Updating: libffi-devel-3.4.2-1.ph5.x86_64
#8 9.758 + groupadd -r -g 1000 group
#8 9.814 + useradd -u 1000 -g 1000 -r user
#8 9.842 useradd warning: user's uid 1000 is greater than SYS_UID_MAX 999
#8 9.888 + chown -R 1000:1000 /job
#8 9.891 + yum autoremove shadow toybox -y
#8 10.57 
#8 10.57 Installing:
#8 10.57 gmp              x86_64       6.2.1-1.ph5      photon-updates 528.44k   263.77k
#8 10.57 coreutils-selinux x86_64       9.1-5.ph5        photon-updates   6.60M     1.27M
#8 10.57 pcre-libs        x86_64       8.45-3.ph5       photon-updates 275.63k   104.71k
#8 10.57 grep             x86_64       3.7-3.ph5        photon-updates 242.73k   130.24k
#8 10.57 
#8 10.57 Total installed size:   7.62M
#8 10.57 Total download size:   1.75M
#8 10.57 
#8 10.57 Removing:
#8 10.57 toybox           x86_64       0.8.9-4.ph5      @System      407.80k     0.00b
#8 10.57 shadow-tools     x86_64       4.13-3.ph5       @System      170.88k     0.00b
#8 10.57 shadow-libs      x86_64       4.13-3.ph5       @System      154.32k     0.00b
#8 10.57 shadow           x86_64       4.13-3.ph5       @System        1.87M     0.00b
#8 10.57 openssl          x86_64       3.0.9-1.ph5      @System      964.55k     0.00b
#8 10.57 libpwquality     x86_64       1.4.4-3.ph5      @System      379.36k     0.00b
#8 10.57 gdbm             x86_64       1.23-2.ph5       @System      322.16k     0.00b
#8 10.57 cracklib         x86_64       2.9.8-1.ph5      @System       44.62k     0.00b
#8 10.57 Linux-PAM        x86_64       1.5.3-1.ph5      @System        1.16M     0.00b
#8 10.57 
#8 10.57 Total installed size:   5.41M
#8 10.57 Total download size:   0.00b
#8 11.01 
#8 11.53 
#8 11.82 
#8 12.12 
#8 12.13 Testing transaction
#8 12.16 Running transaction
#8 12.17 Installing/Updating: pcre-libs-8.45-3.ph5.x86_64
#8 12.19 Installing/Updating: gmp-6.2.1-1.ph5.x86_64
#8 12.23 Installing/Updating: coreutils-selinux-9.1-5.ph5.x86_64
#8 12.28 Installing/Updating: grep-3.7-3.ph5.x86_64
#8 12.30 Removing: shadow-tools-4.13-3.ph5.x86_64
#8 12.31 Removing: shadow-4.13-3.ph5.x86_64
#8 12.35 Removing: shadow-libs-4.13-3.ph5.x86_64
#8 12.36 Removing: libpwquality-1.4.4-3.ph5.x86_64
#8 12.39 Removing: cracklib-2.9.8-1.ph5.x86_64
#8 12.42 Removing: Linux-PAM-1.5.3-1.ph5.x86_64
#8 12.46 Removing: gdbm-1.23-2.ph5.x86_64
#8 12.77 Removing: toybox-0.8.9-4.ph5.x86_64
#8 12.79 Removing: openssl-3.0.9-1.ph5.x86_64
#8 13.10 + echo 'Removing execute permissions for files within the Data job directory, but not for the directories themselves ...'
#8 13.10 + chmod -R -x+X example8/10_python_step.py example8/README.md example8/config.ini example8/requirements.txt example8/test
#8 13.10 Removing execute permissions for files within the Data job directory, but not for the directories themselves ...
#8 13.10 + grep -q -E '^oracledb|^cx_Oracle' example8/requirements.txt
#8 13.10 + '[' -f example8/requirements.txt ']'
#8 13.10 + echo 'Installing native dependencies ...'
#8 13.10 + yum install shadow build-essential gcc glibc-devel git -y
#8 13.10 Installing native dependencies ...
#8 13.80 
#8 13.80 Installing:
#8 13.80 perl             x86_64       5.36.0-3.ph5     photon-updates  58.90M    15.74M
#8 13.80 tar              x86_64       1.34-3.ph5       photon-updates   4.74M   905.86k
#8 13.80 linux-api-headers noarch       6.1.37-1.ph5     photon-updates   5.86M     1.37M
#8 13.80 openssl          x86_64       3.0.9-1.ph5      photon-updates 964.55k   342.59k
#8 13.80 libgomp          x86_64       12.2.0-1.ph5     photon-updates 295.46k   138.37k
#8 13.80 file-libs        x86_64       5.43-1.ph5       photon-updates   7.75M   354.02k
#8 13.80 gettext          x86_64       0.21.1-1.ph5     photon-updates  12.54M     2.76M
#8 13.80 flex             x86_64       2.6.4-3.ph5      photon-updates 885.18k   232.80k
#8 13.80 binutils-libs    x86_64       2.39-3.ph5       photon-updates   4.30M   951.70k
#8 13.80 m4               x86_64       1.4.19-1.ph5     photon-updates 419.41k   172.34k
#8 13.80 gdbm             x86_64       1.23-2.ph5       photon-updates 322.16k   122.51k
#8 13.80 expat            x86_64       2.5.0-1.ph5      photon-updates  35.55k    21.35k
#8 13.80 mpfr             x86_64       4.1.0-1.ph5      photon-updates   2.69M     1.32M
#8 13.80 mpc              x86_64       1.3.1-1.ph5      photon-updates 448.18k    92.58k
#8 13.80 libstdc++-devel  x86_64       12.2.0-1.ph5     photon-updates  17.19M     2.01M
#8 13.80 libstdc++        x86_64       12.2.0-1.ph5     photon-updates   2.33M   733.69k
#8 13.80 libgomp-devel    x86_64       12.2.0-1.ph5     photon-updates 511.07k   141.92k
#8 13.80 libgcc-devel     x86_64       12.2.0-1.ph5     photon-updates 132.00b     9.93k
#8 13.80 libgcc-atomic    x86_64       12.2.0-1.ph5     photon-updates  36.60k    21.66k
#8 13.80 patch            x86_64       2.7.6-5.ph5      photon-updates 206.46k   110.43k
#8 13.80 make             x86_64       4.3-2.ph5        photon-updates   1.36M   319.80k
#8 13.80 libtool          x86_64       2.4.7-1.ph5      photon-updates   1.29M   197.83k
#8 13.80 gzip             x86_64       1.12-2.ph5       photon-updates 150.04k    77.47k
#8 13.80 gawk             x86_64       5.1.1-2.ph5      photon-updates   2.50M   713.98k
#8 13.80 file             x86_64       5.43-1.ph5       photon-updates  43.89k    34.86k
#8 13.80 diffutils        x86_64       3.8-2.ph5        photon-updates   1.37M   292.09k
#8 13.80 bison            x86_64       3.8.2-3.ph5      photon-updates   2.41M   636.33k
#8 13.80 binutils         x86_64       2.39-3.ph5       photon-updates  27.31M     4.58M
#8 13.80 automake         noarch       1.16.5-1.ph5     photon-updates   1.38M   430.90k
#8 13.80 autoconf         noarch       2.71-2.ph5       photon-updates   1.89M   338.65k
#8 13.80 shadow-tools     x86_64       4.13-3.ph5       photon-updates 170.88k    59.31k
#8 13.80 shadow-libs      x86_64       4.13-3.ph5       photon-updates 154.32k    68.57k
#8 13.80 libpwquality     x86_64       1.4.4-3.ph5      photon-updates 379.36k   100.21k
#8 13.80 cracklib         x86_64       2.9.8-1.ph5      photon-updates  44.62k    25.07k
#8 13.80 Linux-PAM        x86_64       1.5.3-1.ph5      photon-updates   1.16M   370.49k
#8 13.80 git              x86_64       2.39.0-4.ph5     photon-updates  22.90M     3.89M
#8 13.80 glibc-devel      x86_64       2.36-6.ph5       photon-updates  11.98M     1.98M
#8 13.80 gcc              x86_64       12.2.0-1.ph5     photon-updates 195.22M    52.11M
#8 13.80 build-essential  x86_64       0.1-4.ph5        photon-updates   0.00b     6.56k
#8 13.80 shadow           x86_64       4.13-3.ph5       photon-updates   1.87M   367.96k
#8 13.80 
#8 13.80 Total installed size: 393.88M
#8 13.80 Total download size:  93.97M
#8 15.61 
#8 16.05 
#8 16.93 
#8 17.31 
#8 17.61 
#8 17.99 
#8 18.70 
#8 19.04 
#8 19.49 
#8 19.80 
#8 20.10 
#8 20.32 
#8 20.89 
#8 21.17 
#8 21.78 
#8 22.21 
#8 22.52 
#8 22.70 
#8 22.91 
#8 23.19 
#8 23.56 
#8 23.89 
#8 24.16 
#8 24.59 
#8 24.80 
#8 25.17 
#8 25.58 
#8 26.43 
#8 26.83 
#8 27.20 
#8 27.44 
#8 27.72 
#8 28.00 
#8 28.32 
#8 28.69 
#8 29.47 
#8 30.11 
#8 35.40 
#8 35.57 
#8 35.95 
#8 35.97 Testing transaction
#8 36.41 Running transaction
#8 36.78 Installing/Updating: cracklib-2.9.8-1.ph5.x86_64
#8 36.83 using empty dict to provide pw_dict
#8 36.84 Installing/Updating: libstdc++-12.2.0-1.ph5.x86_64
#8 36.85 Installing/Updating: mpfr-4.1.0-1.ph5.x86_64
#8 36.89 Installing/Updating: m4-1.4.19-1.ph5.x86_64
#8 36.90 Installing/Updating: gdbm-1.23-2.ph5.x86_64
#8 36.91 Installing/Updating: Linux-PAM-1.5.3-1.ph5.x86_64
#8 36.95 Installing/Updating: perl-5.36.0-3.ph5.x86_64
#8 37.24 Installing/Updating: shadow-libs-4.13-3.ph5.x86_64
#8 37.25 Installing/Updating: libgomp-12.2.0-1.ph5.x86_64
#8 37.26 Installing/Updating: openssl-3.0.9-1.ph5.x86_64
#8 37.28 Installing/Updating: gettext-0.21.1-1.ph5.x86_64
#8 37.35 Installing/Updating: bison-3.8.2-3.ph5.x86_64
#8 37.38 Installing/Updating: libgomp-devel-12.2.0-1.ph5.x86_64
#8 37.40 Installing/Updating: linux-api-headers-6.1.37-1.ph5.noarch
#8 37.46 Installing/Updating: automake-1.16.5-1.ph5.noarch
#8 37.49 Installing/Updating: autoconf-2.71-2.ph5.noarch
#8 37.50 Installing/Updating: libpwquality-1.4.4-3.ph5.x86_64
#8 37.54 Installing/Updating: shadow-tools-4.13-3.ph5.x86_64
#8 37.55 Installing/Updating: shadow-4.13-3.ph5.x86_64
#8 37.67 Installing/Updating: flex-2.6.4-3.ph5.x86_64
#8 37.70 Installing/Updating: mpc-1.3.1-1.ph5.x86_64
#8 37.73 Installing/Updating: gawk-5.1.1-2.ph5.x86_64
#8 37.76 Installing/Updating: libstdc++-devel-12.2.0-1.ph5.x86_64
#8 37.83 Installing/Updating: glibc-devel-2.36-6.ph5.x86_64
#8 37.88 Installing/Updating: diffutils-3.8-2.ph5.x86_64
#8 37.90 Installing/Updating: gzip-1.12-2.ph5.x86_64
#8 37.91 Installing/Updating: libtool-2.4.7-1.ph5.x86_64
#8 37.95 Installing/Updating: make-4.3-2.ph5.x86_64
#8 37.96 Installing/Updating: patch-2.7.6-5.ph5.x86_64
#8 37.97 Installing/Updating: libgcc-atomic-12.2.0-1.ph5.x86_64
#8 37.98 Installing/Updating: libgcc-devel-12.2.0-1.ph5.x86_64
#8 37.99 Installing/Updating: gcc-12.2.0-1.ph5.x86_64
#8 38.51 Installing/Updating: expat-2.5.0-1.ph5.x86_64
#8 38.54 Installing/Updating: binutils-libs-2.39-3.ph5.x86_64
#8 38.58 Installing/Updating: binutils-2.39-3.ph5.x86_64
#8 38.66 Installing/Updating: file-libs-5.43-1.ph5.x86_64
#8 38.70 Installing/Updating: file-5.43-1.ph5.x86_64
#8 38.71 Installing/Updating: tar-1.34-3.ph5.x86_64
#8 38.75 Installing/Updating: build-essential-0.1-4.ph5.x86_64
#8 38.76 Installing/Updating: git-2.39.0-4.ph5.x86_64
#8 40.59 + echo 'Installing requirements.txt ...'
#8 40.59 + pip3 install --disable-pip-version-check -q -r example8/requirements.txt
#8 40.59 Installing requirements.txt ...
#8 41.66 WARNING: The directory '/job/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
#8 42.83 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
#8 42.88 Removing native dependencies ...
#8 42.88 + echo 'Removing native dependencies ...'
#8 42.88 + yum autoremove shadow build-essential gcc glibc-devel git curl unzip -y
#8 43.69 
#8 43.69 Removing:
#8 43.69 tar              x86_64       1.34-3.ph5       @System        4.74M     0.00b
#8 43.69 shadow-tools     x86_64       4.13-3.ph5       @System      170.88k     0.00b
#8 43.69 shadow-libs      x86_64       4.13-3.ph5       @System      154.32k     0.00b
#8 43.69 shadow           x86_64       4.13-3.ph5       @System        1.87M     0.00b
#8 43.69 perl             x86_64       5.36.0-3.ph5     @System       58.90M     0.00b
#8 43.69 patch            x86_64       2.7.6-5.ph5      @System      206.46k     0.00b
#8 43.69 openssl          x86_64       3.0.9-1.ph5      @System      964.55k     0.00b
#8 43.69 mpfr             x86_64       4.1.0-1.ph5      @System        2.69M     0.00b
#8 43.69 mpc              x86_64       1.3.1-1.ph5      @System      448.18k     0.00b
#8 43.69 make             x86_64       4.3-2.ph5        @System        1.36M     0.00b
#8 43.69 m4               x86_64       1.4.19-1.ph5     @System      419.41k     0.00b
#8 43.69 linux-api-headers noarch       6.1.37-1.ph5     @System        5.86M     0.00b
#8 43.69 libtool          x86_64       2.4.7-1.ph5      @System        1.29M     0.00b
#8 43.69 libstdc++-devel  x86_64       12.2.0-1.ph5     @System       17.19M     0.00b
#8 43.69 libstdc++        x86_64       12.2.0-1.ph5     @System        2.33M     0.00b
#8 43.69 libpwquality     x86_64       1.4.4-3.ph5      @System      379.36k     0.00b
#8 43.69 libgomp-devel    x86_64       12.2.0-1.ph5     @System      511.07k     0.00b
#8 43.69 libgomp          x86_64       12.2.0-1.ph5     @System      295.46k     0.00b
#8 43.69 libgcc-devel     x86_64       12.2.0-1.ph5     @System      132.00b     0.00b
#8 43.69 libgcc-atomic    x86_64       12.2.0-1.ph5     @System       36.60k     0.00b
#8 43.69 gzip             x86_64       1.12-2.ph5       @System      150.04k     0.00b
#8 43.69 glibc-devel      x86_64       2.36-6.ph5       @System       11.98M     0.00b
#8 43.69 git              x86_64       2.39.0-4.ph5     @System       22.90M     0.00b
#8 43.69 gettext          x86_64       0.21.1-1.ph5     @System       12.54M     0.00b
#8 43.69 gdbm             x86_64       1.23-2.ph5       @System      322.16k     0.00b
#8 43.69 gcc              x86_64       12.2.0-1.ph5     @System      195.22M     0.00b
#8 43.69 gawk             x86_64       5.1.1-2.ph5      @System        2.50M     0.00b
#8 43.69 flex             x86_64       2.6.4-3.ph5      @System      885.18k     0.00b
#8 43.69 file-libs        x86_64       5.43-1.ph5       @System        7.75M     0.00b
#8 43.69 file             x86_64       5.43-1.ph5       @System       43.89k     0.00b
#8 43.69 expat            x86_64       2.5.0-1.ph5      @System       35.55k     0.00b
#8 43.69 diffutils        x86_64       3.8-2.ph5        @System        1.37M     0.00b
#8 43.69 curl             x86_64       8.1.2-2.ph5      @System      279.46k     0.00b
#8 43.69 cracklib         x86_64       2.9.8-1.ph5      @System       44.62k     0.00b
#8 43.69 ca-certificates  x86_64       20230315-1.ph5   @System      728.89k     0.00b
#8 43.69 build-essential  x86_64       0.1-4.ph5        @System        0.00b     0.00b
#8 43.69 bison            x86_64       3.8.2-3.ph5      @System        2.41M     0.00b
#8 43.69 binutils-libs    x86_64       2.39-3.ph5       @System        4.30M     0.00b
#8 43.69 binutils         x86_64       2.39-3.ph5       @System       27.31M     0.00b
#8 43.69 automake         noarch       1.16.5-1.ph5     @System        1.38M     0.00b
#8 43.69 autoconf         noarch       2.71-2.ph5       @System        1.89M     0.00b
#8 43.69 Linux-PAM        x86_64       1.5.3-1.ph5      @System        1.16M     0.00b
#8 43.69 
#8 43.69 Total installed size: 394.87M
#8 43.69 Total download size:   0.00b
#8 43.72 Testing transaction
#8 43.80 Running transaction
#8 43.86 Removing: build-essential-0.1-4.ph5.x86_64
#8 43.88 Removing: autoconf-2.71-2.ph5.noarch
#8 43.89 Removing: automake-1.16.5-1.ph5.noarch
#8 43.90 Removing: gcc-12.2.0-1.ph5.x86_64
#8 43.97 Removing: binutils-2.39-3.ph5.x86_64
#8 43.98 Removing: shadow-tools-4.13-3.ph5.x86_64
#8 44.00 Removing: linux-api-headers-6.1.37-1.ph5.noarch
#8 44.02 Removing: shadow-4.13-3.ph5.x86_64
#8 44.06 Removing: shadow-libs-4.13-3.ph5.x86_64
#8 44.07 Removing: libpwquality-1.4.4-3.ph5.x86_64
#8 44.13 Removing: perl-5.36.0-3.ph5.x86_64
#8 44.22 Removing: bison-3.8.2-3.ph5.x86_64
#8 44.24 Removing: gettext-0.21.1-1.ph5.x86_64
#8 44.28 Removing: git-2.39.0-4.ph5.x86_64
#8 44.29 Removing: libgomp-devel-12.2.0-1.ph5.x86_64
#8 44.30 Removing: mpc-1.3.1-1.ph5.x86_64
#8 44.34 Removing: libstdc++-devel-12.2.0-1.ph5.x86_64
#8 44.36 Removing: libstdc++-12.2.0-1.ph5.x86_64
#8 44.37 Removing: libgomp-12.2.0-1.ph5.x86_64
#8 44.38 Removing: cracklib-2.9.8-1.ph5.x86_64
#8 44.41 Removing: binutils-libs-2.39-3.ph5.x86_64
#8 44.45 Removing: libgcc-atomic-12.2.0-1.ph5.x86_64
#8 44.46 Removing: libgcc-devel-12.2.0-1.ph5.x86_64
#8 44.47 Removing: glibc-devel-2.36-6.ph5.x86_64
#8 44.49 Removing: libtool-2.4.7-1.ph5.x86_64
#8 44.53 Removing: file-5.43-1.ph5.x86_64
#8 44.55 Removing: gawk-5.1.1-2.ph5.x86_64
#8 44.56 Removing: curl-8.1.2-2.ph5.x86_64
#8 44.59 Removing: Linux-PAM-1.5.3-1.ph5.x86_64
#8 44.63 Removing: flex-2.6.4-3.ph5.x86_64
#8 44.66 Removing: ca-certificates-20230315-1.ph5.x86_64
#8 44.68 Removing: mpfr-4.1.0-1.ph5.x86_64
#8 44.71 Removing: file-libs-5.43-1.ph5.x86_64
#8 44.74 Removing: m4-1.4.19-1.ph5.x86_64
#8 44.75 Removing: gdbm-1.23-2.ph5.x86_64
#8 44.77 Removing: expat-2.5.0-1.ph5.x86_64
#8 44.80 Removing: openssl-3.0.9-1.ph5.x86_64
#8 44.82 Removing: diffutils-3.8-2.ph5.x86_64
#8 44.84 Removing: gzip-1.12-2.ph5.x86_64
#8 44.86 Removing: make-4.3-2.ph5.x86_64
#8 44.88 Removing: patch-2.7.6-5.ph5.x86_64
#8 44.89 Removing: tar-1.34-3.ph5.x86_64
#8 44.98 Deleting system directories ...
#8 44.98 + echo 'Deleting system directories ...'
#8 44.98 + yum install findutils -y
#8 45.56 
#8 45.56 Installing:
#8 45.56 findutils        x86_64       4.9.0-2.ph5      photon-updates 478.33k   191.20k
#8 45.56 
#8 45.56 Total installed size: 478.33k
#8 45.56 Total download size: 191.20k
#8 45.95 
#8 45.96 Testing transaction
#8 45.98 Running transaction
#8 45.98 Installing/Updating: findutils-4.9.0-2.ph5.x86_64
#8 46.16 + rm -rf /boot /home /media /mnt /root /sbin /srv /var /usr/lib/ldscripts /usr/lib/rpm /usr/lib/sysimage /usr/lib/tdnf /usr/lib/perl5 /usr/lib/gcc /usr/share/locale '/tmp/*' /usr/include /usr/libexec /usr/sbin /usr/libexec
#8 46.16 Deleting system binaries ...
#8 46.16 + echo 'Deleting system binaries ...'
#8 46.16 + python -m pip uninstall pip -y
#8 46.42 WARNING: The directory '/job/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
#8 46.44 Found existing installation: pip 23.1.2
#8 46.48 Uninstalling pip-23.1.2:
#8 46.48   Successfully uninstalled pip-23.1.2
#8 46.50 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
#8 46.55 + cd /usr/local/bin
#8 46.55 + ls
#8 46.55 + grep -xv python
#8 46.55 + grep -xv python3
#8 46.55 + xargs rm -rf
#8 46.55 + cd /usr/bin
#8 46.55 + ls
#8 46.55 + grep -xv sh
#8 46.55 + grep -xv bash
#8 46.55 + xargs rm -rf
#8 DONE 46.6s

#9 exporting to image
#9 exporting layers
#9 exporting layers 0.6s done
#9 writing image sha256:78a6c57c754fe880117447162acb613411039659e707e978a0aca8eed4eb7af8 done
#9 naming to docker.io/library/test-38-r 0.0s done
#9 DONE 0.6s

Signed-off-by: Miroslav Ivanov [email protected]

Why
Right now, the starting image is lacking many libraries necessary for installing Python libraries,
such as `gcc`.

What
After installing Python libraries, most of these extra libraries are not required anymore,
so we deleted them once the installation was done. Also move all commands in single layer
in order to reduce the image size (~150 MB uncompressed). This helped make the image smaller,
and we also installed the Oracle library only when needed for jobs that had it listed
in their requirements.txt file.

Testing Done:
Built images locally.

Signed-off-by: Miroslav Ivanov [email protected]
Why
Right now, the starting image is lacking many libraries necessary for installing Python libraries,
such as `gcc`.

What
After installing Python libraries, most of these extra libraries are not required anymore,
so we deleted them once the installation was done. Also move all commands in single layer
in order to reduce the image size (~150 MB uncompressed). This helped make the image smaller,
and we also installed the Oracle library only when needed for jobs that had it listed
in their requirements.txt file.

Testing Done:
Built images locally.

Signed-off-by: Miroslav Ivanov [email protected]
Copy link
Collaborator

@antoniivanov antoniivanov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Built images locally.

Please specify how ere those images built (what commands) and how was it verified (the snippet of log or output relevant for the verification).

if I need to reproduce the test, I should be able to make it only from those instructions.

Please imporve the testing done description. But otherwise looks good to me.

@mivanov1988
Copy link
Collaborator Author

Built images locally.

Please specify how ere those images built (what commands) and how was it verified (the snippet of log or output relevant for the verification).

if I need to reproduce the test, I should be able to make it only from those instructions.

Please imporve the testing done description. But otherwise looks good to me.

Updated Testing Done section.

@mivanov1988 mivanov1988 enabled auto-merge (squash) July 25, 2023 12:39
@mivanov1988 mivanov1988 merged commit b7b0e66 into main Jul 25, 2023
@mivanov1988 mivanov1988 deleted the person/miroslavi/install-necessary-dependencies-to-job-builder-secure branch July 25, 2023 12:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants