Skip to content

Commit

Permalink
control-service: install necessary dependencies to job builder secure (…
Browse files Browse the repository at this point in the history 
…#2472)

Why
Right now, the base image is lacking many libraries necessary for
installing Python libraries, such as `gcc`.

What
After installing Python libraries, most of these extra libraries are not
required anymore, so we deleted them once the installation was done.
Also, move all commands in a single Docker layer in order to reduce the
image size (~150 MB uncompressed). This helped make the image smaller,
and we also installed the Oracle library only when needed for jobs that
had it listed in their requirements.txt file.

Removed the docker-slim since it does not fit in the new building
strategy. We will improve the image cleanup step in the future.

Testing Done:
Built images locally.

Command:
```
docker buildx build -f versatile-data-kit/projects/control-service/projects/job-builder-secure/Dockerfile.python.vdk -t test-image-tag . --build-arg job_githash=localhsh1 --build-arg base_image=registry.hub.docker.com/versatiledatakit/data-job-base-python-3.8-secure:latest --build-arg job_name="data_job_name" --progress=plain

```
Output:
```
#8 4.754 Installing:
#8 4.754 openssl          x86_64       3.0.9-1.ph5      photon-updates 964.55k   342.59k
#8 4.754 pcre2-libs       x86_64       10.40-3.ph5      photon-updates   1.06M   285.92k
#8 4.754 libselinux       x86_64       3.4-3.ph5        photon-updates 183.63k    82.48k
#8 4.754 gdbm             x86_64       1.23-2.ph5       photon-updates 322.16k   122.51k
#8 4.754 pkg-config       x86_64       0.29.2-3.ph5     photon-updates 614.31k   231.85k
#8 4.754 libffi           x86_64       3.4.2-1.ph5      photon-updates  59.92k    29.36k
#8 4.754 shadow-tools     x86_64       4.13-3.ph5       photon-updates 170.88k    59.31k
#8 4.754 shadow-libs      x86_64       4.13-3.ph5       photon-updates 154.32k    68.57k
#8 4.754 libpwquality     x86_64       1.4.4-3.ph5      photon-updates 379.36k   100.21k
#8 4.754 cracklib         x86_64       2.9.8-1.ph5      photon-updates  44.62k    25.07k
#8 4.754 Linux-PAM        x86_64       1.5.3-1.ph5      photon-updates   1.16M   370.49k
#8 4.754 libffi-devel     x86_64       3.4.2-1.ph5      photon-updates  22.46k    16.76k
#8 4.754 shadow           x86_64       4.13-3.ph5       photon-updates   1.87M   367.96k
#8 4.754 
#8 4.754 Total installed size:   6.93M
#8 4.754 Total download size:   2.05M
#8 5.128 
#8 5.481 
#8 5.742 
#8 6.032 
#8 6.373 
#8 6.589 
#8 6.829 
#8 7.092 
#8 7.375 
#8 7.581 
#8 7.980 
#8 8.210 
#8 8.587 
#8 8.657 Testing transaction
#8 8.679 Running transaction
#8 8.699 Installing/Updating: cracklib-2.9.8-1.ph5.x86_64
#8 8.742 using empty dict to provide pw_dict
#8 8.749 Installing/Updating: libffi-3.4.2-1.ph5.x86_64
#8 8.760 Installing/Updating: pkg-config-0.29.2-3.ph5.x86_64
#8 8.772 Installing/Updating: gdbm-1.23-2.ph5.x86_64
#8 8.785 Installing/Updating: pcre2-libs-10.40-3.ph5.x86_64
#8 8.819 Installing/Updating: libselinux-3.4-3.ph5.x86_64
#8 8.851 Installing/Updating: Linux-PAM-1.5.3-1.ph5.x86_64
#8 8.888 Installing/Updating: shadow-libs-4.13-3.ph5.x86_64
#8 8.900 Installing/Updating: libpwquality-1.4.4-3.ph5.x86_64
#8 8.942 Installing/Updating: openssl-3.0.9-1.ph5.x86_64
#8 8.956 Installing/Updating: shadow-tools-4.13-3.ph5.x86_64
#8 8.967 Installing/Updating: shadow-4.13-3.ph5.x86_64
#8 9.134 Installing/Updating: libffi-devel-3.4.2-1.ph5.x86_64
#8 9.758 + groupadd -r -g 1000 group
#8 9.814 + useradd -u 1000 -g 1000 -r user
#8 9.842 useradd warning: user's uid 1000 is greater than SYS_UID_MAX 999
#8 9.888 + chown -R 1000:1000 /job
#8 9.891 + yum autoremove shadow toybox -y
#8 10.57 
#8 10.57 Installing:
#8 10.57 gmp              x86_64       6.2.1-1.ph5      photon-updates 528.44k   263.77k
#8 10.57 coreutils-selinux x86_64       9.1-5.ph5        photon-updates   6.60M     1.27M
#8 10.57 pcre-libs        x86_64       8.45-3.ph5       photon-updates 275.63k   104.71k
#8 10.57 grep             x86_64       3.7-3.ph5        photon-updates 242.73k   130.24k
#8 10.57 
#8 10.57 Total installed size:   7.62M
#8 10.57 Total download size:   1.75M
#8 10.57 
#8 10.57 Removing:
#8 10.57 toybox           x86_64       0.8.9-4.ph5      @System      407.80k     0.00b
#8 10.57 shadow-tools     x86_64       4.13-3.ph5       @System      170.88k     0.00b
#8 10.57 shadow-libs      x86_64       4.13-3.ph5       @System      154.32k     0.00b
#8 10.57 shadow           x86_64       4.13-3.ph5       @System        1.87M     0.00b
#8 10.57 openssl          x86_64       3.0.9-1.ph5      @System      964.55k     0.00b
#8 10.57 libpwquality     x86_64       1.4.4-3.ph5      @System      379.36k     0.00b
#8 10.57 gdbm             x86_64       1.23-2.ph5       @System      322.16k     0.00b
#8 10.57 cracklib         x86_64       2.9.8-1.ph5      @System       44.62k     0.00b
#8 10.57 Linux-PAM        x86_64       1.5.3-1.ph5      @System        1.16M     0.00b
#8 10.57 
#8 10.57 Total installed size:   5.41M
#8 10.57 Total download size:   0.00b
#8 11.01 
#8 11.53 
#8 11.82 
#8 12.12 
#8 12.13 Testing transaction
#8 12.16 Running transaction
#8 12.17 Installing/Updating: pcre-libs-8.45-3.ph5.x86_64
#8 12.19 Installing/Updating: gmp-6.2.1-1.ph5.x86_64
#8 12.23 Installing/Updating: coreutils-selinux-9.1-5.ph5.x86_64
#8 12.28 Installing/Updating: grep-3.7-3.ph5.x86_64
#8 12.30 Removing: shadow-tools-4.13-3.ph5.x86_64
#8 12.31 Removing: shadow-4.13-3.ph5.x86_64
#8 12.35 Removing: shadow-libs-4.13-3.ph5.x86_64
#8 12.36 Removing: libpwquality-1.4.4-3.ph5.x86_64
#8 12.39 Removing: cracklib-2.9.8-1.ph5.x86_64
#8 12.42 Removing: Linux-PAM-1.5.3-1.ph5.x86_64
#8 12.46 Removing: gdbm-1.23-2.ph5.x86_64
#8 12.77 Removing: toybox-0.8.9-4.ph5.x86_64
#8 12.79 Removing: openssl-3.0.9-1.ph5.x86_64
#8 13.10 + echo 'Removing execute permissions for files within the Data job directory, but not for the directories themselves ...'
#8 13.10 + chmod -R -x+X example8/10_python_step.py example8/README.md example8/config.ini example8/requirements.txt example8/test
#8 13.10 Removing execute permissions for files within the Data job directory, but not for the directories themselves ...
#8 13.10 + grep -q -E '^oracledb|^cx_Oracle' example8/requirements.txt
#8 13.10 + '[' -f example8/requirements.txt ']'
#8 13.10 + echo 'Installing native dependencies ...'
#8 13.10 + yum install shadow build-essential gcc glibc-devel git -y
#8 13.10 Installing native dependencies ...
#8 13.80 
#8 13.80 Installing:
#8 13.80 perl             x86_64       5.36.0-3.ph5     photon-updates  58.90M    15.74M
#8 13.80 tar              x86_64       1.34-3.ph5       photon-updates   4.74M   905.86k
#8 13.80 linux-api-headers noarch       6.1.37-1.ph5     photon-updates   5.86M     1.37M
#8 13.80 openssl          x86_64       3.0.9-1.ph5      photon-updates 964.55k   342.59k
#8 13.80 libgomp          x86_64       12.2.0-1.ph5     photon-updates 295.46k   138.37k
#8 13.80 file-libs        x86_64       5.43-1.ph5       photon-updates   7.75M   354.02k
#8 13.80 gettext          x86_64       0.21.1-1.ph5     photon-updates  12.54M     2.76M
#8 13.80 flex             x86_64       2.6.4-3.ph5      photon-updates 885.18k   232.80k
#8 13.80 binutils-libs    x86_64       2.39-3.ph5       photon-updates   4.30M   951.70k
#8 13.80 m4               x86_64       1.4.19-1.ph5     photon-updates 419.41k   172.34k
#8 13.80 gdbm             x86_64       1.23-2.ph5       photon-updates 322.16k   122.51k
#8 13.80 expat            x86_64       2.5.0-1.ph5      photon-updates  35.55k    21.35k
#8 13.80 mpfr             x86_64       4.1.0-1.ph5      photon-updates   2.69M     1.32M
#8 13.80 mpc              x86_64       1.3.1-1.ph5      photon-updates 448.18k    92.58k
#8 13.80 libstdc++-devel  x86_64       12.2.0-1.ph5     photon-updates  17.19M     2.01M
#8 13.80 libstdc++        x86_64       12.2.0-1.ph5     photon-updates   2.33M   733.69k
#8 13.80 libgomp-devel    x86_64       12.2.0-1.ph5     photon-updates 511.07k   141.92k
#8 13.80 libgcc-devel     x86_64       12.2.0-1.ph5     photon-updates 132.00b     9.93k
#8 13.80 libgcc-atomic    x86_64       12.2.0-1.ph5     photon-updates  36.60k    21.66k
#8 13.80 patch            x86_64       2.7.6-5.ph5      photon-updates 206.46k   110.43k
#8 13.80 make             x86_64       4.3-2.ph5        photon-updates   1.36M   319.80k
#8 13.80 libtool          x86_64       2.4.7-1.ph5      photon-updates   1.29M   197.83k
#8 13.80 gzip             x86_64       1.12-2.ph5       photon-updates 150.04k    77.47k
#8 13.80 gawk             x86_64       5.1.1-2.ph5      photon-updates   2.50M   713.98k
#8 13.80 file             x86_64       5.43-1.ph5       photon-updates  43.89k    34.86k
#8 13.80 diffutils        x86_64       3.8-2.ph5        photon-updates   1.37M   292.09k
#8 13.80 bison            x86_64       3.8.2-3.ph5      photon-updates   2.41M   636.33k
#8 13.80 binutils         x86_64       2.39-3.ph5       photon-updates  27.31M     4.58M
#8 13.80 automake         noarch       1.16.5-1.ph5     photon-updates   1.38M   430.90k
#8 13.80 autoconf         noarch       2.71-2.ph5       photon-updates   1.89M   338.65k
#8 13.80 shadow-tools     x86_64       4.13-3.ph5       photon-updates 170.88k    59.31k
#8 13.80 shadow-libs      x86_64       4.13-3.ph5       photon-updates 154.32k    68.57k
#8 13.80 libpwquality     x86_64       1.4.4-3.ph5      photon-updates 379.36k   100.21k
#8 13.80 cracklib         x86_64       2.9.8-1.ph5      photon-updates  44.62k    25.07k
#8 13.80 Linux-PAM        x86_64       1.5.3-1.ph5      photon-updates   1.16M   370.49k
#8 13.80 git              x86_64       2.39.0-4.ph5     photon-updates  22.90M     3.89M
#8 13.80 glibc-devel      x86_64       2.36-6.ph5       photon-updates  11.98M     1.98M
#8 13.80 gcc              x86_64       12.2.0-1.ph5     photon-updates 195.22M    52.11M
#8 13.80 build-essential  x86_64       0.1-4.ph5        photon-updates   0.00b     6.56k
#8 13.80 shadow           x86_64       4.13-3.ph5       photon-updates   1.87M   367.96k
#8 13.80 
#8 13.80 Total installed size: 393.88M
#8 13.80 Total download size:  93.97M
#8 15.61 
#8 16.05 
#8 16.93 
#8 17.31 
#8 17.61 
#8 17.99 
#8 18.70 
#8 19.04 
#8 19.49 
#8 19.80 
#8 20.10 
#8 20.32 
#8 20.89 
#8 21.17 
#8 21.78 
#8 22.21 
#8 22.52 
#8 22.70 
#8 22.91 
#8 23.19 
#8 23.56 
#8 23.89 
#8 24.16 
#8 24.59 
#8 24.80 
#8 25.17 
#8 25.58 
#8 26.43 
#8 26.83 
#8 27.20 
#8 27.44 
#8 27.72 
#8 28.00 
#8 28.32 
#8 28.69 
#8 29.47 
#8 30.11 
#8 35.40 
#8 35.57 
#8 35.95 
#8 35.97 Testing transaction
#8 36.41 Running transaction
#8 36.78 Installing/Updating: cracklib-2.9.8-1.ph5.x86_64
#8 36.83 using empty dict to provide pw_dict
#8 36.84 Installing/Updating: libstdc++-12.2.0-1.ph5.x86_64
#8 36.85 Installing/Updating: mpfr-4.1.0-1.ph5.x86_64
#8 36.89 Installing/Updating: m4-1.4.19-1.ph5.x86_64
#8 36.90 Installing/Updating: gdbm-1.23-2.ph5.x86_64
#8 36.91 Installing/Updating: Linux-PAM-1.5.3-1.ph5.x86_64
#8 36.95 Installing/Updating: perl-5.36.0-3.ph5.x86_64
#8 37.24 Installing/Updating: shadow-libs-4.13-3.ph5.x86_64
#8 37.25 Installing/Updating: libgomp-12.2.0-1.ph5.x86_64
#8 37.26 Installing/Updating: openssl-3.0.9-1.ph5.x86_64
#8 37.28 Installing/Updating: gettext-0.21.1-1.ph5.x86_64
#8 37.35 Installing/Updating: bison-3.8.2-3.ph5.x86_64
#8 37.38 Installing/Updating: libgomp-devel-12.2.0-1.ph5.x86_64
#8 37.40 Installing/Updating: linux-api-headers-6.1.37-1.ph5.noarch
#8 37.46 Installing/Updating: automake-1.16.5-1.ph5.noarch
#8 37.49 Installing/Updating: autoconf-2.71-2.ph5.noarch
#8 37.50 Installing/Updating: libpwquality-1.4.4-3.ph5.x86_64
#8 37.54 Installing/Updating: shadow-tools-4.13-3.ph5.x86_64
#8 37.55 Installing/Updating: shadow-4.13-3.ph5.x86_64
#8 37.67 Installing/Updating: flex-2.6.4-3.ph5.x86_64
#8 37.70 Installing/Updating: mpc-1.3.1-1.ph5.x86_64
#8 37.73 Installing/Updating: gawk-5.1.1-2.ph5.x86_64
#8 37.76 Installing/Updating: libstdc++-devel-12.2.0-1.ph5.x86_64
#8 37.83 Installing/Updating: glibc-devel-2.36-6.ph5.x86_64
#8 37.88 Installing/Updating: diffutils-3.8-2.ph5.x86_64
#8 37.90 Installing/Updating: gzip-1.12-2.ph5.x86_64
#8 37.91 Installing/Updating: libtool-2.4.7-1.ph5.x86_64
#8 37.95 Installing/Updating: make-4.3-2.ph5.x86_64
#8 37.96 Installing/Updating: patch-2.7.6-5.ph5.x86_64
#8 37.97 Installing/Updating: libgcc-atomic-12.2.0-1.ph5.x86_64
#8 37.98 Installing/Updating: libgcc-devel-12.2.0-1.ph5.x86_64
#8 37.99 Installing/Updating: gcc-12.2.0-1.ph5.x86_64
#8 38.51 Installing/Updating: expat-2.5.0-1.ph5.x86_64
#8 38.54 Installing/Updating: binutils-libs-2.39-3.ph5.x86_64
#8 38.58 Installing/Updating: binutils-2.39-3.ph5.x86_64
#8 38.66 Installing/Updating: file-libs-5.43-1.ph5.x86_64
#8 38.70 Installing/Updating: file-5.43-1.ph5.x86_64
#8 38.71 Installing/Updating: tar-1.34-3.ph5.x86_64
#8 38.75 Installing/Updating: build-essential-0.1-4.ph5.x86_64
#8 38.76 Installing/Updating: git-2.39.0-4.ph5.x86_64
#8 40.59 + echo 'Installing requirements.txt ...'
#8 40.59 + pip3 install --disable-pip-version-check -q -r example8/requirements.txt
#8 40.59 Installing requirements.txt ...
#8 41.66 WARNING: The directory '/job/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
#8 42.83 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
#8 42.88 Removing native dependencies ...
#8 42.88 + echo 'Removing native dependencies ...'
#8 42.88 + yum autoremove shadow build-essential gcc glibc-devel git curl unzip -y
#8 43.69 
#8 43.69 Removing:
#8 43.69 tar              x86_64       1.34-3.ph5       @System        4.74M     0.00b
#8 43.69 shadow-tools     x86_64       4.13-3.ph5       @System      170.88k     0.00b
#8 43.69 shadow-libs      x86_64       4.13-3.ph5       @System      154.32k     0.00b
#8 43.69 shadow           x86_64       4.13-3.ph5       @System        1.87M     0.00b
#8 43.69 perl             x86_64       5.36.0-3.ph5     @System       58.90M     0.00b
#8 43.69 patch            x86_64       2.7.6-5.ph5      @System      206.46k     0.00b
#8 43.69 openssl          x86_64       3.0.9-1.ph5      @System      964.55k     0.00b
#8 43.69 mpfr             x86_64       4.1.0-1.ph5      @System        2.69M     0.00b
#8 43.69 mpc              x86_64       1.3.1-1.ph5      @System      448.18k     0.00b
#8 43.69 make             x86_64       4.3-2.ph5        @System        1.36M     0.00b
#8 43.69 m4               x86_64       1.4.19-1.ph5     @System      419.41k     0.00b
#8 43.69 linux-api-headers noarch       6.1.37-1.ph5     @System        5.86M     0.00b
#8 43.69 libtool          x86_64       2.4.7-1.ph5      @System        1.29M     0.00b
#8 43.69 libstdc++-devel  x86_64       12.2.0-1.ph5     @System       17.19M     0.00b
#8 43.69 libstdc++        x86_64       12.2.0-1.ph5     @System        2.33M     0.00b
#8 43.69 libpwquality     x86_64       1.4.4-3.ph5      @System      379.36k     0.00b
#8 43.69 libgomp-devel    x86_64       12.2.0-1.ph5     @System      511.07k     0.00b
#8 43.69 libgomp          x86_64       12.2.0-1.ph5     @System      295.46k     0.00b
#8 43.69 libgcc-devel     x86_64       12.2.0-1.ph5     @System      132.00b     0.00b
#8 43.69 libgcc-atomic    x86_64       12.2.0-1.ph5     @System       36.60k     0.00b
#8 43.69 gzip             x86_64       1.12-2.ph5       @System      150.04k     0.00b
#8 43.69 glibc-devel      x86_64       2.36-6.ph5       @System       11.98M     0.00b
#8 43.69 git              x86_64       2.39.0-4.ph5     @System       22.90M     0.00b
#8 43.69 gettext          x86_64       0.21.1-1.ph5     @System       12.54M     0.00b
#8 43.69 gdbm             x86_64       1.23-2.ph5       @System      322.16k     0.00b
#8 43.69 gcc              x86_64       12.2.0-1.ph5     @System      195.22M     0.00b
#8 43.69 gawk             x86_64       5.1.1-2.ph5      @System        2.50M     0.00b
#8 43.69 flex             x86_64       2.6.4-3.ph5      @System      885.18k     0.00b
#8 43.69 file-libs        x86_64       5.43-1.ph5       @System        7.75M     0.00b
#8 43.69 file             x86_64       5.43-1.ph5       @System       43.89k     0.00b
#8 43.69 expat            x86_64       2.5.0-1.ph5      @System       35.55k     0.00b
#8 43.69 diffutils        x86_64       3.8-2.ph5        @System        1.37M     0.00b
#8 43.69 curl             x86_64       8.1.2-2.ph5      @System      279.46k     0.00b
#8 43.69 cracklib         x86_64       2.9.8-1.ph5      @System       44.62k     0.00b
#8 43.69 ca-certificates  x86_64       20230315-1.ph5   @System      728.89k     0.00b
#8 43.69 build-essential  x86_64       0.1-4.ph5        @System        0.00b     0.00b
#8 43.69 bison            x86_64       3.8.2-3.ph5      @System        2.41M     0.00b
#8 43.69 binutils-libs    x86_64       2.39-3.ph5       @System        4.30M     0.00b
#8 43.69 binutils         x86_64       2.39-3.ph5       @System       27.31M     0.00b
#8 43.69 automake         noarch       1.16.5-1.ph5     @System        1.38M     0.00b
#8 43.69 autoconf         noarch       2.71-2.ph5       @System        1.89M     0.00b
#8 43.69 Linux-PAM        x86_64       1.5.3-1.ph5      @System        1.16M     0.00b
#8 43.69 
#8 43.69 Total installed size: 394.87M
#8 43.69 Total download size:   0.00b
#8 43.72 Testing transaction
#8 43.80 Running transaction
#8 43.86 Removing: build-essential-0.1-4.ph5.x86_64
#8 43.88 Removing: autoconf-2.71-2.ph5.noarch
#8 43.89 Removing: automake-1.16.5-1.ph5.noarch
#8 43.90 Removing: gcc-12.2.0-1.ph5.x86_64
#8 43.97 Removing: binutils-2.39-3.ph5.x86_64
#8 43.98 Removing: shadow-tools-4.13-3.ph5.x86_64
#8 44.00 Removing: linux-api-headers-6.1.37-1.ph5.noarch
#8 44.02 Removing: shadow-4.13-3.ph5.x86_64
#8 44.06 Removing: shadow-libs-4.13-3.ph5.x86_64
#8 44.07 Removing: libpwquality-1.4.4-3.ph5.x86_64
#8 44.13 Removing: perl-5.36.0-3.ph5.x86_64
#8 44.22 Removing: bison-3.8.2-3.ph5.x86_64
#8 44.24 Removing: gettext-0.21.1-1.ph5.x86_64
#8 44.28 Removing: git-2.39.0-4.ph5.x86_64
#8 44.29 Removing: libgomp-devel-12.2.0-1.ph5.x86_64
#8 44.30 Removing: mpc-1.3.1-1.ph5.x86_64
#8 44.34 Removing: libstdc++-devel-12.2.0-1.ph5.x86_64
#8 44.36 Removing: libstdc++-12.2.0-1.ph5.x86_64
#8 44.37 Removing: libgomp-12.2.0-1.ph5.x86_64
#8 44.38 Removing: cracklib-2.9.8-1.ph5.x86_64
#8 44.41 Removing: binutils-libs-2.39-3.ph5.x86_64
#8 44.45 Removing: libgcc-atomic-12.2.0-1.ph5.x86_64
#8 44.46 Removing: libgcc-devel-12.2.0-1.ph5.x86_64
#8 44.47 Removing: glibc-devel-2.36-6.ph5.x86_64
#8 44.49 Removing: libtool-2.4.7-1.ph5.x86_64
#8 44.53 Removing: file-5.43-1.ph5.x86_64
#8 44.55 Removing: gawk-5.1.1-2.ph5.x86_64
#8 44.56 Removing: curl-8.1.2-2.ph5.x86_64
#8 44.59 Removing: Linux-PAM-1.5.3-1.ph5.x86_64
#8 44.63 Removing: flex-2.6.4-3.ph5.x86_64
#8 44.66 Removing: ca-certificates-20230315-1.ph5.x86_64
#8 44.68 Removing: mpfr-4.1.0-1.ph5.x86_64
#8 44.71 Removing: file-libs-5.43-1.ph5.x86_64
#8 44.74 Removing: m4-1.4.19-1.ph5.x86_64
#8 44.75 Removing: gdbm-1.23-2.ph5.x86_64
#8 44.77 Removing: expat-2.5.0-1.ph5.x86_64
#8 44.80 Removing: openssl-3.0.9-1.ph5.x86_64
#8 44.82 Removing: diffutils-3.8-2.ph5.x86_64
#8 44.84 Removing: gzip-1.12-2.ph5.x86_64
#8 44.86 Removing: make-4.3-2.ph5.x86_64
#8 44.88 Removing: patch-2.7.6-5.ph5.x86_64
#8 44.89 Removing: tar-1.34-3.ph5.x86_64
#8 44.98 Deleting system directories ...
#8 44.98 + echo 'Deleting system directories ...'
#8 44.98 + yum install findutils -y
#8 45.56 
#8 45.56 Installing:
#8 45.56 findutils        x86_64       4.9.0-2.ph5      photon-updates 478.33k   191.20k
#8 45.56 
#8 45.56 Total installed size: 478.33k
#8 45.56 Total download size: 191.20k
#8 45.95 
#8 45.96 Testing transaction
#8 45.98 Running transaction
#8 45.98 Installing/Updating: findutils-4.9.0-2.ph5.x86_64
#8 46.16 + rm -rf /boot /home /media /mnt /root /sbin /srv /var /usr/lib/ldscripts /usr/lib/rpm /usr/lib/sysimage /usr/lib/tdnf /usr/lib/perl5 /usr/lib/gcc /usr/share/locale '/tmp/*' /usr/include /usr/libexec /usr/sbin /usr/libexec
#8 46.16 Deleting system binaries ...
#8 46.16 + echo 'Deleting system binaries ...'
#8 46.16 + python -m pip uninstall pip -y
#8 46.42 WARNING: The directory '/job/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
#8 46.44 Found existing installation: pip 23.1.2
#8 46.48 Uninstalling pip-23.1.2:
#8 46.48   Successfully uninstalled pip-23.1.2
#8 46.50 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
#8 46.55 + cd /usr/local/bin
#8 46.55 + ls
#8 46.55 + grep -xv python
#8 46.55 + grep -xv python3
#8 46.55 + xargs rm -rf
#8 46.55 + cd /usr/bin
#8 46.55 + ls
#8 46.55 + grep -xv sh
#8 46.55 + grep -xv bash
#8 46.55 + xargs rm -rf
#8 DONE 46.6s

#9 exporting to image
#9 exporting layers
#9 exporting layers 0.6s done
#9 writing image sha256:78a6c57c754fe880117447162acb613411039659e707e978a0aca8eed4eb7af8 done
#9 naming to docker.io/library/test-38-r 0.0s done
#9 DONE 0.6s
```



Signed-off-by: Miroslav Ivanov [email protected]

---------

Signed-off-by: Miroslav Ivanov [email protected]
  • Loading branch information
@mivanov1988
mivanov1988 authored Jul 25, 2023
1 parent e2cfea7 commit b7b0e66
Show file tree
Hide file tree
Showing 4 changed files with 42 additions and 76 deletions.
7 changes: 0 additions & 7 deletions projects/control-service/cicd/.gitlab-ci.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -162,13 +162,6 @@ control_service_publish_job_base_image:
- docker login --username "${VDK_DOCKER_REGISTRY_USERNAME}" --password "${VDK_DOCKER_REGISTRY_PASSWORD}" "${VDK_DOCKER_REGISTRY_URL}"
- cd projects/control-service/projects/job-base-image-secure
- export VERSION_TAG="1.$CI_PIPELINE_ID"
# Make the docker-slim tool available to the CI job
- >
apk add -u curl
&& curl -L -o ds.tar.gz https://downloads.dockerslim.com/releases/1.37.3/dist_linux.tar.gz
&& tar -xvf ds.tar.gz
&& mv dist_linux/docker-slim /usr/local/bin/
&& mv dist_linux/docker-slim-sensor /usr/local/bin/
- bash -ex ./publish-job-base-image.sh $PYTHON_MAJOR $PYTHON_MINOR
retry: !reference [.control_service_retry, retry_options]
rules:
Expand Down
21 changes: 0 additions & 21 deletions projects/control-service/projects/job-base-image-secure/Dockerfile-data-job-base
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,24 +7,3 @@ FROM photon:latest
# Copies essential binaries, libraries, headers, and Python files from the base Python image,
# excluding build dependencies.
COPY --from=base /usr/local/ /usr/local/

# Set the working directory
WORKDIR /job

# Uninstall native dependencies
RUN yum erase toybox -y

# Install native dependencies
RUN yum install shadow build-essential -y

# Install the native dependencies necessary for oracledb python library
# See https://www.oracle.com/database/technologies/instant-client/linux-x86-64-downloads.html
RUN set -ex \
&& echo "Installing native dependencies related to support for oracledb python library ..." \
&& mkdir -p /opt/lib/native \
&& yum -y install libaio curl unzip \
&& curl --insecure --output oracle-instantclient.zip https://download.oracle.com/otn_software/linux/instantclient/2110000/instantclient-basic-linux.x64-21.10.0.0.0dbru.zip \
&& unzip oracle-instantclient.zip -d /opt/lib/native/oracle && rm -f oracle-instantclient.zip \
&& sh -c "echo /opt/lib/native/oracle/instantclient_21_10 > /etc/ld.so.conf.d/oracle-instantclient.conf" \
&& ldconfig \
&& yum remove -y curl unzip
19 changes: 0 additions & 19 deletions projects/control-service/projects/job-base-image-secure/publish-job-base-image.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,24 +24,5 @@ data_job_base_image_tag_latest="$data_job_base_image_repo:latest"
docker build -t "$data_job_base_image_tag_local" -f "$SCRIPT_DIR/$data_job_base_docker_file" "$SCRIPT_DIR" \
--build-arg base_image="$python_image_tag_latest"

docker-slim build \
--target "$data_job_base_image_tag_local" \
--tag "$data_job_base_image_tag_version" \
--tag "$data_job_base_image_tag_latest" \
--http-probe=false \
--exec "/bin/sh -c \"pip3 list && python3 -m pip install --upgrade pip\"" \
--include-bin "/usr/bin/chmod" \
--include-bin "/usr/bin/chown" \
--include-bin "/usr/bin/rm" \
--include-bin "/usr/bin/bash" \
--include-bin "/usr/sbin/groupadd" \
--include-bin "/usr/sbin/groupdel" \
--include-bin "/usr/sbin/useradd" \
--include-bin "/usr/sbin/userdel" \
--include-path "/usr/lib" \
--include-path "/usr/local/lib/python$PYTHON_MAJOR.$PYTHON_MINOR/" \
--include-path "/opt/lib/native/oracle"


docker_push_vdk.sh "$data_job_base_image_tag_version"
docker_push_vdk.sh "$data_job_base_image_tag_latest"
71 changes: 42 additions & 29 deletions projects/control-service/projects/job-builder-secure/Dockerfile.python.vdk
View file
Original file line number Diff line number Diff line change
@@ -1,45 +1,58 @@
# https://docs.docker.com/develop/develop-images/dockerfile_best-practices

ARG base_image=versatiledatakit/data-job-base-python-3.10-secure:latest

FROM $base_image

ARG UID=1000
ARG GID=1000
ARG job_name
ARG job_githash
ARG requirements_file=requirements.txt

ENV JOB_NAME $job_name
ENV VDK_JOB_GITHASH $job_githash
ENV HOME=/job

# Set the working directory
WORKDIR /job

# Validate base image is python based
RUN python -V
# Create necessary users and set home directory to /job
RUN groupadd -r -g $GID group && useradd -u $UID -g $GID -r user && chown -R $UID:$GID /job
ENV HOME=/job

# Copy the actual job that has to be executed
ARG job_name
COPY --chown=$UID:$GID $job_name $job_name/

# Remove execute permissions for files within the Data job directory, but not for the directories themselves
RUN chmod -R -x+X $job_name/*

# TODO: this would trigger for any change in job even if requirements.txt does not change
# but there's no COPY_IF_EXISTS command in docker to try copy it.
ARG requirements_file=requirements.txt
RUN if [ -f "$job_name/$requirements_file" ]; then pip3 install --disable-pip-version-check -q -r "$job_name/$requirements_file" || ( echo ">requirements_failed<" && exit 1 ) ; fi

ARG job_githash
ENV JOB_NAME $job_name
ENV VDK_JOB_GITHASH $job_githash

# Delete system executables
RUN rm /usr/bin/chmod
RUN rm /usr/bin/chown
RUN rm /usr/sbin/groupadd
RUN rm /usr/sbin/groupdel
RUN rm /usr/sbin/useradd
RUN rm /usr/sbin/userdel
RUN rm /usr/bin/uname
RUN python -m pip uninstall pip -y
# Install native dependencies
RUN : \
&& set -ex \
&& echo "Validating base image is python based ..." \
&& python -V \
&& echo "Creating necessary users and set home directory to /job ..." \
&& yum install shadow libffi-devel -y && groupadd -r -g $GID group && useradd -u $UID -g $GID -r user && chown -R $UID:$GID /job && yum autoremove shadow toybox -y \
&& echo "Removing execute permissions for files within the Data job directory, but not for the directories themselves ..." \
&& chmod -R -x+X $job_name/* \
&& if grep -q -E "^oracledb|^cx_Oracle" "$job_name/$requirements_file"; then \
echo "Installing native dependencies related to support for oracledb python library ..." \
&& mkdir -p /opt/lib/native \
&& yum -y install libaio unzip \
&& curl --insecure --output oracle-instantclient.zip https://download.oracle.com/otn_software/linux/instantclient/2110000/instantclient-basic-linux.x64-21.10.0.0.0dbru.zip \
&& unzip oracle-instantclient.zip -d /opt/lib/native/oracle && rm -f oracle-instantclient.zip \
&& sh -c "echo /opt/lib/native/oracle/instantclient_21_10 > /etc/ld.so.conf.d/oracle-instantclient.conf" \
&& ldconfig; fi \
&& if [ -f "$job_name/$requirements_file" ]; then \
echo "Installing native dependencies ..." \
&& yum install shadow build-essential gcc glibc-devel git -y \
&& echo "Installing requirements.txt ..." \
&& pip3 install --disable-pip-version-check -q -r "$job_name/$requirements_file" \
|| ( echo ">requirements_failed<" && exit 1 ) \
&& echo "Removing native dependencies ..." \
&& yum autoremove shadow build-essential gcc glibc-devel git curl unzip -y; fi \
&& echo "Deleting system directories ..." \
&& yum install findutils -y \
&& rm -rf /boot /home /media /mnt /root /sbin /srv /var /usr/lib/ldscripts /usr/lib/rpm /usr/lib/sysimage \
/usr/lib/tdnf /usr/lib/perl5 /usr/lib/gcc /usr/share/locale /tmp/* /usr/include /usr/libexec /usr/sbin /usr/libexec \
&& echo "Deleting system binaries ..." \
&& python -m pip uninstall pip -y \
&& cd /usr/local/bin \
&& ls | grep -xv "python" | grep -xv "python3" | xargs rm -rf \
&& cd /usr/bin \
&& ls | grep -xv "sh" | grep -xv "bash" | xargs rm -rf

USER $UID

0 comments on commit b7b0e66

Please sign in to comment.