base-job-image: automatic image cleanup

[mivanov1988]

Why

Currently, the base job image contains a lot of binaries/executables (including the package manager) which increases the attack surface of the data job.

What

Integrated docker-slim in order to reduce the number of binaries/executables. Docker-slim analyzes the image and produces a much cleaner one based on the application usage.

Testing done

Local execution of docker-slim and CI/CD - https://gitlab.com/vmware-analytics/versatile-data-kit/-/jobs/3768967625.

Signed-off-by: Miroslav Ivanov [email protected]

[doks5]

Is the gradle jar necessary?

[mivanov1988]

Is the gradle jar necessary?

No, it is not. Thanks!

[antoniivanov]

Local execution of docker-slim and CI/CD - https://gitlab.com/vmware-analytics/versatile-data-kit/-/jobs/3768967625.

If I understand correctly (IIUC) , this tests the creation of the image.
But how can we test its usage ? Basically that control service can use it to build jobs ?

WIth a lot of job-builder images (2 or 3) we need some way to deploy them and run them. Have you thought about that ?

[mivanov1988]

Local execution of docker-slim and CI/CD - https://gitlab.com/vmware-analytics/versatile-data-kit/-/jobs/3768967625.

If I understand correctly (IIUC) , this tests the creation of the image. But how can we test its usage ? Basically that control service can use it to build jobs ?

WIth a lot of job-builder images (2 or 3) we need some way to deploy them and run them. Have you thought about that ?

It does not test the image, it produces a cleaner one. The testing strategy remains the same through the pre_release_test phase.

[antoniivanov]

It does not test the image, it produces a cleaner one. The testing strategy remains the same through the pre_release_test phase.

In pre_release_test we test against deployed version of the control service and that version uses data-job-base-python-3.7 base image and not this one. Which means this image is not tested.

I think we need a strategy for testing it. Until then we should consider the image in development only really (dev version - 0.y.z).

[mivanov1988]

I think we are ready to switch to the secure one.