[infra] Hail should automatically and periodically cleanup unnecessary images

[danking]

What happened?

I have a PR that proposes something roughly along these lines: https://github.com/hail-is/hail/pull/13057/files It has some problems:
1. It should use a cron job.
2. The last time I ran this code, I'm pretty sure I deleted things I shouldn't have. We should audit the find-expired-images.py code again.

We should probably just use https://cloud.google.com/artifact-registry/docs/repositories/cleanup-policy

Version

0.2.120

Relevant log output

No response

[danking]

Our production images are mostly tagged with a deploy- prefix but there are the third-party/images.txt which we need to handle differently.

(base) dking@wm28c-761 gar-cleaner % k get pods -o json | jq -r '.items[].spec.containers[].image' | sort -u
ghost:3.0-alpine
prom/prometheus:v2.34.0
us-docker.pkg.dev/hail-vdc/hail/admin-pod:deploy-qd833uw7kcyn
us-docker.pkg.dev/hail-vdc/hail/auth:deploy-crsithjyoxfg
us-docker.pkg.dev/hail-vdc/hail/batch:deploy-kpd6nqk4t25o
us-docker.pkg.dev/hail-vdc/hail/batch:deploy-v1yv8cgd1003
us-docker.pkg.dev/hail-vdc/hail/blog_nginx:deploy-wnrqjf4h6qto
us-docker.pkg.dev/hail-vdc/hail/ci:deploy-du68h4bouvp9
us-docker.pkg.dev/hail-vdc/hail/envoyproxy/envoy:v1.22.3
us-docker.pkg.dev/hail-vdc/hail/grafana/grafana:9.1.4
us-docker.pkg.dev/hail-vdc/hail/monitoring:deploy-ljz4mgjf132m
us-docker.pkg.dev/hail-vdc/hail/notebook:deploy-gmftvyf0op87
us-docker.pkg.dev/hail-vdc/hail/notebook_nginx:deploy-n9uipfhjn3jg
us-docker.pkg.dev/hail-vdc/hail/website:deploy-gb1372nuge4g

[danking]

Keeping the most recent 10 deploy-m pr- and dev- images seems reasonable. I think we also use the cache- prefix. We maybe should keep 10 each of those?

Anything that's untagged is absolutely good to delete.