feat(docker): Long-term cache for Docker Hub tags

[zharinov]

Changes

• Store all tags in the package cache for 3 months
• Update cached items incrementally

Context

• Closes: Docker Hub tags - cache and reuse multiple pages #28377

Documentation (please check one with an [x])

• I have updated the documentation, or
• No documentation update is required

How I've tested my work (please select one)

I have verified these changes via:

• Code inspection only, or
• Newly added/modified unit tests, or
• No unit tests but ran on a real repository, or
• Both unit tests + ran on a real repository

[rarkins]

@zharinov can we rethink how this long term caching works?

Ideally we don't want to expire a valid cache ever.

So for example we set a hard expiry limit of X days. I no updates, and no use in X days, then we expire the cache. But if there's any update within the X days, or access within <0.5X left, then we update the cache.

e.g. let's say the cache expiry is 10 days. It's created on day 0, expires day 10.

If on day 1 it's updated, then it now expires on day 11.

Now assume it's not updated, so expiry remains on day 11, and we have reached day 7 and it's accessed. Because <5 days remain, we then push the cache expiry out again to day 17.

In other words we never expire a cache as long as it's accessed approximately once every half X.

I think in your current approach that we'll expire after 90 days if no new tags released in that time?

[zharinov]

I think in your current approach that we'll expire after 90 days if no new tags released in that time?

30 days

@zharinov can we rethink how this long term caching works?

Seems like we have to extend our package cache API to read/write TTL data, otherwise we have to rewrite payload on every access operation.

Relevant commands for Redis:

• https://redis.io/docs/latest/commands/ttl/
• https://redis.io/docs/latest/commands/expire/

Also I could figure out the query for the SQLite backend, however we could do nothing but rewrite for the cacache one.

[rarkins]

I'm ok with rewriting the payload once every 15 days

[zharinov]

Yes, but also I don't want to overcomplicate this logic. How about something like this:

• set 45 to 90 days of TTL for all long-term caches
• implement setTTL for 2 of 3 caching backends, i.e. redis and sqlite
• bump TTL every time the long-term cache record is read
• ignore bumping for cacache-based backend, since 45–90 days TTL is good enough
  • as an alternative, we're okay with payload rewrite on every read for long-term cache records

[zharinov]

Anyway, this should be a separate issue/PR

[rarkins]

I'm trying to test this locally over night/tomorrow with a few scenarios

[renovate-release]

🎉 This PR is included in version 37.308.0 🎉

The release is available on:

• GitHub release
• npm package (@latest dist-tag)
• 37.308.0

Your semantic-release bot 📦🚀