Created Timestamp issue with gcr.io repo not detected

[chainlink]

Describe the bug
Flux will not detect Created timestamp for gcr.io repo.

Seeing this error in the logs:

ts=2019-12-17T03:05:02.232839857Z caller=repocachemanager.go:260 component=warmer canonical_name=gcr.io/my/registry auth={map[]} err="failed to parse 1 timestamp label(s) as RFC3339 (org.label-schema.build-date); ask the repository administrator to correct this as it conflicts with the spec" ref=gcr.io/my-registry:a6fb309227fe9f2354cf56eb72e10f60271ceab8

But if I run docker inspect on that image I get:

...
"Created": "2019-12-17T02:43:52.417086026Z"
...

If I run list-images the CREATED column is blank on those images and I see 15 images omitted

Flux version: 1.16.0

I'm currently not using any labels in my docker containers (having trouble adding in build-date label via cloudbuild) I have two other kube installs running flux pulling from gcr.io with no issue (on older flux versions)

[hiddeco]

The error is a warning, and may also be logged while it falls back to the Created timestamp. From what you are reporting, and to be precise

the CREATED column is blank on those images

this fallback seems however not to happen, or the wrong thing is parsed.

Can you share the Config.Labels output from docker inspect <image> with me? Plus the fluxctl version you are using (fluxctl version)?

[chainlink]

Found it! Thanks for the prompt (I should have checked the labels of the docker image before...)

            "Labels": {
                "org.label-schema.build-date": "20191001",
                "org.label-schema.license": "GPLv2",
                "org.label-schema.name": "CentOS Base Image",
                "org.label-schema.schema-version": "1.0",
                "org.label-schema.vendor": "CentOS"
            }

Turns out the base image it setting the label, and in the wrong format to boot. Ugh

[hiddeco]

@chainlink I am glad you consider this issue to be solved, but Flux should still have fallen back to the Created date you originally mentioned.

My suspicion is however that you are using an older fluxctl version, to be precise one before #2594 (<=1.15.0). Can you confirm everything works as expected using fluxctl 1.16.0 or 1.17.0?

[chainlink]

I'm using 1.16.0 on both the client and server. Will try upgrading to 1.17.0

[chainlink]

Upgraded to 1.17.0 and still see the same error in the logs

ts=2019-12-28T19:40:03.352842224Z caller=warming.go:198 component=warmer info="refreshing image" image=gcr.io/repo/repo tag_count=44 to_update=1 of_which_refresh=1 of_which_missing=0
ts=2019-12-28T19:40:03.743939284Z caller=repocachemanager.go:260 component=warmer canonical_name=gcr.io/repo:tag auth={map[]} err="failed to parse 1 timestamp label(s) as RFC3339 (org.label-schema.build-date); ask the repository administrator to correct this as it conflicts with the spec" ref=gcr.io/repo:tag
ts=2019-12-28T19:40:03.745297887Z caller=warming.go:206 component=warmer updated=gcr.io/repo:tag successful=1 attempted=1

fluxctl list-images gives:

...
IMAGE
(18 image(s) omitted)

CREATED
<blank>

I also tried flushing memcached to force a resync and it gives the same behavior

[chainlink]

(Re-opening as this should fallback)

[2opremio]

Assigning this to @hiddeco , as the timestamp expert :)

[chainlink]

Let me know if there is additional information i should provide. I ended up pulling the root docker container and modifying to remove "org.label-schema.build-date" Docker doesn't provide a mechanism to remove labels in downstream images.

[hiddeco]

@chainlink thanks for reopening this.

Is my memory from the base image being some CentOS image correct?

[chainlink]

Yep! https://hub.docker.com/_/centos they actually an (unaddressed) issue for it as well CentOS/sig-cloud-instance-images#153

[hiddeco]

@chainlink thanks for getting back to me, I have put this on my backlog for this week and should be getting back to you soon.

[hiddeco]

@chainlink this was pretty hard to trace down but the devil is in the details.

I think hiddeco/flux:bug-timestamp-label-dd4b80cf plus a memcached cache clear should solve this for you.

[chainlink]

Thanks for tracking this down for me!

[bitsofinfo]

@chainlink @hiddeco i'm running into this issue as well. I'm running 1.18.0 and having the same error in the warmer on new images (which are centos based) reading this, it appears it was fixed? but is this something i need to toggle in a config or something?

[hiddeco]

@bitsofinfo did you clear memcached after upgrading to 1.18.0?

[bitsofinfo]

@hiddeco i've never run anything but 1.18.0, (just started playing w/ flux this week)

[bitsofinfo]

if you want to take it to slack: https://cloud-native.slack.com/archives/CLAJ40HV3/p1582399125009300

[ghost]

The error is a warning, and may also be logged while it falls back to the Created timestamp. From what you are reporting, and to be precise

Is there any way to disable the warning message?

[stretchnate]

Still seeing this on 1.21, will this be fixed in flux v2?