fix(k8s_tagger) Removing k8s tagger warnings, as these have been causing false alarms and cost concerns

[rnishtala-sumo]

Removing k8s tagger warnings, as these have been causing false alarms and cost concerns

[echlebek]

@rnishtala-sumo is it worth making these debug level instead of removing them? Do they carry any debugging value?

[rnishtala-sumo]

@echlebek they only indicate that the cache doesn't yet have expected entry. In most cases it gets there eventually. Having said that, Debug is probably a better option.

[echlebek]

Seems like it would only be useful to say how long the cache has been out of sync for, or maybe only log if it's been out of sync for a much longer period than expected. That said, this is still an improvement. 👍

[chan-tim-sumo]

Should this go down to Info level from warn level instead of going to debug? because i'm thinking this still runs in production right? and if i remember correctly, debug is disabled in production 🤔

[rnishtala-sumo]

@chan-tim-sumo The goal is to not see them in prod by default. But have the option of enabling them. Debug is used for such use cases.