Operator logs in unix epoch timestamps but would prefer iso8601/rfc3339

[jordansissel]

% kubectl logs -n elastic-system elastic-operator-0
{"level":"info","ts":1565963733.0098941,
{"level":"info","ts":1565963733.009926,"
{"level":"info","ts":1565963733.0167916,
{"level":"info","ts":1565963733.016846,"
{"level":"info","ts":1565963743.017154,"
{"level":"info","ts":1565963743.0190873,
{"level":"info","ts":1565963743.0193639,
{"level":"info","ts":1565963743.844426,"

I'd love if these timestamps were human readable. We can achieve that while still making them machine readable by logging in ISO8601/RFC3339 format.

^^ The above logs are from ECK 0.8.1

[jordansissel]

Possibly out of scope, but this might be a chance to move to log in ECS format as well. I dug around the code a bit, and it seems like ECK uses controller-runtime's logger which is based on uber's Zap. If this is right, I have an ECS-friendly log configuration for it that I use in other projects:

  config := zap.NewProductionConfig()

  // Get as close as we can to ECS format
  config.EncoderConfig.LevelKey = "log.level"
  config.EncoderConfig.NameKey = "event.category"
  config.EncoderConfig.TimeKey = "@timestamp"
  config.EncoderConfig.EncodeTime = zapcore.ISO8601TimeEncoder
  logger, err = config.Build()

The logger variable above is a zap.Logger and should be something you can pass here: to logf.SetLogger(...)

[jordansissel]

In the meantime, here's a workaround to convert the epoch times to iso8601 with awk:

% kubectl logs --tail 1 --follow -n elastic-system elastic-operator-0 | awk -F, 'BEGIN { OFS=FS } {split($2,a,":"); split(a[2], s, "."); $2 = sprintf("\"ts\":\"%s.%d%s\"", strftime("%Y-%m-%dT%H:%M:%S", a[2]), s[2], strftime("%z", a[2])); print $0}'
{"level":"info","ts":"2019-08-16T15:48:28.6627967-0700","logger":"elasticsearch-controller","msg":"End reconcile iteration","iteration":15480,"took":0.990385662,"request":"default/example"}

[charith-elastic]

In GCP, StackDriver logging will render structured log messages in a human-friendly way if the log message is associated with a key named message. I propose to do that as a part of this change as well unless there are any objections.