systemd_journald: Transform keys from journald JSON to Cloud Logging JSON

[akkornel]

Hello!

I am looking at using the Ops Agent for my GCP project, and I'm especially interested in using the systemd_journald receiver. I am interested in this because journald is already collecting logs for me, and it would be great to take advantage of journald's structured-log format.

Even though the receiver is doing the work of pulling in logs from journald, the log entries coming from journald aren't in a form that Cloud Logging can process. So, my request is that the Ops Agent's systemd_journald receiver be enhanced to transform the JSON entries from systemd's schema to Cloud Logging's schema.

One key from the JSON needs a simple key change:

• The key MESSAGE needs to change to message.

One key needs both a key change and a value change:

• SYSLOG_PRIORITY is a numeric priority encoded as a string. The key name needs to change to severity, and the value needs to be mapped to an acceptable-to-Cloud-Logging string using the following mapping:
  7 maps to DEBUG
6 maps to INFO
5 maps to NOTICE
4 maps to WARNING
3 maps to ERROR
2 maps to CRITICAL
1 maps to ALERT
0 maps to EMERGENCY

There are three keys which could be present and, if so, need to trigger the creation of the logging.googleapis.com/sourceLocation object:

• CODE_FILE: If present, it should be added to the logging.googleapis.com/sourceLocation object, under key file.
• CODE_LINE: If present, it should be added to the logging.googleapis.com/sourceLocation object, under key line.
• CODE_FUNC: If present, it should be added to the logging.googleapis.com/sourceLocation object, under key function.

Finally, the timestamp object has to be created by applying some math to the __REALTIME_TIMESTAMP key:

For the timestampSeconds key: Take the value from ⌊__REALTIME_TIMESTAMP ÷ 1000000⌋.
For the timestampNanos key: Take the value from __REALTIME_TIMESTAMP mod 1000000 × 1000.

[quentinmit]

Severity mapping and code mapping are both good feature requests.

Are you actually having problems with the message and timestamp? Those should work fine without any additional translation.

[quentinmit]

I'm going to reopen this since it looks like we haven't fixed MESSAGE yet.

[alemnew]

Is there any ETA for this to be fixed and released?

[github-actions]

This issue was marked stale due to lack of activity. It will be closed in 14 days.

[akkornel]

I don’t think this should be marked stale, as I do not think #256 (comment) has been addressed.