Configure podman to log to journald instead of files

[ventifus]

Which issue this PR addresses:

Fixes ARO-8920

This PR will be superseded by #3741.

What this PR does / why we need it:

Switch podman's log driver to journald to avoid writing to per-container log files and subsequently filling the disk. Mdsd already ships the journal, so there should be no change in log availability in Kusto.

Test plan for issue:

Deploy to INT and verify containers are using the correct log driver. Verify container logs show up in Kusto.

Is there any documentation that needs to be updated for this PR?

https://msazure.visualstudio.com/AzureRedHatOpenShift/_wiki/wikis/ARO.wiki/64663/Current-known-and-outstanding-issues?anchor=gateway-log-rotate-not-functioning is no longer needed.

How do you know this will function as expected in production?

Deployed containers.conf onto a VMSS instance in INT and confirmed logging changed as desired and that there was no impact on ServiceLogs.

[kimorris27]

/azp run ci

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[kimorris27]

LGTM as long as testing in INT goes well.

[ventifus]

Modified one of the INT VMSS instances with the change:

[root@rp-v20240618 ~]# cat /etc/containers/containers.conf 
[containers]
log_driver = "journald"
[root@rp-v20240618 ~]# find /var/lib/containers/storage -name "ctr.log" 
[root@rp-v20240618 ~]# 

No more log files. This means that our components now log twice to the journal: once directly and once via stdout.

Aug 12 18:36:03 rp-v20240618.00-000002 aro-rp[2794]: time="2024-08-12T18:36:03Z" level=info msg="read request" func="middleware.LogMiddleware.Log-fm.LogMiddleware.Log.func1()" file="pkg/frontend/middleware/log.go:102" client_principal_name= client_request_id= component=access correlation_id= request_id=abda82b7-3f15-4d62-9628-9138ff2dca8a request_method=GET request_path=/healthz/ready request_proto=HTTP/2.0 request_remote_addr="168.63.129.16:64440" request_user_agent="Load Balancer Agent"
Aug 12 18:36:03 rp-v20240618.00-000002 aro-rp[2794]: time="2024-08-12T18:36:03Z" level=info msg="sent response" func="middleware.LogMiddleware.Log-fm.LogMiddleware.Log.func1.1()" file="pkg/frontend/middleware/log.go:156" body_read_bytes=0 body_written_bytes=3 client_principal_name= client_request_id= component=access correlation_id= duration=0.000227506 request_id=abda82b7-3f15-4d62-9628-9138ff2dca8a request_method=GET request_path=/healthz/ready request_proto=HTTP/2.0 request_remote_addr="168.63.129.16:64440" request_user_agent="Load Balancer Agent" response_status_code=200
Aug 12 18:36:03 rp-v20240618.00-000002 docker[2068]: time="2024-08-12T18:36:03Z" level=info msg="read request" func="middleware.LogMiddleware.Log-fm.LogMiddleware.Log.func1()" file="pkg/frontend/middleware/log.go:102" client_principal_name= client_request_id= component=access correlation_id= request_id=abda82b7-3f15-4d62-9628-9138ff2dca8a request_method=GET request_path=/healthz/ready request_proto=HTTP/2.0 request_remote_addr="168.63.129.16:64440" request_user_agent="Load Balancer Agent"
Aug 12 18:36:03 rp-v20240618.00-000002 docker[2068]: time="2024-08-12T18:36:03Z" level=info msg="sent response" func="middleware.LogMiddleware.Log-fm.LogMiddleware.Log.func1.1()" file="pkg/frontend/middleware/log.go:156" body_read_bytes=0 body_written_bytes=3 client_principal_name= client_request_id= component=access correlation_id= duration=0.000227506 request_id=abda82b7-3f15-4d62-9628-9138ff2dca8a request_method=GET request_path=/healthz/ready request_proto=HTTP/2.0 request_remote_addr="168.63.129.16:64440" request_user_agent="Load Balancer Agent" response_status_code=200

Also confirmed components still log to Geneva in ARORPLogs / ServiceLogs as expected.

[SudoBrendan]

I like the direction here regardless of how the Jira is groomed. My only question is if we can take this a step further and stop -v on journal as well in RP components.

[SudoBrendan]

question: elsewhere in this file, container systemd units are configured with:

-v /run/systemd/journal:/run/systemd/journal

does this change make that irrelevant, or would we need additional changes to our logger configuration at a golang level to stop "double logging" to journal (or, at the very least, implement principal of least privilege for host mounting of system files)?

[SudoBrendan]

oh lol, you already left a comment about this elsewhere haha - I think this is worth fixing, if for nothing else, kusto query performance.

[github-actions]

Please rebase pull request.