PXP-7805 Fetch audit logs from an AWS SQS #2
Conversation
|
The style in this PR agrees with This formatting comment was generated automatically by a script in uc-cdis/wool. |
src/audit/config-default.yaml
Outdated
| sqs_url: | ||
| region: | ||
| aws_access_key_id: | ||
| aws_secret_access_key: |
There was a problem hiding this comment.
I guess this is just my opinion. Should we put these under the type: aws_sqs block just to have configs organized under different types of queues?
src/audit/pull_from_queue.py
Outdated
| try: | ||
| response = sqs.receive_message( | ||
| QueueUrl=config["QUEUE_CONFIG"]["sqs_url"], | ||
| MaxNumberOfMessages=10, # 10 is the max |
There was a problem hiding this comment.
So you can only do 10 the limit from the sqs side? If not should make this configurable.
There was a problem hiding this comment.
that's a limit on the AWS side (doc)
| @@ -70,6 +71,15 @@ def app_init() -> FastAPI: | |||
| db.init_app(app) | |||
| load_modules(app) | |||
|
|
|||
| @app.on_event("startup") | |||
There was a problem hiding this comment.
So this keeps on running and pulls in from the queue?
There was a problem hiding this comment.
yes, pull_from_queue_loop is called when the app starts and keeps pulling from the queue in an infinite loop
Jira Ticket: PXP-7805
goes with uc-cdis/fence#923 and uc-cdis/cloud-automation#1603
New Features
Improvements
Dependency updates
Deployment changes
kubectl delete secret audit-g3autoandgen3 kube-setup-audit-serviceto configure the audit SQS and update the configuration file. REQUIRES Fence 5.1.0 or more recent. OR to disable pulling from a queue, update the configuration file manually to disable "PULL_FROM_QUEUE"