The Centralized Logging solution helps organizations collect, analyze, and display Amazon CloudWatch Logs in a single dashboard. This solution consolidates, manages, and analyzes log files from Amazon CloudWatch Logs from one Source Account in us-east-1 Region into a Centralized Logging Account in a single region (us-east-1).
For the purpose of this tutorial we will use the following account numbers.
Source-Account= 222222222222
Logging= 111111111111
- Intermedial to advance level in Python. So, you can adapt and customized the
lambda.zip
files to your need an use cases. - Basic to intermedial level in json to edit the json file policy.json to modify it if needed to your use case, since we give granular limited access to AWS resources.
- One AWS Account known as the "Logging" to centralize the logs from the "Source-Acccount".
- A second AWS Account in which the CloudWatch Log Group can be used as the source for the logs data an has us-east-1 in "Logging" as Log Destination Endpoint
- The AWS Command Line Interface (AWS CLI) properly installed, configured, and updated. Version 2 ‐ is the most recent major version of the AWS CLI and supports all of the latest features.
- IAM
- Lambda
- CloudWatch
- CloudTrail
- SSM Parameter Store
- The AWS Command Line Interface (AWS CLI)
- ElasticSearch* (Need to add this step yet)
- Python 3.9
In this case Identity and Access Management (IAM)
is a global element, so do not worry about what region it is configured in you profile in the CLI. However, though some AWS Services like EventBridge
, CloudWatch
, and Lambda
are regional. Therefore, be sure you are have us-east-1 (N. Virginia) as region for most of the purposes of this project.
We need 2 roles; assume-role-for-logging and centralized-logs with limited granular permissions to interact with other AWS Services such as: Lambda, IAM, SSM, and CloudWatch Logs for this project
The following limited policy will be attached to the role centralized-logs
central-logging-policy.json - IAM Policy to authorize centralized-logs to create a subscription filter in CloudWatch, get Parameter, and assume role
See policy.json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"logs:DeleteSubscriptionFilter",
"logs:DescribeSubscriptionFilters",
"logs:PutSubscriptionFilter",
"ssm:GetParameters",
"ssm:GetParameter"
],
"Resource": "*"
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Resource": "arn:aws:iam::596294512340:role/assume-role-for-logging"
}
]
}
The following limited policy will be attached to the role assume-role-for-logging.
To provide write permissions to CloudWatch Logs.
See the AWS Managed Policy AWSLambdaBasicExecutionRole
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "*"
}
]
}
EventRule.json - This rule filters CreateLogGroup and CreateLogStream events coming from AWS API Call via CloudTrail
and uses as a target centralized-logs.
Take a pick at the rule here...
{
"source": [
"aws.logs"
],
"detail-type": [
"AWS API Call via CloudTrail"
],
"detail": {
"eventSource": [
"logs.amazonaws.com"
],
"eventName": [
"CreateLogGroup",
"CreateLogStream"
]
}
}
Note: Sometimes when creating or editing complex custom rules such as when using prefix feature it is necessary to create them in EventBridge or to update the very same rules. If it is done CloudWatch directly it may not not work. Hence, we best configure the rules in EventBridge even though the end result is also shown in CloudWatch.
In every command line described here replace the placeholder accounts with your own account numbers Source-Account= 222222222222 Logging= 111111111111 Therefore, in every command line replace 222222222222 with the account number you designated as you Source-Account and replace 111111111111 with the account number that you designated as your Logging account
To Create a role in Source-Account by using the CLI command line. Type the following command line, be sure you are using your Source-Account credentials.
aws iam create-role --role-name centralized-logs --assume-role-policy-document '{"Version": "2012-10-17","Statement": [{ "Effect": "Allow", "Principal": {"Service": "lambda.amazonaws.com"}, "Action": "sts:AssumeRole"}]}'
Output
{
"Role": {
"Path": "/",
"RoleName": "centralized-logs",
"RoleId": "AROAV2E2RDEX5MW4GU6KE",
"Arn": "arn:aws:iam::222222222222:role/centralized-logs",
"CreateDate": "2021-09-13T14:38:42+00:00",
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "lambda.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
}
}
Be sure you are launching the commands from the folder where you downloaded the proyect files… /logging-project
Let us use the file policy.json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"logs:DeleteSubscriptionFilter",
"logs:DescribeSubscriptionFilters",
"logs:PutSubscriptionFilter",
"ssm:GetParameters",
"ssm:GetParameter"
],
"Resource": "*"
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Resource": "arn:aws:iam::111111111111:role/replicate_log_assume_role"
}
]
}
Execute the following command line
aws iam create-policy --policy-name central-logging-policy --policy-document file://policy.json
Output
{
"Policy": {
"PolicyName": "central-logging-policy",
"PolicyId": "ANPAV2E2RDEXRDVWAACTJ",
"Arn": "arn:aws:iam::222222222222:policy/central-logging-policy",
"Path": "/",
"DefaultVersionId": "v1",
"AttachmentCount": 0,
"PermissionsBoundaryUsageCount": 0,
"IsAttachable": true,
"CreateDate": "2021-09-13T15:07:58+00:00",
"UpdateDate": "2021-09-13T15:07:58+00:00"
}
}
aws iam attach-role-policy --role-name centralized-logs --policy-arn arn:aws:iam::222222222222:policy/central-logging-policy
Type the following CLI command to attach the AWS managed policy AWSLambdaBasicExecutionRole
aws iam attach-role-policy --role-name centralized-log --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
For this command line there is not output
Now lets go to Logging Account...
Be sure you are using your Logging account credentials. In case of doubt you can check which profile you are using with the following command line
aws configure list
Type the following command line to create the role assume-role-for-logging. In this command line we include the policy to enable the trusted relationship so as to enable assume-role-for-logging to assume role centralized-logs in AWS Source-Account.
aws iam create-role --role-name assume-role-for-logging --assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":"arn:aws:iam::222222222222:role/centralized-logs"},"Action":"sts:AssumeRole"}]}'
Output
{
"Role": {
"Path": "/",
"RoleName": "assume-role-for-logging",
"RoleId": "AROAYVVPMF3KIJP4Q24BT",
"Arn": "arn:aws:iam::111111111111:role/assume-role-for-logging",
"CreateDate": "2021-09-13T14:28:24+00:00",
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::222222222222:role/centralized-logs"
},
"Action": "sts:AssumeRole"
}
]
}
}
}
Type the following CLI command to attach the AWS managed policy CloudWatchLogsFullAccess
aws iam attach-role-policy --role-name assume-role-for-logging --policy-arn arn:aws:iam::aws:policy/service-role/CloudWatchLogsFullAccess
For this command line there is not output
Let us go back to Source-Account credentials Be sure that you have configure your CLI credentials for in us-east-1
aws lambda create-function \
--function-name centralized-logs \
--zip-file fileb://lambda.zip \
--role arn:aws:iam::222222222222:role/centralized-logs\
--handler lambda_function.lambda_handler \
--runtime python3.9
Output:
{
"FunctionName": "centralized-logs",
"FunctionArn": "arn:aws:lambda:us-east-1:222222222222:function:centralized-logs",
"Runtime": "python3.9",
"Role": "arn:aws:iam::222222222222:role/centralized-logs",
"Handler": "lambda.handler",
"CodeSize": 1651,
"Description": "",
"Timeout": 3,
"MemorySize": 128,
"LastModified": "2021-09-13T16:28:36.134+0000",
"CodeSha256": "HgW7rkaOikKYVZeQrOE9Yi+bDYrhx1oYp9y5b0zszTs=",
"Version": "$LATEST",
"TracingConfig": {
"Mode": "PassThrough"
},
"RevisionId": "53edc92e-a99c-4fca-94b2-4dee0794670a",
"State": "Active",
"LastUpdateStatus": "Successful",
"PackageType": "Zip"
}
3.2 Grant CloudWatch Logs the permission to execute your lambda function. Use the following command, replacing the placeholder account with your own account and the placeholder log group with the log group to process:
aws lambda add-permission \
--function-name "centralized-logs" \
--statement-id "centralizesLogs" \
--principal "logs.amazonaws.com" \
--action "lambda:InvokeFunction" \
--source-arn "arn:aws:logs:us-east-1:222222222222:log-group:*" \
--source-account "222222222222"
OUTPUT:
{
"Statement": "{\"Sid\":\"centralizesLogs\",\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"logs.amazonaws.com\"},\"Action\":\"lambda:InvokeFunction\",\"Resource\":\"arn:aws:lambda:us-east-1:222222222222:function:centralized-logs\",\"Condition\":{\"StringEquals\":{\"AWS:SourceAccount\":\"222222222222\"},\"ArnLike\":{\"AWS:SourceArn\":\"arn:aws:logs:us-east-1:222222222222:log-group:*\"}}}"
}
Now we are going to create a rule to trigger the lambda function
aws events put-rule --name "sent-logs-to-lambda" --event-pattern "{\"source\":[\"aws.logs\"],\"detail-type\":[\"AWS API Call via CloudTrail\"],\"detail\":{\"eventSource\":[\"logs.amazonaws.com\"],\"eventName\":[\"CreateLogGroup\",\"CreateLogStream\"]}}"
OUTPUT:
{
"RuleArn": "arn:aws:events:us-east-1:222222222222:rule/sent-logs-to-lambda"
}
4.2 To grant permissions to the lambda “centralized-logs” to allow it to be invoked by "arn:aws:events:us-east-1:222222222222:rule/sent-logs-to-lambda"
aws lambda add-permission \
--function-name centralized-logs \
--statement-id my-send-logs-to-lambda-permission \
--action 'lambda:InvokeFunction' \
--principal events.amazonaws.com \
--source-arn arn:aws:events:us-east-1:222222222222:rule/sent-logs-to-lambda
OUTPUT:
{
"Statement": "{\"Sid\":\"my-send-logs-to-lambda-permission\",\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"events.amazonaws.com\"},\"Action\":\"lambda:InvokeFunction\",\"Resource\":\"arn:aws:lambda:us-east-1:222222222222:function:centralized-logs\",\"Condition\":{\"ArnLike\":{\"AWS:SourceArn\":\"arn:aws:events:us-east-1:222222222222:rule/sent-logs-to-lambda\"}}}"
}
4.3 To add the target to the rule arn arn:aws:events:us-east-1:222222222222:rule/sent-logs-to-lambda to the lambda function centralized-logs
aws events put-targets --rule sent-logs-to-lambda --targets file://targets.json
OUTPUT:
{
"FailedEntryCount": 0,
"FailedEntries": []
}
aws ssm put-parameter \
--name "LogDestination" \
--type "String" \
--value "arn:aws:lambda:us-east-1:222222222222:function:centralized-logs" \
--overwrite
OUTPUT:
{
"Version": 1,
"Tier": "Standard"
}
aws ssm put-parameter \
--name "LogArnAssume" \
--type "String" \
--value "arn:aws:iam::111111111111:role/assume-role-for-logging" \
--overwrite
OUTPUT:
{
"Version": 1,
"Tier": "Standard"
}
aws logs put-log-events --log-group-name testreplicatelogs --log-stream-name teststream1 --log-events "[{\"timestamp\":1631554348, \"message\": \"Simple logs replication Test\"}]"