feat(stack): add Firelens sidecar deployment support

[iamhopaul123]

Partially address #875. With this PR now we can deploy a Copilot service with Firelens as the log driver to deliver logs of the main container to destinations.

Users can config Firelens in two ways:

1. Set up configurations in the manifest (unable to sent to multiple destinations though).
2. Configure using fluentbit config file. Note that this requires users to build their own custom fluentbit image and specifying their config file path.

Additionally users might need to add permissions to task role if necessary. And they are able to add their own permissions by using Copilot addons feature. Any addons policy will be added to the task role.

This PR also adjusts svc logs to accommodate Firelens changes.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[iamhopaul123]

Manual Test

Permission file

AdditionalResourcesPolicy:
  Type: AWS::IAM::ManagedPolicy
  Properties:
    PolicyDocument:
      Version: 2012-10-17
      Statement:
      - Effect: Allow
        Action:
        - logs:CreateLogStream
        - logs:CreateLogGroup
        - logs:DescribeLogStreams
        - logs:PutLogEvents
        Resource: "*"

Not using config file

manifest

logging:
  destination:
    Name: cloudwatch
    region: us-west-2
    log_group_name: /copilot/my-project-test-my-app
    log_stream_prefix: copilot/

Using config file

manifest

logging:
  image: 403971813171.dkr.ecr.us-west-2.amazonaws.com/fluentbit:latest
  configFilePath: /extra.conf

config file

[OUTPUT]
    Name cloudwatch
    Match *
    log_group_name /copilot/my-project-test-my-app
    log_stream_prefix copilot/
    region us-west-2

output from svc logs

With ECS meta data

my-app-firelens-8b9a74b77 {"container_id":"425c275d61eb758c4dedf6725321480196145ec0a4d5f398a1d76f311cf0b5fc","container_name":"/ecs-my-project-test-my-app-81-my-app-d096f8f790c6f0f1ba01","ecs_cluster":"arn:aws:ecs:us-west-2:403971813171:cluster/my-project-test-Cluster-uJakRgWSmCCO","ecs_task_arn":"arn:aws:ecs:us-west-2:403971813171:task/my-project-test-Cluster-uJakRgWSmCCO/8b9a74b77d054dd4872bd07532a3e414","ecs_task_definition":"my-project-test-my-app:81","log":"10.0.1.33 - - [04/Jun/2020 16:51:23] \"GET / HTTP/1.1\" 200 -","source":"stderr"}
my-app-firelens-8b9a74b77 {"container_id":"425c275d61eb758c4dedf6725321480196145ec0a4d5f398a1d76f311cf0b5fc","container_name":"/ecs-my-project-test-my-app-81-my-app-d096f8f790c6f0f1ba01","ecs_cluster":"arn:aws:ecs:us-west-2:403971813171:cluster/my-project-test-Cluster-uJakRgWSmCCO","ecs_task_arn":"arn:aws:ecs:us-west-2:403971813171:task/my-project-test-Cluster-uJakRgWSmCCO/8b9a74b77d054dd4872bd07532a3e414","ecs_task_definition":"my-project-test-my-app:81","log":"10.0.0.111 - - [04/Jun/2020 16:51:24] \"GET / HTTP/1.1\" 200 -","source":"stderr"}
my-app-firelens-8b9a74b77 {"container_id":"425c275d61eb758c4dedf6725321480196145ec0a4d5f398a1d76f311cf0b5fc","container_name":"/ecs-my-project-test-my-app-81-my-app-d096f8f790c6f0f1ba01","ecs_cluster":"arn:aws:ecs:us-west-2:403971813171:cluster/my-project-test-Cluster-uJakRgWSmCCO","ecs_task_arn":"arn:aws:ecs:us-west-2:403971813171:task/my-project-test-Cluster-uJakRgWSmCCO/8b9a74b77d054dd4872bd07532a3e414","ecs_task_definition":"my-project-test-my-app:81","log":"10.0.1.33 - - [04/Jun/2020 16:51:33] \"GET / HTTP/1.1\" 200 -","source":"stderr"}
my-app-firelens-8b9a74b77 {"container_id":"425c275d61eb758c4dedf6725321480196145ec0a4d5f398a1d76f311cf0b5fc","container_name":"/ecs-my-project-test-my-app-81-my-app-d096f8f790c6f0f1ba01","ecs_cluster":"arn:aws:ecs:us-west-2:403971813171:cluster/my-project-test-Cluster-uJakRgWSmCCO","ecs_task_arn":"arn:aws:ecs:us-west-2:403971813171:task/my-project-test-Cluster-uJakRgWSmCCO/8b9a74b77d054dd4872bd07532a3e414","ecs_task_definition":"my-project-test-my-app:81","log":"10.0.0.111 - - [04/Jun/2020 16:51:34] \"GET / HTTP/1.1\" 200 -","source":"stderr"}
my-app-firelens-8b9a74b77 {"container_id":"425c275d61eb758c4dedf6725321480196145ec0a4d5f398a1d76f311cf0b5fc","container_name":"/ecs-my-project-test-my-app-81-my-app-d096f8f790c6f0f1ba01","ecs_cluster":"arn:aws:ecs:us-west-2:403971813171:cluster/my-project-test-Cluster-uJakRgWSmCCO","ecs_task_arn":"arn:aws:ecs:us-west-2:403971813171:task/my-project-test-Cluster-uJakRgWSmCCO/8b9a74b77d054dd4872bd07532a3e414","ecs_task_definition":"my-project-test-my-app:81","log":"10.0.1.33 - - [04/Jun/2020 16:51:43] \"GET / HTTP/1.1\" 200 -","source":"stderr"}
my-app-firelens-8b9a74b77 {"container_id":"425c275d61eb758c4dedf6725321480196145ec0a4d5f398a1d76f311cf0b5fc","container_name":"/ecs-my-project-test-my-app-81-my-app-d096f8f790c6f0f1ba01","ecs_cluster":"arn:aws:ecs:us-west-2:403971813171:cluster/my-project-test-Cluster-uJakRgWSmCCO","ecs_task_arn":"arn:aws:ecs:us-west-2:403971813171:task/my-project-test-Cluster-uJakRgWSmCCO/8b9a74b77d054dd4872bd07532a3e414","ecs_task_definition":"my-project-test-my-app:81","log":"10.0.0.111 - - [04/Jun/2020 16:51:44] \"GET / HTTP/1.1\" 200 -","source":"stderr"}
my-app-firelens-8b9a74b77 {"container_id":"425c275d61eb758c4dedf6725321480196145ec0a4d5f398a1d76f311cf0b5fc","container_name":"/ecs-my-project-test-my-app-81-my-app-d096f8f790c6f0f1ba01","ecs_cluster":"arn:aws:ecs:us-west-2:403971813171:cluster/my-project-test-Cluster-uJakRgWSmCCO","ecs_task_arn":"arn:aws:ecs:us-west-2:403971813171:task/my-project-test-Cluster-uJakRgWSmCCO/8b9a74b77d054dd4872bd07532a3e414","ecs_task_definition":"my-project-test-my-app:81","log":"10.0.1.33 - - [04/Jun/2020 16:51:53] \"GET / HTTP/1.1\" 200 -","source":"stderr"}
my-app-firelens-8b9a74b77 {"container_id":"425c275d61eb758c4dedf6725321480196145ec0a4d5f398a1d76f311cf0b5fc","container_name":"/ecs-my-project-test-my-app-81-my-app-d096f8f790c6f0f1ba01","ecs_cluster":"arn:aws:ecs:us-west-2:403971813171:cluster/my-project-test-Cluster-uJakRgWSmCCO","ecs_task_arn":"arn:aws:ecs:us-west-2:403971813171:task/my-project-test-Cluster-uJakRgWSmCCO/8b9a74b77d054dd4872bd07532a3e414","ecs_task_definition":"my-project-test-my-app:81","log":"10.0.0.111 - - [04/Jun/2020 16:51:54] \"GET / HTTP/1.1\" 200 -","source":"stderr"}
my-app-firelens-8b9a74b77 {"container_id":"425c275d61eb758c4dedf6725321480196145ec0a4d5f398a1d76f311cf0b5fc","container_name":"/ecs-my-project-test-my-app-81-my-app-d096f8f790c6f0f1ba01","ecs_cluster":"arn:aws:ecs:us-west-2:403971813171:cluster/my-project-test-Cluster-uJakRgWSmCCO","ecs_task_arn":"arn:aws:ecs:us-west-2:403971813171:task/my-project-test-Cluster-uJakRgWSmCCO/8b9a74b77d054dd4872bd07532a3e414","ecs_task_definition":"my-project-test-my-app:81","log":"10.0.1.33 - - [04/Jun/2020 16:52:03] \"GET / HTTP/1.1\" 200 -","source":"stderr"}
my-app-firelens-8b9a74b77 {"container_id":"425c275d61eb758c4dedf6725321480196145ec0a4d5f398a1d76f311cf0b5fc","container_name":"/ecs-my-project-test-my-app-81-my-app-d096f8f790c6f0f1ba01","ecs_cluster":"arn:aws:ecs:us-west-2:403971813171:cluster/my-project-test-Cluster-uJakRgWSmCCO","ecs_task_arn":"arn:aws:ecs:us-west-2:403971813171:task/my-project-test-Cluster-uJakRgWSmCCO/8b9a74b77d054dd4872bd07532a3e414","ecs_task_definition":"my-project-test-my-app:81","log":"10.0.0.111 - - [04/Jun/2020 16:52:04] \"GET / HTTP/1.1\" 200 -","source":"stderr"}

[efekarakus]

🚀

[kohidave]

Could you update the SvcLogsOutput in the e2e/outputs.go plz? Since we renamed TaskID

[iamhopaul123]

Fixed broken e2e test because of

1. svc logs output change
2. "www" app in "multi-svc-app" e2e test does not handle "/" for default health check making the service unstable.

[kohidave]

Thank you! <3