logzio-aws-metrics

⚠️ Please read: This project is archived, and will have no more new features. You can use logzio/cloudwatch-metrics instead

This project allows you to collect Prometheus format metrics from Amazon CloudWatch with the CloudWatch exporter, and ship them to Logz.io using the OpenTelemetry collector.

We simplify the data export and collection for your metrics: You tell us the desired namespaces and regions you want to send your data from and we fetch the most relevant metrics to display in the Logz.io pre-built Infrastructure Monitoring dashboards.

Overview

Within the applications included in the Docker compose file:

1. The logzio/cloudwatch-config-builder writes your environment data to configuration files shared with both the CloudWatch exporter and the OpenTelemetry collector containers.

   The configuration files specify:

   • Which data is collected
   • Which data is exposed
   • To which endpoints the data is sent

2. The CloudWatch exporter and OpenTelemetry collector containers use the configuration files to export and move the metrics data to Logz.io as follows:

   a. Via API, the CloudWatch exporter fetches the requested metrics data collected by CloudWatch from the AWS services in your platform and exposes these metrics to the designated endpoint.

   b. The OpenTelemetry collector receives the data via a metrics pipeline and sends it to Logz.io for ingestion.

Note:

• Extra calls and charges on AWS API requests may be generated by this project.

• You can track the number of AWS API requests with the cloudwatch_requests_total metric, which provides labels with the API name and namespace specification. For example : cloudwatch_requests_total{action="getMetricStatistics",namespace="AWS/EC2",} 876.0.

• For more information about the prom/cloudwatch-exporter cost, refer to the relevant Prometheus documentation and to the AWS cloudwatch API pricing page.

Before you begin

Set up your IAM user

You'll need an IAM user with the following permissions:

• cloudwatch:ListMetrics
• cloudwatch:GetMetricStatistics
• tag:GetResources

If you don't have one, set that up now.

Create an Access key ID and Secret access key for the IAM user, and paste them in your text editor.

Get your metrics region

You'll need to specify the AWS region you're collecting metrics from.

Find your region's slug in the region menu (in the top menu, on the right side).

For example: The slug for US East (N. Virginia) is "us-east-1" and the slug for Canada (Central) is "ca-central-1".

Collect CloudWatch metrics

To collect CloudWatch metrics, you'll deploy a docker compose file that uses containers with the following images:

• otel/opentelemetry-collector
• prom/cloudwatch-exporter
• logzio/cloudwatch-config-builder

Download the docker-compose.yml file.

curl https://raw.githubusercontent.com/logzio/logzio-aws-metrics/main/docker-compose.yml -o docker-compose.yml

Define the environment variables

AWS_ACCESS_KEY_ID=<<AWS_ACCESS_KEY_ID>> \
AWS_SECRET_ACCESS_KEY=<<AWS_SECRET_ACCESS_KEY>> \
AWS_DEFAULT_REGION=<<AWS_DEFAULT_REGION>> \
LOGZIO_REGION= <<LOGZIO_REGION>> \
LOGZIO_TOKEN=<<LOGZIO_TOKEN>> \
AWS_NAMESPACES=<<AWS_NAMESPACES>> \
SCRAPE_INTERVAL=<<SCRAPE_INTERVAL>> \
P8S_LOGZIO_NAME=<<P8S_LOGZIO_NAME>> \
docker-compose up

Set environment variables for the logzio/cloudwatch-config-builder container

Environment variable	Description
AWS_DEFAULT_REGION (Required)	Your region's slug. You can find this in the AWS Console region menu (in the top menu, to the right). Note: This is the region that you will collect metrics from.
LOGZIO_REGION (Required)	Your Logz.io region code. For example if your region is US, then your region code is us. You can find your region code here: https://docs.logz.io/user-guide/accounts/account-region.html#regions-and-urls.
LOGZIO_TOKEN (Required)	Token for shipping metrics to your Logz.io account. Find it under Settings > Manage accounts. How do I look up my Metrics account token?
SCRAPE_INTERVAL (Required)	The time interval (in seconds) during which the Cloudwatch exporter retrieves metrics from Cloudwatch, and the Opentelemtry collector scrapes and sends the metrics to Logz.io. Default = 300. Note: This value must be a multiple of 60.
AWS_NAMESPACES (Required)	Comma-separated list of namespaces of the metrics you want to collect. You can find a complete list of namespaces at AWS Services That Publish CloudWatch Metrics. Note: This Environment variable is required unless you define the CUSTOM_CONFIG_PATH Environment variable
P8S_LOGZIO_NAME	The value of the p8s_logzio_name external label. This variable identifies which Prometheus environment the metrics arriving at Logz.io came from. Default = logzio-cloudwatch-metrics.
CUSTOM_CONFIG_PATH	Path to your Cloudwatch exporter configuration file. For more information refer to the documentation. Note: Set the period_seconds parameter according to your SCRAPE_INTERVAL
CUSTOM_LISTENER	Set a custom URL to ship metrics to (for example, http://localhost:9200). This overrides the LOGZIO_REGION Environment variable.
AWS_ROLE_ARN	Your IAM role to assume.

Set environment variables for the prom/cloudwatch-exporter container

Environment variable	Description
AWS_ACCESS_KEY_ID	Your IAM user's access key ID.
AWS_SECRET_ACCESS_KEY	Your IAM user's secret key.
AWS_CREDENTIAL_PROFILES_FILE	The path to your shared credentials file.
AWS_PROFILE	Your profile name in shared credentials file.

Run Docker.

Run the command: docker-compose up.

View service configurations.

You can view the exporter and collector configurations in the browser:

• CloudWatch exporter: http://localhost:5001/config/cloudwatch
• Opentelemtry collector: http://localhost:5001/config/otel