Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updates the monitoring guide #2511

Merged
merged 1 commit into from
Apr 24, 2023
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 23 additions & 17 deletions content/rdi/monitoring-guide.md
Original file line number Diff line number Diff line change
@@ -1,28 +1,31 @@
---
Title: Monitoring guide
linkTitle: Monitoring
description:
description: Monitor RDI Engine and data processing jobs
weight: 70
alwaysopen: false
categories: ["redis-di"]
aliases:
---

This document describes how to monitor RDI Engine and data processing jobs. RDI Engine accumulates its operating statistics which can be observed and analyzed to discover various types of problems or used for optimization purposes.
RDI Engine accumulates operating statistics that you can:

* Observe and analyze to discover various types of problems.
* Use for optimization purposes.

## Console metrics

Some basic RDI operating metrics can be displayed using the [`redis-di status`]({{<relref "/rdi/reference/cli/redis-di-status">}}) command. The command provides information about the current RDI Engine status, target database configuration and processing statistics broken down by stream. This tool is intended to be used by an Operator to get the current snapshot of the system.

## Prometheus Integration
## Prometheus integration

RDI allows exporting its metrics to [Prometheus](https://prometheus.io/) and visualizing them in [Grafana](https://grafana.com/). Currently, RDI relies on the external [OSS Redis Exporter](https://github.com/oliver006/redis_exporter) that connects to RDI database to source the metrics and serve them for Prometheus job scraping. The diagram below describes this flow and components involved.

![Metrics architecture](/images/rdi/monitoring-architecture.png)

> Note: The host names and ports above are examples only and can be changed as needed.

## Installing and configuring the Exporter
## Install and configure the Exporter

OSS Metrics Exporter is available as a [pre-built docker container](https://hub.docker.com/r/oliver006/redis_exporter) so that you integrate it into container-based or Kubernetes environments. Alternatively, you can be [build and install](https://hub.docker.com/r/oliver006/redis_exporter) it as a binary to any compute node that has access to an RDI database that needs to be monitored.

Expand All @@ -43,19 +46,19 @@ Create the following [Lua script](https://redis.io/docs/manual/programmability/e
return (redis.call('RG.TRIGGER', 'GetMetrics', '*'))[1]
```

### Testing OSS Metrics Exporter
### Test OSS Metrics Exporter

Start the OSS Metrics Exporter and navigate to `http://localhost:9121/metrics` to see the exported metrics. You should be able to see the following metric:

```
redis_script_values{key="rdi_engine_status{status=RUNNING}"} 1
```

> Note: The actual value of the metric above can be 0, if you haven't started RDI Engine yet. You must have the RDI database created and configured before observing any metrics. If you are not seeing it or getting an error value instead, this indicates either a misconfiguration of the OSS Metrics Exporter, or the RDI database not properly configured.
> Note: The actual value of the metric above can be 0, if you haven't started RDI Engine yet. You must have the RDI database created and configured before observing any metrics. If you are not seeing it or getting an error value instead, this indicates that either the OSS Metrics Exporter or the RDI database is not properly configured.

## Configuring Prometheus
## Configure Prometheus

The next step is to configure the Prometheus scraper. This is done in `prometheus.yml` file. Edit this file to add the following scraper config:
Next, configure the Prometheus scraper. Edit the `prometheus.yml` file to add the following scraper config:

```yaml
scrape_configs:
Expand Down Expand Up @@ -88,39 +91,42 @@ scrape_configs:
regex: "metric_name|key"
```

> Note: Make sure the `targets` value above points to the host and port you have configured to run the OSS Metrics Exporter.
> Note: Make sure the `targets` value above points to the host and port you configured to run the OSS Metrics Exporter.

### Test Prometheus scraper

### Testing Prometheus scraper
After the scraper config is added to the Prometheus configuration, you should now be able to navigate to `http://<HOSTNAME>:9090/graph` (replace `<HOSTNAME>` with a valid Prometheus hostname or IP address).
Explore RDI metrics using the [expression browser](https://prometheus.io/docs/visualization/browser/).

After the scraper config is added to the Prometheus configuration, you should now be able to navigate to `http://localhost:9090/graph` (replace `localhost` with a valid Prometheus hostname or IP address) and see RDI metrics in the following format:
In the expression box, type in a metric name (for example, `rdi_engine_status`) and select `Enter` or the `Execute` button to see the following result:

```
rdi_engine_status{instance="redis-exporter:9121", job="redis-exporter", status="RUNNING"} 1
```

> Note: You may see more than just one RDI metric, if RDI Engine has already processed any data. If you do not see any metrics please check your scraper job configuration in Prometheus.

## Using Grafana to analyze metrics
## Use Grafana to analyze metrics

Optionally, you may deploy the sample Grafana dashboard to monitor the status of RDI Engine and its individual jobs. Follow the steps below in order to import the dashboard:
Optionally, you may deploy the sample Grafana dashboard to monitor the status of RDI Engine and its individual jobs. Follow these steps to import the dashboard:

1. Download the **dashboard file** to your local machine.

2. Log into Grafana and navigate to the list of dashboards, then choose **New -> Import**:
1. Log into Grafana and navigate to the list of dashboards, then choose **New -> Import**:

![New dashboard creation](/images/rdi/monitoring-grafana-new-dash.png.png)

3. On the next screen choose **Upload JSON file** and upload the file you've downloaded in step 1. Make sure you select the data source that is connected to the OSS Metrics Exporter:
1. On the next screen, select **Upload JSON file** and upload the file you downloaded in step 1. Make sure you select the data source that is connected to the OSS Metrics Exporter:

![Data source connection](/images/rdi/monitoring-grafana-dash-configure.png)

4. Click **Import** and make sure you choose the jobs to monitor in a drop-down list (this will be empty if you don't have any jobs running yet):
1. Select **Import** and make sure you choose the jobs to monitor in a drop-down list (this will be empty if you don't have any jobs running yet):

![Dashboard running](/images/rdi/monitoring-grafana-dash-running.png)

## RDI metrics

Below is the list of RDI metrics exported along with their description:
This list shows exported RDI metrics along with their descriptions:

| Metric Name | Labels | Values | Description |
| --------------------------- | ---------------------------------------------------------------------------------------------------------- | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
Expand Down