From 1de47d0d037ae600a5decf11757c696202619077 Mon Sep 17 00:00:00 2001 From: Kim Nylander Date: Fri, 27 Sep 2024 20:46:24 -0400 Subject: [PATCH 1/5] Update metrics for Alloy troubleshooting doc --- .../troubleshooting/{agent.md => alloy.md} | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) rename docs/sources/tempo/troubleshooting/{agent.md => alloy.md} (62%) diff --git a/docs/sources/tempo/troubleshooting/agent.md b/docs/sources/tempo/troubleshooting/alloy.md similarity index 62% rename from docs/sources/tempo/troubleshooting/agent.md rename to docs/sources/tempo/troubleshooting/alloy.md index b959e484cdb..c2176aa9e53 100644 --- a/docs/sources/tempo/troubleshooting/agent.md +++ b/docs/sources/tempo/troubleshooting/alloy.md @@ -5,6 +5,7 @@ description: Gain visibility on how many traces are being pushed to Grafana Allo weight: 472 aliases: - ../operations/troubleshooting/agent/ +- ./agent.md # /docs/tempo//troubleshooting/agent.md --- # Troubleshoot Grafana Alloy @@ -33,6 +34,22 @@ exporter_sent_spans_ratio_total exporter_send_failed_spans_ratio_total ``` +### Check metrics in Grafana Cloud + +If you are using Grafana Alloy to send traces to Grafana Cloud, the metrics are visible at +`http://localhost:12345/metrics`. +The `/metrics` HTTP endpoint of the Alloy HTTP server exposes the Alloy component and controller metrics. +Refer to the [Monitor the Grafana Alloy component controller](https://grafana.com/docs/alloy/latest/troubleshoot/controller_metrics/) documentation for more information. + +In your Grafana Cloud instance, they can be checked in the `grafanacloud-usage` data source. +To view the metrics, use the following steps: + +1. From your Grafana instance, select **Explore** in the left menu. +1. Change the data source to `grafanacloud-usage`. +1. Type the metric to verify in the text box. If you start with `grafanacloud_traces_`, you can use autocomplete to browse the list of available metrics. + +![Use Explore to check the metrics for traces sent to Grafana Cloud](/media/docs/tempo/screenshot-tempo-trouble-metrics-search.png) + ## Trace span logging If metrics and logs are looking good, but you are still unable to find traces in Grafana Cloud, you can configure Alloy to output all the traces it receives to the [console](https://grafana.com/docs/tempo//configuration/grafana-alloy/automatic-logging/). From 85e1eabe08336722b24f1f58f56d25b0b8524e79 Mon Sep 17 00:00:00 2001 From: Kim Nylander Date: Fri, 27 Sep 2024 20:53:39 -0400 Subject: [PATCH 2/5] Add link to cloud usage metrics --- docs/sources/tempo/troubleshooting/alloy.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/sources/tempo/troubleshooting/alloy.md b/docs/sources/tempo/troubleshooting/alloy.md index c2176aa9e53..6bb70e49dab 100644 --- a/docs/sources/tempo/troubleshooting/alloy.md +++ b/docs/sources/tempo/troubleshooting/alloy.md @@ -34,20 +34,22 @@ exporter_sent_spans_ratio_total exporter_send_failed_spans_ratio_total ``` -### Check metrics in Grafana Cloud - If you are using Grafana Alloy to send traces to Grafana Cloud, the metrics are visible at `http://localhost:12345/metrics`. The `/metrics` HTTP endpoint of the Alloy HTTP server exposes the Alloy component and controller metrics. Refer to the [Monitor the Grafana Alloy component controller](https://grafana.com/docs/alloy/latest/troubleshoot/controller_metrics/) documentation for more information. -In your Grafana Cloud instance, they can be checked in the `grafanacloud-usage` data source. +### Check metrics in Grafana Cloud + +In your Grafana Cloud instance, you can check metrics using the `grafanacloud-usage` data source. To view the metrics, use the following steps: 1. From your Grafana instance, select **Explore** in the left menu. 1. Change the data source to `grafanacloud-usage`. 1. Type the metric to verify in the text box. If you start with `grafanacloud_traces_`, you can use autocomplete to browse the list of available metrics. +Refer to [Cloud Traces usage metrics](https://grafana.com/docs/grafana-cloud/cost-management-and-billing/understand-your-invoice/usage-limits/#cloud-traces-usage) for a list of metrics related to tracing usage. + ![Use Explore to check the metrics for traces sent to Grafana Cloud](/media/docs/tempo/screenshot-tempo-trouble-metrics-search.png) ## Trace span logging From 29c9527516742832d5948a452866ab9150a4f29e Mon Sep 17 00:00:00 2001 From: Kim Nylander Date: Fri, 27 Sep 2024 21:03:07 -0400 Subject: [PATCH 3/5] Fix broken links and correct link format --- docs/sources/tempo/troubleshooting/_index.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/sources/tempo/troubleshooting/_index.md b/docs/sources/tempo/troubleshooting/_index.md index 26824cd5992..fe7abdf0b13 100644 --- a/docs/sources/tempo/troubleshooting/_index.md +++ b/docs/sources/tempo/troubleshooting/_index.md @@ -16,18 +16,18 @@ In addition, the [Tempo runbook](https://github.com/grafana/tempo/blob/main/oper ## Sending traces -- [Spans are being refused with "pusher failed to consume trace data"]({{< relref "./max-trace-limit-reached" >}}) -- [Is Grafana Alloy sending to the backend?]({{< relref "./agent" >}}) +- [Spans are being refused with "pusher failed to consume trace data"](https://grafana.com/docs/tempo//troubleshooting/max-trace-limit-reached/) +- [Is Grafana Alloy sending to the backend?](https://grafana.com/docs/tempo//troubleshooting/alloy/) ## Querying -- [Unable to find my traces in Tempo]({{< relref "./unable-to-see-trace" >}}) -- [Error message "Too many jobs in the queue"]({{< relref "./too-many-jobs-in-queue" >}}) -- [Queries fail with 500 and "error using pageFinder"]({{< relref "./bad-blocks" >}}) -- [I can search traces, but there are no service name or span name values available]({{< relref "./search-tag" >}}) -- [Error message `response larger than the max ( vs )`]({{< relref "./response-too-large" >}}) -- [Search results don't match trace lookup results with long-running traces]({{< relref "./long-running-traces" >}}) +- [Unable to find my traces in Tempo](https://grafana.com/docs/tempo//troubleshooting/unable-to-see-trace/) +- [Error message "Too many jobs in the queue"](https://grafana.com/docs/tempo//troubleshooting/too-many-jobs-in-queue) +- [Queries fail with 500 and "error using pageFinder"](https://grafana.com/docs/tempo//troubleshooting/bad-blocks) +- [I can search traces, but there are no service name or span name values available](https://grafana.com/docs/tempo//troubleshooting/search-tag) +- [Error message `response larger than the max ( vs )`](https://grafana.com/docs/tempo//troubleshooting/response-too-large) +- [Search results don't match trace lookup results with long-running traces](https://grafana.com/docs/tempo//troubleshooting/long-running-traces") ## Metrics-generator -- [Metrics or service graphs seem incomplete]({{< relref "./metrics-generator" >}}) +- [Metrics or service graphs seem incomplete](https://grafana.com/docs/tempo//troubleshooting/metrics-generator) From 5b909c363be71f6c3dc06350148b1d64454275a3 Mon Sep 17 00:00:00 2001 From: Kim Nylander <104772500+knylander-grafana@users.noreply.github.com> Date: Fri, 27 Sep 2024 21:10:09 -0400 Subject: [PATCH 4/5] Apply suggestions from code review --- docs/sources/tempo/troubleshooting/_index.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/sources/tempo/troubleshooting/_index.md b/docs/sources/tempo/troubleshooting/_index.md index fe7abdf0b13..ba1ba9a2750 100644 --- a/docs/sources/tempo/troubleshooting/_index.md +++ b/docs/sources/tempo/troubleshooting/_index.md @@ -22,12 +22,12 @@ In addition, the [Tempo runbook](https://github.com/grafana/tempo/blob/main/oper ## Querying - [Unable to find my traces in Tempo](https://grafana.com/docs/tempo//troubleshooting/unable-to-see-trace/) -- [Error message "Too many jobs in the queue"](https://grafana.com/docs/tempo//troubleshooting/too-many-jobs-in-queue) -- [Queries fail with 500 and "error using pageFinder"](https://grafana.com/docs/tempo//troubleshooting/bad-blocks) +- [Error message "Too many jobs in the queue"](https://grafana.com/docs/tempo//troubleshooting/too-many-jobs-in-queue/) +- [Queries fail with 500 and "error using pageFinder"](https://grafana.com/docs/tempo//troubleshooting/bad-blocks/) - [I can search traces, but there are no service name or span name values available](https://grafana.com/docs/tempo//troubleshooting/search-tag) -- [Error message `response larger than the max ( vs )`](https://grafana.com/docs/tempo//troubleshooting/response-too-large) -- [Search results don't match trace lookup results with long-running traces](https://grafana.com/docs/tempo//troubleshooting/long-running-traces") +- [Error message `response larger than the max ( vs )`](https://grafana.com/docs/tempo//troubleshooting/response-too-large/) +- [Search results don't match trace lookup results with long-running traces](https://grafana.com/docs/tempo//troubleshooting/long-running-traces/) ## Metrics-generator -- [Metrics or service graphs seem incomplete](https://grafana.com/docs/tempo//troubleshooting/metrics-generator) +- [Metrics or service graphs seem incomplete](https://grafana.com/docs/tempo//troubleshooting/metrics-generator/) From 90aace8fe8c62baf04288afd1fc11d5fee97a075 Mon Sep 17 00:00:00 2001 From: Kim Nylander <104772500+knylander-grafana@users.noreply.github.com> Date: Tue, 1 Oct 2024 12:32:11 -0400 Subject: [PATCH 5/5] Apply suggestions from code review --- docs/sources/tempo/troubleshooting/alloy.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/sources/tempo/troubleshooting/alloy.md b/docs/sources/tempo/troubleshooting/alloy.md index 6bb70e49dab..1fc938a0c95 100644 --- a/docs/sources/tempo/troubleshooting/alloy.md +++ b/docs/sources/tempo/troubleshooting/alloy.md @@ -34,8 +34,7 @@ exporter_sent_spans_ratio_total exporter_send_failed_spans_ratio_total ``` -If you are using Grafana Alloy to send traces to Grafana Cloud, the metrics are visible at -`http://localhost:12345/metrics`. +Alloy has a Prometheus scrape endpoint, `/metrics`, that you can use to check metrics locally by opening a browser to `http://localhost:12345/metrics`. The `/metrics` HTTP endpoint of the Alloy HTTP server exposes the Alloy component and controller metrics. Refer to the [Monitor the Grafana Alloy component controller](https://grafana.com/docs/alloy/latest/troubleshoot/controller_metrics/) documentation for more information.