Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tracing : Service Map for particular Trace results in a Error in Panel #427

Closed
rkarthikr opened this issue Jul 24, 2024 · 4 comments · Fixed by #428
Closed

Tracing : Service Map for particular Trace results in a Error in Panel #427

rkarthikr opened this issue Jul 24, 2024 · 4 comments · Fixed by #428
Labels
datasource/OpenSearch effort/small type/bug Something isn't working

Comments

@rkarthikr
Copy link

What happened:

Search for a trace ID , View the trace span and then enable service map and refresh the panel. Service Map does not load properly

What you expected to happen:
Service map to load properly

image

How to reproduce it (as minimally and precisely as possible):
Search for a trace ID , View the trace span and then enable service map and refresh the panel

Anything else we need to know?:

Environment:
-Grafana version: Grafana v11.2.0-73451
-OpenSearch version: AWS 2.13
-Plugin version: 2.17.1
@rkarthikr rkarthikr added datasource/OpenSearch type/bug Something isn't working labels Jul 24, 2024
@kevinwcyu
Copy link
Contributor

Hi @rkarthikr, I wasn't able to reproduce this following your repro steps.

  • Are there any other errors in the console?
  • Can you share what the query request looks like from the Query Inspector?

@kevinwcyu kevinwcyu moved this from Incoming to Waiting in AWS Datasources Jul 26, 2024
@rkarthikr
Copy link
Author 

{
  "traceId": "93b2f7cef93288fd3b575e2613182d43",
  "request": {
    "url": "api/ds/query?ds_type=grafana-opensearch-datasource&requestId=explore_rqf",
    "method": "POST",
    "data": {
      "queries": [
        {
          "luceneQueryType": "Traces",
          "query": "traceId: 0272bba222d1ba0d2c0897e55d9c71f6",
          "refId": "A",
          "queryType": "lucene",
          "alias": "",
          "metrics": [
            {
              "type": "count",
              "id": "1"
            }
          ],
          "bucketAggs": [
            {
              "type": "date_histogram",
              "id": "2",
              "settings": {
                "interval": "auto"
              },
              "field": "startTime"
            }
          ],
          "format": "table",
          "timeField": "startTime",
          "serviceMap": true,
          "datasource": {
            "type": "grafana-opensearch-datasource",
            "uid": "ads67lnsevj0gd"
          },
          "datasourceId": 12,
          "intervalMs": 60000,
          "maxDataPoints": 808
        }
      ],
      "from": "1722180658024",
      "to": "1722180958024"
    },
    "hideFromInspector": false
  },
  "response": {
    "results": {
      "A": {
        "status": 200,
        "frames": [
          {
            "schema": {
              "refId": "A",
              "meta": {
                "typeVersion": [
                  0,
                  0
                ],
                "preferredVisualisationType": "trace"
              },
              "fields": [
                {
                  "name": "_id",
                  "type": "string",
                  "typeInfo": {
                    "frame": "string",
                    "nullable": true
                  },
                  "config": {
                    "filterable": false
                  }
                },
                {
                  "name": "_index",
                  "type": "string",
                  "typeInfo": {
                    "frame": "string",
                    "nullable": true
                  },
                  "config": {
                    "filterable": false
                  }
                },
                {
                  "name": "_source",
                  "type": "other",
                  "typeInfo": {
                    "frame": "json.RawMessage",
                    "nullable": true
                  },
                  "config": {
                    "filterable": false
                  }
                },
                {
                  "name": "_type",
                  "type": "other",
                  "typeInfo": {
                    "frame": "json.RawMessage",
                    "nullable": true
                  },
                  "config": {
                    "filterable": false
                  }
                },
                {
                  "name": "droppedAttributesCount",
                  "type": "number",
                  "typeInfo": {
                    "frame": "float64",
                    "nullable": true
                  },
                  "config": {
                    "filterable": false
                  }
                },
                {
                  "name": "droppedEventsCount",
                  "type": "number",
                  "typeInfo": {
                    "frame": "float64",
                    "nullable": true
                  },
                  "config": {
                    "filterable": false
                  }
                },
                {
                  "name": "droppedLinksCount",
                  "type": "number",
                  "typeInfo": {
                    "frame": "float64",
                    "nullable": true
                  },
                  "config": {
                    "filterable": false
                  }
                },
                {
                  "name": "duration",
                  "type": "number",
                  "typeInfo": {
                    "frame": "float64",
                    "nullable": true
                  },
                  "config": {
                    "filterable": false
                  }
                },
                {
                  "name": "endTime",
                  "type": "string",
                  "typeInfo": {
                    "frame": "string",
                    "nullable": true
                  },
                  "config": {
                    "filterable": false
                  }
                },
                {
                  "name": "instrumentationScope",
                  "type": "other",
                  "typeInfo": {
                    "frame": "json.RawMessage",
                    "nullable": true
                  },
                  "config": {
                    "filterable": false
                  }
                },
                {
                  "name": "kind",
                  "type": "string",
                  "typeInfo": {
                    "frame": "string",
                    "nullable": true
                  },
                  "config": {
                    "filterable": false
                  }
                },
                {
                  "name": "links",
                  "type": "other",
                  "typeInfo": {
                    "frame": "json.RawMessage",
                    "nullable": true
                  },
                  "config": {
                    "filterable": false
                  }
                },
                {
                  "name": "logs",
                  "type": "other",
                  "typeInfo": {
                    "frame": "json.RawMessage",
                    "nullable": true
                  },
                  "config": {
                    "filterable": false
                  }
                },
                {
                  "name": "operationName",
                  "type": "string",
                  "typeInfo": {
                    "frame": "string",
                    "nullable": true
                  },
                  "config": {
                    "filterable": false
                  }
                },
                {
                  "name": "parentSpanID",
                  "type": "string",
                  "typeInfo": {
                    "frame": "string",
                    "nullable": true
                  },
                  "config": {
                    "filterable": false
                  }
                },
                {
                  "name": "serviceName",
                  "type": "string",
                  "typeInfo": {
                    "frame": "string",
                    "nullable": true
                  },
                  "config": {
                    "filterable": false
                  }
                },
                {
                  "name": "serviceTags",
                  "type": "other",
                  "typeInfo": {
                    "frame": "json.RawMessage",
                    "nullable": true
                  },
                  "config": {
                    "filterable": false
                  }
                },
                {
                  "name": "spanID",
                  "type": "string",
                  "typeInfo": {
                    "frame": "string",
                    "nullable": true
                  },
                  "config": {
                    "filterable": false
                  }
                },
                {
                  "name": "startTime",
                  "type": "number",
                  "typeInfo": {
                    "frame": "int64",
                    "nullable": true
                  },
                  "config": {
                    "filterable": false
                  }
                },
                {
                  "name": "status",
                  "type": "other",
                  "typeInfo": {
                    "frame": "json.RawMessage",
                    "nullable": true
                  },
                  "config": {
                    "filterable": false
                  }
                },
                {
                  "name": "tags",
                  "type": "other",
                  "typeInfo": {
                    "frame": "json.RawMessage",
                    "nullable": true
                  },
                  "config": {
                    "filterable": false
                  }
                },
                {
                  "name": "traceGroup",
                  "type": "string",
                  "typeInfo": {
                    "frame": "string",
                    "nullable": true
                  },
                  "config": {
                    "filterable": false
                  }
                },
                {
                  "name": "traceGroupFields",
                  "type": "other",
                  "typeInfo": {
                    "frame": "json.RawMessage",
                    "nullable": true
                  },
                  "config": {
                    "filterable": false
                  }
                },
                {
                  "name": "traceID",
                  "type": "string",
                  "typeInfo": {
                    "frame": "string",
                    "nullable": true
                  },
                  "config": {
                    "filterable": false
                  }
                },
                {
                  "name": "traceState",
                  "type": "string",
                  "typeInfo": {
                    "frame": "string",
                    "nullable": true
                  },
                  "config": {
                    "filterable": false
                  }
                }
              ]
            },
            "data": {
              "values": [
                [
                  "d864587007168cf0",
                  "3d2e27909554bd49",
                  "477f1795a718fafb",
                  "9d1a0a31fb14bb1e",
                  "015bb8c502cfd036",
                  "103f56279de45f1e",
                  "5a4103e6aee350f5",
                  "02a302d0b7cb1047",
                  "19c1081f8f05ac8f"
                ],
                [
                  "otel-v1-apm-span-000012",
                  "otel-v1-apm-span-000012",
                  "otel-v1-apm-span-000012",
                  "otel-v1-apm-span-000012",
                  "otel-v1-apm-span-000012",
                  "otel-v1-apm-span-000012",
                  "otel-v1-apm-span-000012",
                  "otel-v1-apm-span-000012",
                  "otel-v1-apm-span-000012"
                ],
                [
                  {
                    "droppedAttributesCount": 0,
                    "droppedEventsCount": 0,
                    "droppedLinksCount": 0,
                    "durationInNanos": 18604885,
                    "endTime": "2024-07-28T15:32:14.136604885Z",
                    "events": [],
                    "instrumentationScope": {
                      "name": "@opentelemetry/instrumentation-grpc",
                      "version": "0.51.1"
                    },
                    "kind": "SPAN_KIND_CLIENT",
                    "links": [],
                    "name": "grpc.oteldemo.CartService/GetCart",
                    "parentSpanId": "477f1795a718fafb",
                    "resource": {
                      "attributes": {
                        "container@id": "0139e7e2c3f3eedc6ae167adee962b9b1743245202d8c08a23fe5f59700d13b3",
                        "host@arch": "amd64",
                        "host@name": "opentelemetry-demo-frontend-865f97d679-5jmwp",
                        "k8s@deployment@name": "opentelemetry-demo-frontend",
                        "k8s@namespace@name": "otel-demo",
                        "k8s@node@name": "ip-172-31-37-251.ec2.internal",
                        "k8s@pod@ip": "172.31.34.40",
                        "k8s@pod@name": "opentelemetry-demo-frontend-865f97d679-5jmwp",
                        "k8s@pod@start_time": "2024-07-16T21:54:24Z",
                        "k8s@pod@uid": "d54f284d-b2f0-48c5-be8b-cfbf199f89cf",
                        "os@type": "linux",
                        "os@version": "5.10.219-208.866.amzn2.x86_64",
                        "process@command": "/app/server.js",
                        "process@command_args": "[\"/usr/local/bin/node\",\"--require\",\"./Instrumentation.js\",\"/app/server.js\"]",
                        "process@executable@name": "node",
                        "process@executable@path": "/usr/local/bin/node",
                        "process@owner": "nextjs",
                        "process@pid": 17,
                        "process@runtime@description": "Node.js",
                        "process@runtime@name": "nodejs",
                        "process@runtime@version": "18.20.3",
                        "service@instance@id": "d54f284d-b2f0-48c5-be8b-cfbf199f89cf",
                        "service@name": "frontend",
                        "service@namespace": "opentelemetry-demo",
                        "service@version": "1.11.0",
                        "telemetry@sdk@language": "nodejs",
                        "telemetry@sdk@name": "opentelemetry",
                        "telemetry@sdk@version": "1.24.1"
                      }
                    },
                    "serviceName": "frontend",
                    "span": {
                      "attributes": {
                        "net@peer@name": "opentelemetry-demo-cartservice",
                        "net@peer@port": 8080,
                        "rpc@grpc@status_code": 0,
                        "rpc@method": "GetCart",
                        "rpc@service": "oteldemo.CartService",
                        "rpc@system": "grpc"
                      }
                    },
                    "spanId": "d864587007168cf0",
                    "startTime": "2024-07-28T15:32:14.118Z",
                    "status": {
                      "code": 0
                    },
                    "traceGroup": "HTTP GET",
                    "traceGroupFields": {
                      "durationInNanos": 757000000,
                      "endTime": "2024-07-28T15:32:14.575Z",
                      "statusCode": 0
                    },
                    "traceId": "0272bba222d1ba0d2c0897e55d9c71f6",
                    "traceState": ""
                  },
                  {
                    "droppedAttributesCount": 0,
                    "droppedEventsCount": 0,
                    "droppedLinksCount": 0,
                    "durationInNanos": 757000000,
                    "endTime": "2024-07-28T15:32:14.575Z",
                    "events": [
                      {
                        "attributes": {},
                        "droppedAttributesCount": 0,
                        "name": "fetchStart",
                        "time": "2024-07-28T15:32:13.818600097Z"
                      },
                      {
                        "attributes": {},
                        "droppedAttributesCount": 0,
                        "name": "domainLookupStart",
                        "time": "2024-07-28T15:32:13.818600097Z"
                      },
                      {
                        "attributes": {},
                        "droppedAttributesCount": 0,
                        "name": "domainLookupEnd",
                        "time": "2024-07-28T15:32:13.818600097Z"
                      },
                      {
                        "attributes": {},
                        "droppedAttributesCount": 0,
                        "name": "connectStart",
                        "time": "2024-07-28T15:32:13.818600097Z"
                      },
                      {
                        "attributes": {},
                        "droppedAttributesCount": 0,
                        "name": "connectEnd",
                        "time": "2024-07-28T15:32:13.818600097Z"
                      },
                      {
                        "attributes": {},
                        "droppedAttributesCount": 0,
                        "name": "requestStart",
                        "time": "2024-07-28T15:32:14.115000097Z"
                      },
                      {
                        "attributes": {},
                        "droppedAttributesCount": 0,
                        "name": "responseStart",
                        "time": "2024-07-28T15:32:14.319700098Z"
                      },
                      {
                        "attributes": {},
                        "droppedAttributesCount": 0,
                        "name": "responseEnd",
                        "time": "2024-07-28T15:32:14.321200098Z"
                      }
                    ],
                    "instrumentationScope": {
                      "name": "@opentelemetry/instrumentation-fetch",
                      "version": "0.51.1"
                    },
                    "kind": "SPAN_KIND_CLIENT",
                    "links": [],
                    "name": "HTTP GET",
                    "parentSpanId": "",
                    "resource": {
                      "attributes": {
                        "k8s@deployment@name": "opentelemetry-demo-loadgenerator",
                        "k8s@namespace@name": "otel-demo",
                        "k8s@node@name": "ip-172-31-11-146.ec2.internal",
                        "k8s@pod@ip": "172.31.13.250",
                        "k8s@pod@name": "opentelemetry-demo-loadgenerator-d7f6f8c8c-9pkjc",
                        "k8s@pod@start_time": "2024-07-16T21:54:25Z",
                        "k8s@pod@uid": "8851ae96-f5df-42d6-9587-032f05a80fa8",
                        "process@runtime@description": "Web Browser",
                        "process@runtime@name": "browser",
                        "process@runtime@version": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/120.0.6099.28 Safari/537.36",
                        "service@instance@id": "8851ae96-f5df-42d6-9587-032f05a80fa8",
                        "service@name": "frontend-web",
                        "telemetry@sdk@language": "webjs",
                        "telemetry@sdk@name": "opentelemetry",
                        "telemetry@sdk@version": "1.24.1"
                      }
                    },
                    "serviceName": "frontend-web",
                    "span": {
                      "attributes": {
                        "app@synthetic_request": "true",
                        "component": "fetch",
                        "http@host": "opentelemetry-demo-frontendproxy:8080",
                        "http@method": "GET",
                        "http@response_content_length": 24,
                        "http@scheme": "http",
                        "http@status_code": 200,
                        "http@status_text": "OK",
                        "http@url": "http://opentelemetry-demo-frontendproxy:8080/api/cart?sessionId=7fa4fafb-875f-40c9-a839-c68560988a13&currencyCode=USD",
                        "http@user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/120.0.6099.28 Safari/537.36",
                        "session@id": "7fa4fafb-875f-40c9-a839-c68560988a13"
                      }
                    },
                    "spanId": "3d2e27909554bd49",
                    "startTime": "2024-07-28T15:32:13.818Z",
                    "status": {
                      "code": 0
                    },
                    "traceGroup": "HTTP GET",
                    "traceGroupFields": {
                      "durationInNanos": 757000000,
                      "endTime": "2024-07-28T15:32:14.575Z",
                      "statusCode": 0
                    },
                    "traceId": "0272bba222d1ba0d2c0897e55d9c71f6",
                    "traceState": ""
                  },
                  {
                    "droppedAttributesCount": 0,
                    "droppedEventsCount": 0,
                    "droppedLinksCount": 0,
                    "durationInNanos": 28042194,
                    "endTime": "2024-07-28T15:32:14.146042194Z",
                    "events": [],
                    "instrumentationScope": {
                      "name": "next.js",
                      "version": "0.0.1"
                    },
                    "kind": "SPAN_KIND_INTERNAL",
                    "links": [],
                    "name": "executing api route (pages) /api/cart",
                    "parentSpanId": "103f56279de45f1e",
                    "resource": {
                      "attributes": {
                        "container@id": "0139e7e2c3f3eedc6ae167adee962b9b1743245202d8c08a23fe5f59700d13b3",
                        "host@arch": "amd64",
                        "host@name": "opentelemetry-demo-frontend-865f97d679-5jmwp",
                        "k8s@deployment@name": "opentelemetry-demo-frontend",
                        "k8s@namespace@name": "otel-demo",
                        "k8s@node@name": "ip-172-31-37-251.ec2.internal",
                        "k8s@pod@ip": "172.31.34.40",
                        "k8s@pod@name": "opentelemetry-demo-frontend-865f97d679-5jmwp",
                        "k8s@pod@start_time": "2024-07-16T21:54:24Z",
                        "k8s@pod@uid": "d54f284d-b2f0-48c5-be8b-cfbf199f89cf",
                        "os@type": "linux",
                        "os@version": "5.10.219-208.866.amzn2.x86_64",
                        "process@command": "/app/server.js",
                        "process@command_args": "[\"/usr/local/bin/node\",\"--require\",\"./Instrumentation.js\",\"/app/server.js\"]",
                        "process@executable@name": "node",
                        "process@executable@path": "/usr/local/bin/node",
                        "process@owner": "nextjs",
                        "process@pid": 17,
                        "process@runtime@description": "Node.js",
                        "process@runtime@name": "nodejs",
                        "process@runtime@version": "18.20.3",
                        "service@instance@id": "d54f284d-b2f0-48c5-be8b-cfbf199f89cf",
                        "service@name": "frontend",
                        "service@namespace": "opentelemetry-demo",
                        "service@version": "1.11.0",
                        "telemetry@sdk@language": "nodejs",
                        "telemetry@sdk@name": "opentelemetry",
                        "telemetry@sdk@version": "1.24.1"
                      }
                    },
                    "serviceName": "frontend",
                    "span": {
                      "attributes": {
                        "http@status_code": 200,
                        "next@span_name": "executing api route (pages) /api/cart",
                        "next@span_type": "Node.runHandler"
                      }
                    },
                    "spanId": "477f1795a718fafb",
                    "startTime": "2024-07-28T15:32:14.118Z",
                    "status": {
                      "code": 0
                    },
                    "traceGroup": "HTTP GET",
                    "traceGroupFields": {
                      "durationInNanos": 757000000,
                      "endTime": "2024-07-28T15:32:14.575Z",
                      "statusCode": 0
                    },
                    "traceId": "0272bba222d1ba0d2c0897e55d9c71f6",
                    "traceState": ""
                  },
                  {
                    "droppedAttributesCount": 0,
                    "droppedEventsCount": 0,
                    "droppedLinksCount": 0,
                    "durationInNanos": 30947000,
                    "endTime": "2024-07-28T15:32:14.147229Z",
                    "events": [],
                    "kind": "SPAN_KIND_SERVER",
                    "links": [],
                    "name": "ingress",
                    "parentSpanId": "3d2e27909554bd49",
                    "resource": {
                      "attributes": {
                        "k8s@deployment@name": "opentelemetry-demo-frontendproxy",
                        "k8s@namespace@name": "otel-demo",
                        "k8s@node@name": "ip-172-31-11-146.ec2.internal",
                        "k8s@pod@ip": "172.31.5.154",
                        "k8s@pod@name": "opentelemetry-demo-frontendproxy-7f75d5c97b-z6qlm",
                        "k8s@pod@start_time": "2024-07-16T21:54:24Z",
                        "k8s@pod@uid": "a78e752a-8e8f-4263-9acd-6b37642237d3",
                        "service@instance@id": "a78e752a-8e8f-4263-9acd-6b37642237d3",
                        "service@name": "frontendproxy",
                        "service@namespace": "opentelemetry-demo",
                        "service@version": "1.11.0"
                      }
                    },
                    "serviceName": "frontendproxy",
                    "span": {
                      "attributes": {
                        "component": "proxy",
                        "downstream_cluster": "-",
                        "guid:x-request-id": "f37bf44a-f29d-9bb2-9815-3f9965441d49",
                        "http@method": "GET",
                        "http@protocol": "HTTP/1.1",
                        "http@status_code": "200",
                        "http@url": "http://opentelemetry-demo-frontendproxy:8080/api/cart?sessionId=7fa4fafb-875f-40c9-a839-c68560988a13&currencyCode=USD",
                        "node_id": "",
                        "peer@address": "172.31.13.250",
                        "request_size": "0",
                        "response_flags": "-",
                        "response_size": "24",
                        "upstream_cluster": "frontend",
                        "upstream_cluster@name": "frontend",
                        "user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/120.0.6099.28 Safari/537.36",
                        "zone": ""
                      }
                    },
                    "spanId": "9d1a0a31fb14bb1e",
                    "startTime": "2024-07-28T15:32:14.116282Z",
                    "status": {
                      "code": 0
                    },
                    "traceGroup": "HTTP GET",
                    "traceGroupFields": {
                      "durationInNanos": 757000000,
                      "endTime": "2024-07-28T15:32:14.575Z",
                      "statusCode": 0
                    },
                    "traceId": "0272bba222d1ba0d2c0897e55d9c71f6",
                    "traceState": ""
                  },
                  {
                    "droppedAttributesCount": 0,
                    "droppedEventsCount": 0,
                    "droppedLinksCount": 0,
                    "durationInNanos": 30906287,
                    "endTime": "2024-07-28T15:32:14.147906287Z",
                    "events": [],
                    "instrumentationScope": {
                      "name": "@opentelemetry/instrumentation-http",
                      "version": "0.51.1"
                    },
                    "kind": "SPAN_KIND_SERVER",
                    "links": [],
                    "name": "GET",
                    "parentSpanId": "19c1081f8f05ac8f",
                    "resource": {
                      "attributes": {
                        "container@id": "0139e7e2c3f3eedc6ae167adee962b9b1743245202d8c08a23fe5f59700d13b3",
                        "host@arch": "amd64",
                        "host@name": "opentelemetry-demo-frontend-865f97d679-5jmwp",
                        "k8s@deployment@name": "opentelemetry-demo-frontend",
                        "k8s@namespace@name": "otel-demo",
                        "k8s@node@name": "ip-172-31-37-251.ec2.internal",
                        "k8s@pod@ip": "172.31.34.40",
                        "k8s@pod@name": "opentelemetry-demo-frontend-865f97d679-5jmwp",
                        "k8s@pod@start_time": "2024-07-16T21:54:24Z",
                        "k8s@pod@uid": "d54f284d-b2f0-48c5-be8b-cfbf199f89cf",
                        "os@type": "linux",
                        "os@version": "5.10.219-208.866.amzn2.x86_64",
                        "process@command": "/app/server.js",
                        "process@command_args": "[\"/usr/local/bin/node\",\"--require\",\"./Instrumentation.js\",\"/app/server.js\"]",
                        "process@executable@name": "node",
                        "process@executable@path": "/usr/local/bin/node",
                        "process@owner": "nextjs",
                        "process@pid": 17,
                        "process@runtime@description": "Node.js",
                        "process@runtime@name": "nodejs",
                        "process@runtime@version": "18.20.3",
                        "service@instance@id": "d54f284d-b2f0-48c5-be8b-cfbf199f89cf",
                        "service@name": "frontend",
                        "service@namespace": "opentelemetry-demo",
                        "service@version": "1.11.0",
                        "telemetry@sdk@language": "nodejs",
                        "telemetry@sdk@name": "opentelemetry",
                        "telemetry@sdk@version": "1.24.1"
                      }
                    },
                    "serviceName": "frontend",
                    "span": {
                      "attributes": {
                        "http@flavor": "1.1",
                        "http@host": "opentelemetry-demo-frontendproxy:8080",
                        "http@method": "GET",
                        "http@scheme": "http",
                        "http@status_code": 200,
                        "http@status_text": "OK",
                        "http@target": "/api/cart?sessionId=7fa4fafb-875f-40c9-a839-c68560988a13&currencyCode=USD",
                        "http@url": "http://opentelemetry-demo-frontendproxy:8080/api/cart?sessionId=7fa4fafb-875f-40c9-a839-c68560988a13&currencyCode=USD",
                        "http@user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/120.0.6099.28 Safari/537.36",
                        "net@host@ip": "172.31.34.40",
                        "net@host@name": "opentelemetry-demo-frontendproxy",
                        "net@host@port": 8080,
                        "net@peer@ip": "172.31.5.154",
                        "net@peer@port": 48032,
                        "net@transport": "ip_tcp"
                      }
                    },
                    "spanId": "015bb8c502cfd036",
                    "startTime": "2024-07-28T15:32:14.117Z",
                    "status": {
                      "code": 0
                    },
                    "traceGroup": "HTTP GET",
                    "traceGroupFields": {
                      "durationInNanos": 757000000,
                      "endTime": "2024-07-28T15:32:14.575Z",
                      "statusCode": 0
                    },
                    "traceId": "0272bba222d1ba0d2c0897e55d9c71f6",
                    "traceState": ""
                  },
                  {
                    "droppedAttributesCount": 0,
                    "droppedEventsCount": 0,
                    "droppedLinksCount": 0,
                    "durationInNanos": 28806505,
                    "endTime": "2024-07-28T15:32:14.145806505Z",
                    "events": [],
                    "instrumentationScope": {
                      "name": "next.js",
                      "version": "0.0.1"
                    },
                    "kind": "SPAN_KIND_SERVER",
                    "links": [],
                    "name": "GET /api/cart?sessionId=7fa4fafb-875f-40c9-a839-c68560988a13&currencyCode=USD",
                    "parentSpanId": "015bb8c502cfd036",
                    "resource": {
                      "attributes": {
                        "container@id": "0139e7e2c3f3eedc6ae167adee962b9b1743245202d8c08a23fe5f59700d13b3",
                        "host@arch": "amd64",
                        "host@name": "opentelemetry-demo-frontend-865f97d679-5jmwp",
                        "k8s@deployment@name": "opentelemetry-demo-frontend",
                        "k8s@namespace@name": "otel-demo",
                        "k8s@node@name": "ip-172-31-37-251.ec2.internal",
                        "k8s@pod@ip": "172.31.34.40",
                        "k8s@pod@name": "opentelemetry-demo-frontend-865f97d679-5jmwp",
                        "k8s@pod@start_time": "2024-07-16T21:54:24Z",
                        "k8s@pod@uid": "d54f284d-b2f0-48c5-be8b-cfbf199f89cf",
                        "os@type": "linux",
                        "os@version": "5.10.219-208.866.amzn2.x86_64",
                        "process@command": "/app/server.js",
                        "process@command_args": "[\"/usr/local/bin/node\",\"--require\",\"./Instrumentation.js\",\"/app/server.js\"]",
                        "process@executable@name": "node",
                        "process@executable@path": "/usr/local/bin/node",
                        "process@owner": "nextjs",
                        "process@pid": 17,
                        "process@runtime@description": "Node.js",
                        "process@runtime@name": "nodejs",
                        "process@runtime@version": "18.20.3",
                        "service@instance@id": "d54f284d-b2f0-48c5-be8b-cfbf199f89cf",
                        "service@name": "frontend",
                        "service@namespace": "opentelemetry-demo",
                        "service@version": "1.11.0",
                        "telemetry@sdk@language": "nodejs",
                        "telemetry@sdk@name": "opentelemetry",
                        "telemetry@sdk@version": "1.24.1"
                      }
                    },
                    "serviceName": "frontend",
                    "span": {
                      "attributes": {
                        "http@method": "GET",
                        "http@status_code": 200,
                        "http@target": "/api/cart?sessionId=7fa4fafb-875f-40c9-a839-c68560988a13&currencyCode=USD",
                        "next@span_name": "GET /api/cart?sessionId=7fa4fafb-875f-40c9-a839-c68560988a13&currencyCode=USD",
                        "next@span_type": "BaseServer.handleRequest"
                      }
                    },
                    "spanId": "103f56279de45f1e",
                    "startTime": "2024-07-28T15:32:14.117Z",
                    "status": {
                      "code": 0
                    },
                    "traceGroup": "HTTP GET",
                    "traceGroupFields": {
                      "durationInNanos": 757000000,
                      "endTime": "2024-07-28T15:32:14.575Z",
                      "statusCode": 0
                    },
                    "traceId": "0272bba222d1ba0d2c0897e55d9c71f6",
                    "traceState": ""
                  },
                  {
                    "droppedAttributesCount": 0,
                    "droppedEventsCount": 0,
                    "droppedLinksCount": 0,
                    "durationInNanos": 3467000,
                    "endTime": "2024-07-28T15:32:14.122907200Z",
                    "events": [
                      {
                        "attributes": {},
                        "droppedAttributesCount": 0,
                        "name": "Fetch cart",
                        "time": "2024-07-28T15:32:14.121239100Z"
                      }
                    ],
                    "instrumentationScope": {
                      "name": "Microsoft.AspNetCore"
                    },
                    "kind": "SPAN_KIND_SERVER",
                    "links": [],
                    "name": "POST /oteldemo.CartService/GetCart",
                    "parentSpanId": "d864587007168cf0",
                    "resource": {
                      "attributes": {
                        "container@id": "cdfdc005564f3ece43d39609e4b67f765ca0b12a4a84584ae0c6222382027572",
                        "host@name": "opentelemetry-demo-cartservice-6ccdffdf85-9rrzm",
                        "k8s@deployment@name": "opentelemetry-demo-cartservice",
                        "k8s@namespace@name": "otel-demo",
                        "k8s@node@name": "ip-172-31-37-251.ec2.internal",
                        "k8s@pod@ip": "172.31.41.41",
                        "k8s@pod@name": "opentelemetry-demo-cartservice-6ccdffdf85-9rrzm",
                        "k8s@pod@start_time": "2024-07-16T21:54:23Z",
                        "k8s@pod@uid": "775bf942-c64a-41d1-8139-e71e6200a0d2",
                        "service@instance@id": "775bf942-c64a-41d1-8139-e71e6200a0d2",
                        "service@name": "cartservice",
                        "service@namespace": "opentelemetry-demo",
                        "service@version": "1.11.0",
                        "telemetry@sdk@language": "dotnet",
                        "telemetry@sdk@name": "opentelemetry",
                        "telemetry@sdk@version": "1.9.0"
                      }
                    },
                    "serviceName": "cartservice",
                    "span": {
                      "attributes": {
                        "app@cart@items@count": 0,
                        "app@user@id": "7fa4fafb-875f-40c9-a839-c68560988a13",
                        "grpc@method": "/oteldemo.CartService/GetCart",
                        "grpc@status_code": "0",
                        "http@request@method": "POST",
                        "http@response@status_code": 200,
                        "http@route": "/oteldemo.CartService/GetCart",
                        "network@protocol@version": "2",
                        "server@address": "opentelemetry-demo-cartservice",
                        "server@port": 8080,
                        "url@path": "/oteldemo.CartService/GetCart",
                        "url@scheme": "http",
                        "user_agent@original": "grpc-node-js/1.10.8"
                      }
                    },
                    "spanId": "5a4103e6aee350f5",
                    "startTime": "2024-07-28T15:32:14.119440200Z",
                    "status": {
                      "code": 0
                    },
                    "traceGroup": "HTTP GET",
                    "traceGroupFields": {
                      "durationInNanos": 757000000,
                      "endTime": "2024-07-28T15:32:14.575Z",
                      "statusCode": 0
                    },
                    "traceId": "0272bba222d1ba0d2c0897e55d9c71f6",
                    "traceState": ""
                  },
                  {
                    "droppedAttributesCount": 0,
                    "droppedEventsCount": 0,
                    "droppedLinksCount": 0,
                    "durationInNanos": 1529500,
                    "endTime": "2024-07-28T15:32:14.122811200Z",
                    "events": [
                      {
                        "attributes": {},
                        "droppedAttributesCount": 0,
                        "name": "Enqueued",
                        "time": "2024-07-28T15:32:14.121293300Z"
                      },
                      {
                        "attributes": {},
                        "droppedAttributesCount": 0,
                        "name": "Sent",
                        "time": "2024-07-28T15:32:14.121305Z"
                      },
                      {
                        "attributes": {},
                        "droppedAttributesCount": 0,
                        "name": "ResponseReceived",
                        "time": "2024-07-28T15:32:14.122810700Z"
                      }
                    ],
                    "instrumentationScope": {
                      "name": "OpenTelemetry.Instrumentation.StackExchangeRedis",
                      "version": "1.0.0-rc9.15"
                    },
                    "kind": "SPAN_KIND_CLIENT",
                    "links": [],
                    "name": "HGET",
                    "parentSpanId": "5a4103e6aee350f5",
                    "resource": {
                      "attributes": {
                        "container@id": "cdfdc005564f3ece43d39609e4b67f765ca0b12a4a84584ae0c6222382027572",
                        "host@name": "opentelemetry-demo-cartservice-6ccdffdf85-9rrzm",
                        "k8s@deployment@name": "opentelemetry-demo-cartservice",
                        "k8s@namespace@name": "otel-demo",
                        "k8s@node@name": "ip-172-31-37-251.ec2.internal",
                        "k8s@pod@ip": "172.31.41.41",
                        "k8s@pod@name": "opentelemetry-demo-cartservice-6ccdffdf85-9rrzm",
                        "k8s@pod@start_time": "2024-07-16T21:54:23Z",
                        "k8s@pod@uid": "775bf942-c64a-41d1-8139-e71e6200a0d2",
                        "service@instance@id": "775bf942-c64a-41d1-8139-e71e6200a0d2",
                        "service@name": "cartservice",
                        "service@namespace": "opentelemetry-demo",
                        "service@version": "1.11.0",
                        "telemetry@sdk@language": "dotnet",
                        "telemetry@sdk@name": "opentelemetry",
                        "telemetry@sdk@version": "1.9.0"
                      }
                    },
                    "serviceName": "cartservice",
                    "span": {
                      "attributes": {
                        "db@redis@database_index": 0,
                        "db@redis@flags": "None",
                        "db@statement": "HGET 7fa4fafb-875f-40c9-a839-c68560988a13",
                        "db@system": "redis",
                        "net@peer@name": "opentelemetry-demo-valkey",
                        "net@peer@port": 6379,
                        "peer@service": "opentelemetry-demo-valkey:6379"
                      }
                    },
                    "spanId": "02a302d0b7cb1047",
                    "startTime": "2024-07-28T15:32:14.121281700Z",
                    "status": {
                      "code": 0
                    },
                    "traceGroup": "HTTP GET",
                    "traceGroupFields": {
                      "durationInNanos": 757000000,
                      "endTime": "2024-07-28T15:32:14.575Z",
                      "statusCode": 0
                    },
                    "traceId": "0272bba222d1ba0d2c0897e55d9c71f6",
                    "traceState": ""
                  },
                  {
                    "droppedAttributesCount": 0,
                    "droppedEventsCount": 0,
                    "droppedLinksCount": 0,
                    "durationInNanos": 30706000,
                    "endTime": "2024-07-28T15:32:14.147189Z",
                    "events": [],
                    "kind": "SPAN_KIND_CLIENT",
                    "links": [],
                    "name": "router frontend egress",
                    "parentSpanId": "9d1a0a31fb14bb1e",
                    "resource": {
                      "attributes": {
                        "k8s@deployment@name": "opentelemetry-demo-frontendproxy",
                        "k8s@namespace@name": "otel-demo",
                        "k8s@node@name": "ip-172-31-11-146.ec2.internal",
                        "k8s@pod@ip": "172.31.5.154",
                        "k8s@pod@name": "opentelemetry-demo-frontendproxy-7f75d5c97b-z6qlm",
                        "k8s@pod@start_time": "2024-07-16T21:54:24Z",
                        "k8s@pod@uid": "a78e752a-8e8f-4263-9acd-6b37642237d3",
                        "service@instance@id": "a78e752a-8e8f-4263-9acd-6b37642237d3",
                        "service@name": "frontendproxy",
                        "service@namespace": "opentelemetry-demo",
                        "service@version": "1.11.0"
                      }
                    },
                    "serviceName": "frontendproxy",
                    "span": {
                      "attributes": {
                        "component": "proxy",
                        "http@protocol": "HTTP/1.1",
                        "http@status_code": "200",
                        "peer@address": "10.100.160.248:8080",
                        "response_flags": "-",
                        "upstream_address": "10.100.160.248:8080",
                        "upstream_cluster": "frontend",
                        "upstream_cluster@name": "frontend"
                      }
                    },
                    "spanId": "19c1081f8f05ac8f",
                    "startTime": "2024-07-28T15:32:14.116483Z",
                    "status": {
                      "code": 0
                    },
                    "traceGroup": "HTTP GET",
                    "traceGroupFields": {
                      "durationInNanos": 757000000,
                      "endTime": "2024-07-28T15:32:14.575Z",
                      "statusCode": 0
                    },
                    "traceId": "0272bba222d1ba0d2c0897e55d9c71f6",
                    "traceState": ""
                  }
                ],
                [
                  null,
                  null,
                  null,
                  null,
                  null,
                  null,
                  null,
                  null,
                  null
                ],
                [
                  0,
                  0,
                  0,
                  0,
                  0,
                  0,
                  0,
                  0,
                  0
                ],
                [
                  0,
                  0,
                  0,
                  0,
                  0,
                  0,
                  0,
                  0,
                  0
                ],
                [
                  0,
                  0,
                  0,
                  0,
                  0,
                  0,
                  0,
                  0,
                  0
                ],
                [
                  18.604885,
                  757,
                  28.042194,
                  30.947,
                  30.906287,
                  28.806504999999998,
                  3.4669999999999996,
                  1.5294999999999999,
                  30.706
                ],
                [
                  "2024-07-28T15:32:14.136604885Z",
                  "2024-07-28T15:32:14.575Z",
                  "2024-07-28T15:32:14.146042194Z",
                  "2024-07-28T15:32:14.147229Z",
                  "2024-07-28T15:32:14.147906287Z",
                  "2024-07-28T15:32:14.145806505Z",
                  "2024-07-28T15:32:14.122907200Z",
                  "2024-07-28T15:32:14.122811200Z",
                  "2024-07-28T15:32:14.147189Z"
                ],
                [
                  {
                    "name": "@opentelemetry/instrumentation-grpc",
                    "version": "0.51.1"
                  },
                  {
                    "name": "@opentelemetry/instrumentation-fetch",
                    "version": "0.51.1"
                  },
                  {
                    "name": "next.js",
                    "version": "0.0.1"
                  },
                  null,
                  {
                    "name": "@opentelemetry/instrumentation-http",
                    "version": "0.51.1"
                  },
                  {
                    "name": "next.js",
                    "version": "0.0.1"
                  },
                  {
                    "name": "Microsoft.AspNetCore"
                  },
                  {
                    "name": "OpenTelemetry.Instrumentation.StackExchangeRedis",
                    "version": "1.0.0-rc9.15"
                  },
                  null
                ],
                [
                  "SPAN_KIND_CLIENT",
                  "SPAN_KIND_CLIENT",
                  "SPAN_KIND_INTERNAL",
                  "SPAN_KIND_SERVER",
                  "SPAN_KIND_SERVER",
                  "SPAN_KIND_SERVER",
                  "SPAN_KIND_SERVER",
                  "SPAN_KIND_CLIENT",
                  "SPAN_KIND_CLIENT"
                ],
                [
                  [],
                  [],
                  [],
                  [],
                  [],
                  [],
                  [],
                  [],
                  []
                ],
                [
                  [],
                  [
                    {
                      "fields": [
                        {
                          "key": "name",
                          "value": "fetchStart"
                        }
                      ],
                      "timestamp": 1722180733818
                    },
                    {
                      "fields": [
                        {
                          "key": "name",
                          "value": "domainLookupStart"
                        }
                      ],
                      "timestamp": 1722180733818
                    },
                    {
                      "fields": [
                        {
                          "key": "name",
                          "value": "domainLookupEnd"
                        }
                      ],
                      "timestamp": 1722180733818
                    },
                    {
                      "fields": [
                        {
                          "key": "name",
                          "value": "connectStart"
                        }
                      ],
                      "timestamp": 1722180733818
                    },
                    {
                      "fields": [
                        {
                          "key": "name",
                          "value": "connectEnd"
                        }
                      ],
                      "timestamp": 1722180733818
                    },
                    {
                      "fields": [
                        {
                          "key": "name",
                          "value": "requestStart"
                        }
                      ],
                      "timestamp": 1722180734115
                    },
                    {
                      "fields": [
                        {
                          "key": "name",
                          "value": "responseStart"
                        }
                      ],
                      "timestamp": 1722180734319
                    },
                    {
                      "fields": [
                        {
                          "key": "name",
                          "value": "responseEnd"
                        }
                      ],
                      "timestamp": 1722180734321
                    }
                  ],
                  [],
                  [],
                  [],
                  [],
                  [
                    {
                      "fields": [
                        {
                          "key": "name",
                          "value": "Fetch cart"
                        }
                      ],
                      "timestamp": 1722180734121
                    }
                  ],
                  [
                    {
                      "fields": [
                        {
                          "key": "name",
                          "value": "Enqueued"
                        }
                      ],
                      "timestamp": 1722180734121
                    },
                    {
                      "fields": [
                        {
                          "key": "name",
                          "value": "Sent"
                        }
                      ],
                      "timestamp": 1722180734121
                    },
                    {
                      "fields": [
                        {
                          "key": "name",
                          "value": "ResponseReceived"
                        }
                      ],
                      "timestamp": 1722180734122
                    }
                  ],
                  []
                ],
                [
                  "grpc.oteldemo.CartService/GetCart",
                  "HTTP GET",
                  "executing api route (pages) /api/cart",
                  "ingress",
                  "GET",
                  "GET /api/cart?sessionId=7fa4fafb-875f-40c9-a839-c68560988a13&currencyCode=USD",
                  "POST /oteldemo.CartService/GetCart",
                  "HGET",
                  "router frontend egress"
                ],
                [
                  "477f1795a718fafb",
                  "",
                  "103f56279de45f1e",
                  "3d2e27909554bd49",
                  "19c1081f8f05ac8f",
                  "015bb8c502cfd036",
                  "d864587007168cf0",
                  "5a4103e6aee350f5",
                  "9d1a0a31fb14bb1e"
                ],
                [
                  "frontend",
                  "frontend-web",
                  "frontend",
                  "frontendproxy",
                  "frontend",
                  "frontend",
                  "cartservice",
                  "cartservice",
                  "frontendproxy"
                ],
                [
                  [
                    {
                      "key": "service@instance@id",
                      "value": "d54f284d-b2f0-48c5-be8b-cfbf199f89cf"
                    },
                    {
                      "key": "host@arch",
                      "value": "amd64"
                    },
                    {
                      "key": "k8s@node@name",
                      "value": "ip-172-31-37-251.ec2.internal"
                    },
                    {
                      "key": "process@pid",
                      "value": 17
                    },
                    {
                      "key": "k8s@pod@start_time",
                      "value": "2024-07-16T21:54:24Z"
                    },
                    {
                      "key": "os@type",
                      "value": "linux"
                    },
                    {
                      "key": "k8s@namespace@name",
                      "value": "otel-demo"
                    },
                    {
                      "key": "k8s@deployment@name",
                      "value": "opentelemetry-demo-frontend"
                    },
                    {
                      "key": "process@executable@path",
                      "value": "/usr/local/bin/node"
                    },
                    {
                      "key": "process@executable@name",
                      "value": "node"
                    },
                    {
                      "key": "process@runtime@name",
                      "value": "nodejs"
                    },
                    {
                      "key": "telemetry@sdk@name",
                      "value": "opentelemetry"
                    },
                    {
                      "key": "process@runtime@description",
                      "value": "Node.js"
                    },
                    {
                      "key": "process@command",
                      "value": "/app/server.js"
                    },
                    {
                      "key": "service@version",
                      "value": "1.11.0"
                    },
                    {
                      "key": "service@namespace",
                      "value": "opentelemetry-demo"
                    },
                    {
                      "key": "process@owner",
                      "value": "nextjs"
                    },
                    {
                      "key": "k8s@pod@ip",
                      "value": "172.31.34.40"
                    },
                    {
                      "key": "k8s@pod@uid",
                      "value": "d54f284d-b2f0-48c5-be8b-cfbf199f89cf"
                    },
                    {
                      "key": "container@id",
                      "value": "0139e7e2c3f3eedc6ae167adee962b9b1743245202d8c08a23fe5f59700d13b3"
                    },
                    {
                      "key": "telemetry@sdk@version",
                      "value": "1.24.1"
                    },
                    {
                      "key": "host@name",
                      "value": "opentelemetry-demo-frontend-865f97d679-5jmwp"
                    },
                    {
                      "key": "telemetry@sdk@language",
                      "value": "nodejs"
                    },
                    {
                      "key": "k8s@pod@name",
                      "value": "opentelemetry-demo-frontend-865f97d679-5jmwp"
                    },
                    {
                      "key": "process@runtime@version",
                      "value": "18.20.3"
                    },
                    {
                      "key": "service@name",
                      "value": "frontend"
                    },
                    {
                      "key": "os@version",
                      "value": "5.10.219-208.866.amzn2.x86_64"
                    },
                    {
                      "key": "process@command_args",
                      "value": "[\"/usr/local/bin/node\",\"--require\",\"./Instrumentation.js\",\"/app/server.js\"]"
                    }
                  ],
                  [
                    {
                      "key": "process@runtime@description",
                      "value": "Web Browser"
                    },
                    {
                      "key": "k8s@pod@name",
                      "value": "opentelemetry-demo-loadgenerator-d7f6f8c8c-9pkjc"
                    },
                    {
                      "key": "k8s@pod@uid",
                      "value": "8851ae96-f5df-42d6-9587-032f05a80fa8"
                    },
                    {
                      "key": "telemetry@sdk@language",
                      "value": "webjs"
                    },
                    {
                      "key": "k8s@pod@start_time",
                      "value": "2024-07-16T21:54:25Z"
                    },
                    {
                      "key": "k8s@node@name",
                      "value": "ip-172-31-11-146.ec2.internal"
                    },
                    {
                      "key": "telemetry@sdk@version",
                      "value": "1.24.1"
                    },
                    {
                      "key": "k8s@namespace@name",
                      "value": "otel-demo"
                    },
                    {
                      "key": "k8s@deployment@name",
                      "value": "opentelemetry-demo-loadgenerator"
                    },
                    {
                      "key": "service@instance@id",
                      "value": "8851ae96-f5df-42d6-9587-032f05a80fa8"
                    },
                    {
                      "key": "k8s@pod@ip",
                      "value": "172.31.13.250"
                    },
                    {
                      "key": "process@runtime@version",
                      "value": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/120.0.6099.28 Safari/537.36"
                    },
                    {
                      "key": "telemetry@sdk@name",
                      "value": "opentelemetry"
                    },
                    {
                      "key": "process@runtime@name",
                      "value": "browser"
                    },
                    {
                      "key": "service@name",
                      "value": "frontend-web"
                    }
                  ],
                  [
                    {
                      "key": "process@executable@path",
                      "value": "/usr/local/bin/node"
                    },
                    {
                      "key": "telemetry@sdk@language",
                      "value": "nodejs"
                    },
                    {
                      "key": "os@version",
                      "value": "5.10.219-208.866.amzn2.x86_64"
                    },
                    {
                      "key": "process@runtime@description",
                      "value": "Node.js"
                    },
                    {
                      "key": "host@name",
                      "value": "opentelemetry-demo-frontend-865f97d679-5jmwp"
                    },
                    {
                      "key": "k8s@pod@start_time",
                      "value": "2024-07-16T21:54:24Z"
                    },
                    {
                      "key": "os@type",
                      "value": "linux"
                    },
                    {
                      "key": "container@id",
                      "value": "0139e7e2c3f3eedc6ae167adee962b9b1743245202d8c08a23fe5f59700d13b3"
                    },
                    {
                      "key": "process@runtime@version",
                      "value": "18.20.3"
                    },
                    {
                      "key": "process@executable@name",
                      "value": "node"
                    },
                    {
                      "key": "host@arch",
                      "value": "amd64"
                    },
                    {
                      "key": "k8s@node@name",
                      "value": "ip-172-31-37-251.ec2.internal"
                    },
                    {
                      "key": "service@namespace",
                      "value": "opentelemetry-demo"
                    },
                    {
                      "key": "process@command",
                      "value": "/app/server.js"
                    },
                    {
                      "key": "k8s@pod@name",
                      "value": "opentelemetry-demo-frontend-865f97d679-5jmwp"
                    },
                    {
                      "key": "telemetry@sdk@name",
                      "value": "opentelemetry"
                    },
                    {
                      "key": "telemetry@sdk@version",
                      "value": "1.24.1"
                    },
                    {
                      "key": "k8s@pod@ip",
                      "value": "172.31.34.40"
                    },
                    {
                      "key": "k8s@deployment@name",
                      "value": "opentelemetry-demo-frontend"
                    },
                    {
                      "key": "k8s@namespace@name",
                      "value": "otel-demo"
                    },
                    {
                      "key": "process@runtime@name",
                      "value": "nodejs"
                    },
                    {
                      "key": "service@version",
                      "value": "1.11.0"
                    },
                    {
                      "key": "process@owner",
                      "value": "nextjs"
                    },
                    {
                      "key": "k8s@pod@uid",
                      "value": "d54f284d-b2f0-48c5-be8b-cfbf199f89cf"
                    },
                    {
                      "key": "process@pid",
                      "value": 17
                    },
                    {
                      "key": "service@name",
                      "value": "frontend"
                    },
                    {
                      "key": "service@instance@id",
                      "value": "d54f284d-b2f0-48c5-be8b-cfbf199f89cf"
                    },
                    {
                      "key": "process@command_args",
                      "value": "[\"/usr/local/bin/node\",\"--require\",\"./Instrumentation.js\",\"/app/server.js\"]"
                    }
                  ],
                  [
                    {
                      "key": "service@version",
                      "value": "1.11.0"
                    },
                    {
                      "key": "k8s@node@name",
                      "value": "ip-172-31-11-146.ec2.internal"
                    },
                    {
                      "key": "service@name",
                      "value": "frontendproxy"
                    },
                    {
                      "key": "k8s@pod@name",
                      "value": "opentelemetry-demo-frontendproxy-7f75d5c97b-z6qlm"
                    },
                    {
                      "key": "k8s@pod@start_time",
                      "value": "2024-07-16T21:54:24Z"
                    },
                    {
                      "key": "service@instance@id",
                      "value": "a78e752a-8e8f-4263-9acd-6b37642237d3"
                    },
                    {
                      "key": "k8s@deployment@name",
                      "value": "opentelemetry-demo-frontendproxy"
                    },
                    {
                      "key": "k8s@pod@uid",
                      "value": "a78e752a-8e8f-4263-9acd-6b37642237d3"
                    },
                    {
                      "key": "k8s@pod@ip",
                      "value": "172.31.5.154"
                    },
                    {
                      "key": "k8s@namespace@name",
                      "value": "otel-demo"
                    },
                    {
                      "key": "service@namespace",
                      "value": "opentelemetry-demo"
                    }
                  ],
                  [
                    {
                      "key": "host@arch",
                      "value": "amd64"
                    },
                    {
                      "key": "k8s@pod@name",
                      "value": "opentelemetry-demo-frontend-865f97d679-5jmwp"
                    },
                    {
                      "key": "os@type",
                      "value": "linux"
                    },
                    {
                      "key": "k8s@pod@ip",
                      "value": "172.31.34.40"
                    },
                    {
                      "key": "os@version",
                      "value": "5.10.219-208.866.amzn2.x86_64"
                    },
                    {
                      "key": "k8s@node@name",
                      "value": "ip-172-31-37-251.ec2.internal"
                    },
                    {
                      "key": "container@id",
                      "value": "0139e7e2c3f3eedc6ae167adee962b9b1743245202d8c08a23fe5f59700d13b3"
                    },
                    {
                      "key": "k8s@pod@uid",
                      "value": "d54f284d-b2f0-48c5-be8b-cfbf199f89cf"
                    },
                    {
                      "key": "process@pid",
                      "value": 17
                    },
                    {
                      "key": "process@command_args",
                      "value": "[\"/usr/local/bin/node\",\"--require\",\"./Instrumentation.js\",\"/app/server.js\"]"
                    },
                    {
                      "key": "process@runtime@name",
                      "value": "nodejs"
                    },
                    {
                      "key": "host@name",
                      "value": "opentelemetry-demo-frontend-865f97d679-5jmwp"
                    },
                    {
                      "key": "k8s@pod@start_time",
                      "value": "2024-07-16T21:54:24Z"
                    },
                    {
                      "key": "service@version",
                      "value": "1.11.0"
                    },
                    {
                      "key": "process@runtime@version",
                      "value": "18.20.3"
                    },
                    {
                      "key": "service@name",
                      "value": "frontend"
                    },
                    {
                      "key": "process@owner",
                      "value": "nextjs"
                    },
                    {
                      "key": "k8s@namespace@name",
                      "value": "otel-demo"
                    },
                    {
                      "key": "process@executable@path",
                      "value": "/usr/local/bin/node"
                    },
                    {
                      "key": "telemetry@sdk@language",
                      "value": "nodejs"
                    },
                    {
                      "key": "process@command",
                      "value": "/app/server.js"
                    },
                    {
                      "key": "process@executable@name",
                      "value": "node"
                    },
                    {
                      "key": "service@namespace",
                      "value": "opentelemetry-demo"
                    },
                    {
                      "key": "k8s@deployment@name",
                      "value": "opentelemetry-demo-frontend"
                    },
                    {
                      "key": "telemetry@sdk@version",
                      "value": "1.24.1"
                    },
                    {
                      "key": "telemetry@sdk@name",
                      "value": "opentelemetry"
                    },
                    {
                      "key": "process@runtime@description",
                      "value": "Node.js"
                    },
                    {
                      "key": "service@instance@id",
                      "value": "d54f284d-b2f0-48c5-be8b-cfbf199f89cf"
                    }
                  ],
                  [
                    {
                      "key": "process@owner",
                      "value": "nextjs"
                    },
                    {
                      "key": "k8s@pod@uid",
                      "value": "d54f284d-b2f0-48c5-be8b-cfbf199f89cf"
                    },
                    {
                      "key": "k8s@pod@name",
                      "value": "opentelemetry-demo-frontend-865f97d679-5jmwp"
                    },
                    {
                      "key": "container@id",
                      "value": "0139e7e2c3f3eedc6ae167adee962b9b1743245202d8c08a23fe5f59700d13b3"
                    },
                    {
                      "key": "telemetry@sdk@name",
                      "value": "opentelemetry"
                    },
                    {
                      "key": "host@arch",
                      "value": "amd64"
                    },
                    {
                      "key": "service@version",
                      "value": "1.11.0"
                    },
                    {
                      "key": "process@pid",
                      "value": 17
                    },
                    {
                      "key": "os@type",
                      "value": "linux"
                    },
                    {
                      "key": "process@executable@path",
                      "value": "/usr/local/bin/node"
                    },
                    {
                      "key": "host@name",
                      "value": "opentelemetry-demo-frontend-865f97d679-5jmwp"
                    },
                    {
                      "key": "os@version",
                      "value": "5.10.219-208.866.amzn2.x86_64"
                    },
                    {
                      "key": "k8s@deployment@name",
                      "value": "opentelemetry-demo-frontend"
                    },
                    {
                      "key": "k8s@namespace@name",
                      "value": "otel-demo"
                    },
                    {
                      "key": "process@command",
                      "value": "/app/server.js"
                    },
                    {
                      "key": "process@runtime@description",
                      "value": "Node.js"
                    },
                    {
                      "key": "service@instance@id",
                      "value": "d54f284d-b2f0-48c5-be8b-cfbf199f89cf"
                    },
                    {
                      "key": "process@runtime@name",
                      "value": "nodejs"
                    },
                    {
                      "key": "process@executable@name",
                      "value": "node"
                    },
                    {
                      "key": "service@name",
                      "value": "frontend"
                    },
                    {
                      "key": "telemetry@sdk@version",
                      "value": "1.24.1"
                    },
                    {
                      "key": "process@runtime@version",
                      "value": "18.20.3"
                    },
                    {
                      "key": "process@command_args",
                      "value": "[\"/usr/local/bin/node\",\"--require\",\"./Instrumentation.js\",\"/app/server.js\"]"
                    },
                    {
                      "key": "service@namespace",
                      "value": "opentelemetry-demo"
                    },
                    {
                      "key": "telemetry@sdk@language",
                      "value": "nodejs"
                    },
                    {
                      "key": "k8s@node@name",
                      "value": "ip-172-31-37-251.ec2.internal"
                    },
                    {
                      "key": "k8s@pod@start_time",
                      "value": "2024-07-16T21:54:24Z"
                    },
                    {
                      "key": "k8s@pod@ip",
                      "value": "172.31.34.40"
                    }
                  ],
                  [
                    {
                      "key": "telemetry@sdk@version",
                      "value": "1.9.0"
                    },
                    {
                      "key": "k8s@namespace@name",
                      "value": "otel-demo"
                    },
                    {
                      "key": "container@id",
                      "value": "cdfdc005564f3ece43d39609e4b67f765ca0b12a4a84584ae0c6222382027572"
                    },
                    {
                      "key": "telemetry@sdk@language",
                      "value": "dotnet"
                    },
                    {
                      "key": "telemetry@sdk@name",
                      "value": "opentelemetry"
                    },
                    {
                      "key": "k8s@node@name",
                      "value": "ip-172-31-37-251.ec2.internal"
                    },
                    {
                      "key": "k8s@pod@ip",
                      "value": "172.31.41.41"
                    },
                    {
                      "key": "k8s@pod@name",
                      "value": "opentelemetry-demo-cartservice-6ccdffdf85-9rrzm"
                    },
                    {
                      "key": "host@name",
                      "value": "opentelemetry-demo-cartservice-6ccdffdf85-9rrzm"
                    },
                    {
                      "key": "k8s@pod@start_time",
                      "value": "2024-07-16T21:54:23Z"
                    },
                    {
                      "key": "service@name",
                      "value": "cartservice"
                    },
                    {
                      "key": "service@instance@id",
                      "value": "775bf942-c64a-41d1-8139-e71e6200a0d2"
                    },
                    {
                      "key": "service@version",
                      "value": "1.11.0"
                    },
                    {
                      "key": "k8s@pod@uid",
                      "value": "775bf942-c64a-41d1-8139-e71e6200a0d2"
                    },
                    {
                      "key": "service@namespace",
                      "value": "opentelemetry-demo"
                    },
                    {
                      "key": "k8s@deployment@name",
                      "value": "opentelemetry-demo-cartservice"
                    }
                  ],
                  [
                    {
                      "key": "service@version",
                      "value": "1.11.0"
                    },
                    {
                      "key": "k8s@pod@ip",
                      "value": "172.31.41.41"
                    },
                    {
                      "key": "k8s@pod@start_time",
                      "value": "2024-07-16T21:54:23Z"
                    },
                    {
                      "key": "k8s@deployment@name",
                      "value": "opentelemetry-demo-cartservice"
                    },
                    {
                      "key": "k8s@namespace@name",
                      "value": "otel-demo"
                    },
                    {
                      "key": "service@instance@id",
                      "value": "775bf942-c64a-41d1-8139-e71e6200a0d2"
                    },
                    {
                      "key": "telemetry@sdk@name",
                      "value": "opentelemetry"
                    },
                    {
                      "key": "host@name",
                      "value": "opentelemetry-demo-cartservice-6ccdffdf85-9rrzm"
                    },
                    {
                      "key": "k8s@node@name",
                      "value": "ip-172-31-37-251.ec2.internal"
                    },
                    {
                      "key": "k8s@pod@uid",
                      "value": "775bf942-c64a-41d1-8139-e71e6200a0d2"
                    },
                    {
                      "key": "container@id",
                      "value": "cdfdc005564f3ece43d39609e4b67f765ca0b12a4a84584ae0c6222382027572"
                    },
                    {
                      "key": "service@namespace",
                      "value": "opentelemetry-demo"
                    },
                    {
                      "key": "k8s@pod@name",
                      "value": "opentelemetry-demo-cartservice-6ccdffdf85-9rrzm"
                    },
                    {
                      "key": "telemetry@sdk@language",
                      "value": "dotnet"
                    },
                    {
                      "key": "telemetry@sdk@version",
                      "value": "1.9.0"
                    },
                    {
                      "key": "service@name",
                      "value": "cartservice"
                    }
                  ],
                  [
                    {
                      "key": "k8s@pod@uid",
                      "value": "a78e752a-8e8f-4263-9acd-6b37642237d3"
                    },
                    {
                      "key": "k8s@namespace@name",
                      "value": "otel-demo"
                    },
                    {
                      "key": "k8s@pod@start_time",
                      "value": "2024-07-16T21:54:24Z"
                    },
                    {
                      "key": "service@instance@id",
                      "value": "a78e752a-8e8f-4263-9acd-6b37642237d3"
                    },
                    {
                      "key": "service@version",
                      "value": "1.11.0"
                    },
                    {
                      "key": "k8s@deployment@name",
                      "value": "opentelemetry-demo-frontendproxy"
                    },
                    {
                      "key": "service@namespace",
                      "value": "opentelemetry-demo"
                    },
                    {
                      "key": "k8s@pod@ip",
                      "value": "172.31.5.154"
                    },
                    {
                      "key": "k8s@pod@name",
                      "value": "opentelemetry-demo-frontendproxy-7f75d5c97b-z6qlm"
                    },
                    {
                      "key": "k8s@node@name",
                      "value": "ip-172-31-11-146.ec2.internal"
                    },
                    {
                      "key": "service@name",
                      "value": "frontendproxy"
                    }
                  ]
                ],
                [
                  "d864587007168cf0",
                  "3d2e27909554bd49",
                  "477f1795a718fafb",
                  "9d1a0a31fb14bb1e",
                  "015bb8c502cfd036",
                  "103f56279de45f1e",
                  "5a4103e6aee350f5",
                  "02a302d0b7cb1047",
                  "19c1081f8f05ac8f"
                ],
                [
                  1722180734118,
                  1722180733818,
                  1722180734118,
                  1722180734116,
                  1722180734117,
                  1722180734117,
                  1722180734119,
                  1722180734121,
                  1722180734116
                ],
                [
                  {
                    "code": 0
                  },
                  {
                    "code": 0
                  },
                  {
                    "code": 0
                  },
                  {
                    "code": 0
                  },
                  {
                    "code": 0
                  },
                  {
                    "code": 0
                  },
                  {
                    "code": 0
                  },
                  {
                    "code": 0
                  },
                  {
                    "code": 0
                  }
                ],
                [
                  [
                    {
                      "key": "rpc@method",
                      "value": "GetCart"
                    },
                    {
                      "key": "rpc@service",
                      "value": "oteldemo.CartService"
                    },
                    {
                      "key": "rpc@grpc@status_code",
                      "value": 0
                    },
                    {
                      "key": "net@peer@port",
                      "value": 8080
                    },
                    {
                      "key": "rpc@system",
                      "value": "grpc"
                    },
                    {
                      "key": "net@peer@name",
                      "value": "opentelemetry-demo-cartservice"
                    }
                  ],
                  [
                    {
                      "key": "http@status_code",
                      "value": 200
                    },
                    {
                      "key": "component",
                      "value": "fetch"
                    },
                    {
                      "key": "session@id",
                      "value": "7fa4fafb-875f-40c9-a839-c68560988a13"
                    },
                    {
                      "key": "http@user_agent",
                      "value": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/120.0.6099.28 Safari/537.36"
                    },
                    {
                      "key": "http@scheme",
                      "value": "http"
                    },
                    {
                      "key": "app@synthetic_request",
                      "value": "true"
                    },
                    {
                      "key": "http@url",
                      "value": "http://opentelemetry-demo-frontendproxy:8080/api/cart?sessionId=7fa4fafb-875f-40c9-a839-c68560988a13&currencyCode=USD"
                    },
                    {
                      "key": "http@status_text",
                      "value": "OK"
                    },
                    {
                      "key": "http@method",
                      "value": "GET"
                    },
                    {
                      "key": "http@response_content_length",
                      "value": 24
                    },
                    {
                      "key": "http@host",
                      "value": "opentelemetry-demo-frontendproxy:8080"
                    }
                  ],
                  [
                    {
                      "key": "next@span_type",
                      "value": "Node.runHandler"
                    },
                    {
                      "key": "next@span_name",
                      "value": "executing api route (pages) /api/cart"
                    },
                    {
                      "key": "http@status_code",
                      "value": 200
                    }
                  ],
                  [
                    {
                      "key": "http@protocol",
                      "value": "HTTP/1.1"
                    },
                    {
                      "key": "response_size",
                      "value": "24"
                    },
                    {
                      "key": "response_flags",
                      "value": "-"
                    },
                    {
                      "key": "zone",
                      "value": ""
                    },
                    {
                      "key": "http@status_code",
                      "value": "200"
                    },
                    {
                      "key": "http@method",
                      "value": "GET"
                    },
                    {
                      "key": "user_agent",
                      "value": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/120.0.6099.28 Safari/537.36"
                    },
                    {
                      "key": "node_id",
                      "value": ""
                    },
                    {
                      "key": "http@url",
                      "value": "http://opentelemetry-demo-frontendproxy:8080/api/cart?sessionId=7fa4fafb-875f-40c9-a839-c68560988a13&currencyCode=USD"
                    },
                    {
                      "key": "request_size",
                      "value": "0"
                    },
                    {
                      "key": "downstream_cluster",
                      "value": "-"
                    },
                    {
                      "key": "peer@address",
                      "value": "172.31.13.250"
                    },
                    {
                      "key": "upstream_cluster",
                      "value": "frontend"
                    },
                    {
                      "key": "guid:x-request-id",
                      "value": "f37bf44a-f29d-9bb2-9815-3f9965441d49"
                    },
                    {
                      "key": "component",
                      "value": "proxy"
                    },
                    {
                      "key": "upstream_cluster@name",
                      "value": "frontend"
                    }
                  ],
                  [
                    {
                      "key": "http@method",
                      "value": "GET"
                    },
                    {
                      "key": "net@peer@ip",
                      "value": "172.31.5.154"
                    },
                    {
                      "key": "net@host@name",
                      "value": "opentelemetry-demo-frontendproxy"
                    },
                    {
                      "key": "http@user_agent",
                      "value": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/120.0.6099.28 Safari/537.36"
                    },
                    {
                      "key": "net@host@port",
                      "value": 8080
                    },
                    {
                      "key": "http@status_code",
                      "value": 200
                    },
                    {
                      "key": "http@scheme",
                      "value": "http"
                    },
                    {
                      "key": "net@peer@port",
                      "value": 48032
                    },
                    {
                      "key": "http@host",
                      "value": "opentelemetry-demo-frontendproxy:8080"
                    },
                    {
                      "key": "http@status_text",
                      "value": "OK"
                    },
                    {
                      "key": "http@target",
                      "value": "/api/cart?sessionId=7fa4fafb-875f-40c9-a839-c68560988a13&currencyCode=USD"
                    },
                    {
                      "key": "net@host@ip",
                      "value": "172.31.34.40"
                    },
                    {
                      "key": "net@transport",
                      "value": "ip_tcp"
                    },
                    {
                      "key": "http@flavor",
                      "value": "1.1"
                    },
                    {
                      "key": "http@url",
                      "value": "http://opentelemetry-demo-frontendproxy:8080/api/cart?sessionId=7fa4fafb-875f-40c9-a839-c68560988a13&currencyCode=USD"
                    }
                  ],
                  [
                    {
                      "key": "http@status_code",
                      "value": 200
                    },
                    {
                      "key": "next@span_name",
                      "value": "GET /api/cart?sessionId=7fa4fafb-875f-40c9-a839-c68560988a13&currencyCode=USD"
                    },
                    {
                      "key": "http@target",
                      "value": "/api/cart?sessionId=7fa4fafb-875f-40c9-a839-c68560988a13&currencyCode=USD"
                    },
                    {
                      "key": "http@method",
                      "value": "GET"
                    },
                    {
                      "key": "next@span_type",
                      "value": "BaseServer.handleRequest"
                    }
                  ],
                  [
                    {
                      "key": "app@cart@items@count",
                      "value": 0
                    },
                    {
                      "key": "http@route",
                      "value": "/oteldemo.CartService/GetCart"
                    },
                    {
                      "key": "http@response@status_code",
                      "value": 200
                    },
                    {
                      "key": "url@path",
                      "value": "/oteldemo.CartService/GetCart"
                    },
                    {
                      "key": "http@request@method",
                      "value": "POST"
                    },
                    {
                      "key": "server@address",
                      "value": "opentelemetry-demo-cartservice"
                    },
                    {
                      "key": "app@user@id",
                      "value": "7fa4fafb-875f-40c9-a839-c68560988a13"
                    },
                    {
                      "key": "url@scheme",
                      "value": "http"
                    },
                    {
                      "key": "user_agent@original",
                      "value": "grpc-node-js/1.10.8"
                    },
                    {
                      "key": "network@protocol@version",
                      "value": "2"
                    },
                    {
                      "key": "grpc@method",
                      "value": "/oteldemo.CartService/GetCart"
                    },
                    {
                      "key": "grpc@status_code",
                      "value": "0"
                    },
                    {
                      "key": "server@port",
                      "value": 8080
                    }
                  ],
                  [
                    {
                      "key": "net@peer@port",
                      "value": 6379
                    },
                    {
                      "key": "db@system",
                      "value": "redis"
                    },
                    {
                      "key": "peer@service",
                      "value": "opentelemetry-demo-valkey:6379"
                    },
                    {
                      "key": "net@peer@name",
                      "value": "opentelemetry-demo-valkey"
                    },
                    {
                      "key": "db@redis@database_index",
                      "value": 0
                    },
                    {
                      "key": "db@statement",
                      "value": "HGET 7fa4fafb-875f-40c9-a839-c68560988a13"
                    },
                    {
                      "key": "db@redis@flags",
                      "value": "None"
                    }
                  ],
                  [
                    {
                      "key": "peer@address",
                      "value": "10.100.160.248:8080"
                    },
                    {
                      "key": "http@status_code",
                      "value": "200"
                    },
                    {
                      "key": "http@protocol",
                      "value": "HTTP/1.1"
                    },
                    {
                      "key": "upstream_cluster",
                      "value": "frontend"
                    },
                    {
                      "key": "component",
                      "value": "proxy"
                    },
                    {
                      "key": "upstream_cluster@name",
                      "value": "frontend"
                    },
                    {
                      "key": "response_flags",
                      "value": "-"
                    },
                    {
                      "key": "upstream_address",
                      "value": "10.100.160.248:8080"
                    }
                  ]
                ],
                [
                  "HTTP GET",
                  "HTTP GET",
                  "HTTP GET",
                  "HTTP GET",
                  "HTTP GET",
                  "HTTP GET",
                  "HTTP GET",
                  "HTTP GET",
                  "HTTP GET"
                ],
                [
                  {
                    "durationInNanos": 757000000,
                    "endTime": "2024-07-28T15:32:14.575Z",
                    "statusCode": 0
                  },
                  {
                    "durationInNanos": 757000000,
                    "endTime": "2024-07-28T15:32:14.575Z",
                    "statusCode": 0
                  },
                  {
                    "durationInNanos": 757000000,
                    "endTime": "2024-07-28T15:32:14.575Z",
                    "statusCode": 0
                  },
                  {
                    "durationInNanos": 757000000,
                    "endTime": "2024-07-28T15:32:14.575Z",
                    "statusCode": 0
                  },
                  {
                    "durationInNanos": 757000000,
                    "endTime": "2024-07-28T15:32:14.575Z",
                    "statusCode": 0
                  },
                  {
                    "durationInNanos": 757000000,
                    "endTime": "2024-07-28T15:32:14.575Z",
                    "statusCode": 0
                  },
                  {
                    "durationInNanos": 757000000,
                    "endTime": "2024-07-28T15:32:14.575Z",
                    "statusCode": 0
                  },
                  {
                    "durationInNanos": 757000000,
                    "endTime": "2024-07-28T15:32:14.575Z",
                    "statusCode": 0
                  },
                  {
                    "durationInNanos": 757000000,
                    "endTime": "2024-07-28T15:32:14.575Z",
                    "statusCode": 0
                  }
                ],
                [
                  "0272bba222d1ba0d2c0897e55d9c71f6",
                  "0272bba222d1ba0d2c0897e55d9c71f6",
                  "0272bba222d1ba0d2c0897e55d9c71f6",
                  "0272bba222d1ba0d2c0897e55d9c71f6",
                  "0272bba222d1ba0d2c0897e55d9c71f6",
                  "0272bba222d1ba0d2c0897e55d9c71f6",
                  "0272bba222d1ba0d2c0897e55d9c71f6",
                  "0272bba222d1ba0d2c0897e55d9c71f6",
                  "0272bba222d1ba0d2c0897e55d9c71f6"
                ],
                [
                  "",
                  "",
                  "",
                  "",
                  "",
                  "",
                  "",
                  "",
                  ""
                ]
              ]
            }
          },
          {
            "schema": {
              "name": "edges",
              "refId": "A",
              "meta": {
                "typeVersion": [
                  0,
                  0
                ],
                "preferredVisualisationType": "nodeGraph"
              },
              "fields": [
                {
                  "name": "id",
                  "type": "string",
                  "typeInfo": {
                    "frame": "string"
                  }
                },
                {
                  "name": "source",
                  "type": "string",
                  "typeInfo": {
                    "frame": "string"
                  }
                },
                {
                  "name": "target",
                  "type": "string",
                  "typeInfo": {
                    "frame": "string"
                  }
                },
                {
                  "name": "detail__operation",
                  "type": "string",
                  "typeInfo": {
                    "frame": "string"
                  },
                  "config": {
                    "displayName": "Operation(s)"
                  }
                }
              ]
            },
            "data": {
              "values": [
                [
                  "frontendproxy_frontend",
                  "frontend_cartservice",
                  "frontend-web_frontendproxy",
                  "loadgenerator_frontendproxy",
                  "checkoutservice_cartservice"
                ],
                [
                  "frontendproxy",
                  "frontend",
                  "frontend-web",
                  "loadgenerator",
                  "checkoutservice"
                ],
                [
                  "frontend",
                  "cartservice",
                  "frontendproxy",
                  "frontendproxy",
                  "cartservice"
                ],
                [
                  "GET,POST",
                  "POST /oteldemo.CartService/GetCart,POST /oteldemo.CartService/AddItem",
                  "ingress",
                  "ingress",
                  "POST /oteldemo.CartService/EmptyCart,POST /oteldemo.CartService/GetCart"
                ]
              ]
            }
          },
          {
            "schema": {
              "name": "nodes",
              "refId": "A",
              "meta": {
                "typeVersion": [
                  0,
                  0
                ],
                "preferredVisualisationType": "nodeGraph"
              },
              "fields": [
                {
                  "name": "id",
                  "type": "string",
                  "typeInfo": {
                    "frame": "string"
                  }
                },
                {
                  "name": "title",
                  "type": "string",
                  "typeInfo": {
                    "frame": "string"
                  },
                  "config": {
                    "displayName": "Service name"
                  }
                },
                {
                  "name": "arc__errors",
                  "type": "number",
                  "typeInfo": {
                    "frame": "float64"
                  },
                  "config": {
                    "displayName": "Error rate",
                    "color": {
                      "fixedColor": "red",
                      "mode": "fixed"
                    }
                  }
                },
                {
                  "name": "detail__errors",
                  "type": "number",
                  "typeInfo": {
                    "frame": "float64"
                  },
                  "config": {
                    "displayName": "Error rate",
                    "unit": "%"
                  }
                },
                {
                  "name": "arc__success",
                  "type": "number",
                  "typeInfo": {
                    "frame": "float64"
                  },
                  "config": {
                    "displayName": "Success rate",
                    "color": {
                      "fixedColor": "green",
                      "mode": "fixed"
                    }
                  }
                },
                {
                  "name": "mainstat",
                  "type": "number",
                  "typeInfo": {
                    "frame": "float64"
                  },
                  "config": {
                    "displayName": "Avg. Latency",
                    "unit": "ms"
                  }
                }
              ]
            },
            "data": {
              "values": [
                [
                  "frontendproxy",
                  "frontend",
                  "frontend-web",
                  "cartservice"
                ],
                [
                  "frontendproxy",
                  "frontend",
                  "frontend-web",
                  "cartservice"
                ],
                [
                  0,
                  0,
                  0,
                  0
                ],
                [
                  0,
                  0,
                  0,
                  0
                ],
                [
                  1,
                  1,
                  1,
                  1
                ],
                [
                  30.947,
                  30.906287,
                  757,
                  3.467
                ]
              ]
            }
          }
        ],
        "refId": "A"
      }
    }
  }
}

image

Stack Trace

An unexpected error happened
Details
TypeError: Cannot read properties of undefined (reading 'nodeRadius')

    at tt (https://grafana-assets.grafana.net/grafana/11.2.0-73451/public/build/5511.5fe23ce61d3edd96362c.js:104:36)
    at div
    at div
    at section
    at O (https://grafana-assets.grafana.net/grafana/11.2.0-73451/public/build/6311.8c8758f3a48b051a272e.js:83:31338)
    at ja (https://grafana-assets.grafana.net/grafana/11.2.0-73451/public/build/explore.51af08cbb249cf3db47d.js:191:39)
    at to (https://grafana-assets.grafana.net/grafana/11.2.0-73451/public/build/4307.2967209eee3feb5b5d14.js:1371:122261)
    at div
    at Ce (https://grafana-assets.grafana.net/grafana/11.2.0-73451/public/build/explore.51af08cbb249cf3db47d.js:9:11066)
    at g (https://grafana-assets.grafana.net/grafana/11.2.0-73451/public/build/6311.8c8758f3a48b051a272e.js:63:133930)
    at o (https://grafana-assets.grafana.net/grafana/11.2.0-73451/public/build/6311.8c8758f3a48b051a272e.js:63:134439)
    at g (https://grafana-assets.grafana.net/grafana/11.2.0-73451/public/build/6311.8c8758f3a48b051a272e.js:63:133930)
    at o (https://grafana-assets.grafana.net/grafana/11.2.0-73451/public/build/6311.8c8758f3a48b051a272e.js:63:134439)
    at main
    at div
    at f (https://grafana-assets.grafana.net/grafana/11.2.0-73451/public/build/4307.2967209eee3feb5b5d14.js:1028:100453)
    at div
    at div
    at div
    at J (https://grafana-assets.grafana.net/grafana/11.2.0-73451/public/build/4307.2967209eee3feb5b5d14.js:880:18697)
    at u (https://grafana-assets.grafana.net/grafana/11.2.0-73451/public/build/6311.8c8758f3a48b051a272e.js:63:33933)
    at div
    at div
    at Uo (https://grafana-assets.grafana.net/grafana/11.2.0-73451/public/build/explore.51af08cbb249cf3db47d.js:9:925)
    at tl (https://grafana-assets.grafana.net/grafana/11.2.0-73451/public/build/explore.51af08cbb249cf3db47d.js:212:8511)
    at to (https://grafana-assets.grafana.net/grafana/11.2.0-73451/public/build/4307.2967209eee3feb5b5d14.js:1371:122261)
    at WithTheme(Connect(tl))
    at div
    at div
    at div
    at J (https://grafana-assets.grafana.net/grafana/11.2.0-73451/public/build/4307.2967209eee3feb5b5d14.js:880:18697)
    at u (https://grafana-assets.grafana.net/grafana/11.2.0-73451/public/build/6311.8c8758f3a48b051a272e.js:63:33933)
    at yl (https://grafana-assets.grafana.net/grafana/11.2.0-73451/public/build/explore.51af08cbb249cf3db47d.js:212:20645)
    at to (https://grafana-assets.grafana.net/grafana/11.2.0-73451/public/build/4307.2967209eee3feb5b5d14.js:1371:122261)
    at g (https://grafana-assets.grafana.net/grafana/11.2.0-73451/public/build/6311.8c8758f3a48b051a272e.js:63:133930)
    at o (https://grafana-assets.grafana.net/grafana/11.2.0-73451/public/build/6311.8c8758f3a48b051a272e.js:63:134439)
    at div
    at o (https://grafana-assets.grafana.net/grafana/11.2.0-73451/public/build/660.33765798357bf190dd71.js:1:2310)
    at div
    at o (https://grafana-assets.grafana.net/grafana/11.2.0-73451/public/build/660.33765798357bf190dd71.js:1:4298)
    at G (https://grafana-assets.grafana.net/grafana/11.2.0-73451/public/build/660.33765798357bf190dd71.js:1:9639)
    at div
    at jc (https://grafana-assets.grafana.net/grafana/11.2.0-73451/public/build/explore.51af08cbb249cf3db47d.js:402:8842)
    at xr (https://grafana-assets.grafana.net/grafana/11.2.0-73451/public/build/explore.51af08cbb249cf3db47d.js:11:6061)
    at Cc
    at Suspense
    at g (https://grafana-assets.grafana.net/grafana/11.2.0-73451/public/build/6311.8c8758f3a48b051a272e.js:63:133930)
    at Ba (https://grafana-assets.grafana.net/grafana/11.2.0-73451/public/build/6311.8c8758f3a48b051a272e.js:184:12305)
    at V (https://grafana-assets.grafana.net/grafana/11.2.0-73451/public/build/4307.2967209eee3feb5b5d14.js:1018:29174)
    at lo (https://grafana-assets.grafana.net/grafana/11.2.0-73451/public/build/4307.2967209eee3feb5b5d14.js:1009:4452)
    at gs (https://grafana-assets.grafana.net/grafana/11.2.0-73451/public/build/4307.2967209eee3feb5b5d14.js:1009:12001)
    at Ut (https://grafana-assets.grafana.net/grafana/11.2.0-73451/public/build/4307.2967209eee3feb5b5d14.js:1018:17388)
    at V (https://grafana-assets.grafana.net/grafana/11.2.0-73451/public/build/4307.2967209eee3feb5b5d14.js:1018:31352)
    at div
    at https://grafana-assets.grafana.net/grafana/11.2.0-73451/public/build/6311.8c8758f3a48b051a272e.js:82:52942
    at main
    at div
    at div
    at div
    at Cs (https://grafana-assets.grafana.net/grafana/11.2.0-73451/public/build/6311.8c8758f3a48b051a272e.js:184:2406)
    at div
    at Di (https://grafana-assets.grafana.net/grafana/11.2.0-73451/public/build/6311.8c8758f3a48b051a272e.js:184:7580)
    at lo (https://grafana-assets.grafana.net/grafana/11.2.0-73451/public/build/4307.2967209eee3feb5b5d14.js:1009:4452)
    at gs (https://grafana-assets.grafana.net/grafana/11.2.0-73451/public/build/4307.2967209eee3feb5b5d14.js:1009:12001)
    at ss (https://grafana-assets.grafana.net/grafana/11.2.0-73451/public/build/4307.2967209eee3feb5b5d14.js:1009:11337)
    at yr (https://grafana-assets.grafana.net/grafana/11.2.0-73451/public/build/4307.2967209eee3feb5b5d14.js:1018:17687)
    at V (https://grafana-assets.grafana.net/grafana/11.2.0-73451/public/build/4307.2967209eee3feb5b5d14.js:1018:25388)
    at v (https://grafana-assets.grafana.net/grafana/11.2.0-73451/public/build/4307.2967209eee3feb5b5d14.js:164:8978)
    at b (https://grafana-assets.grafana.net/grafana/11.2.0-73451/public/build/4307.2967209eee3feb5b5d14.js:1371:112899)
    at c (https://grafana-assets.grafana.net/grafana/11.2.0-73451/public/build/6311.8c8758f3a48b051a272e.js:429:27883)
    at g (https://grafana-assets.grafana.net/grafana/11.2.0-73451/public/build/6311.8c8758f3a48b051a272e.js:63:133930)
    at o (https://grafana-assets.grafana.net/grafana/11.2.0-73451/public/build/6311.8c8758f3a48b051a272e.js:63:134439)
    at lt (https://grafana-assets.grafana.net/grafana/11.2.0-73451/public/build/4307.2967209eee3feb5b5d14.js:1375:508)
    at qi (https://grafana-assets.grafana.net/grafana/11.2.0-73451/public/build/6311.8c8758f3a48b051a272e.js:184:14420)

@kevinwcyu kevinwcyu moved this from Waiting to Incoming in AWS Datasources Jul 29, 2024
@iwysiu iwysiu mentioned this issue Aug 1, 2024
Merged
@github-project-automation github-project-automation bot moved this from Incoming to Done in AWS Datasources Aug 2, 2024
@rkarthikr
Copy link
Author

Which release this fix will be available in ? @kevinwcyu @iwysiu

@iwysiu
Copy link
Contributor

iwysiu commented Aug 6, 2024

Hi @rkarthikr , I just released a new version with the fix!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
datasource/OpenSearch effort/small type/bug Something isn't working
Projects
AWS Datasources
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix serviceMap when source node doesn't have stats grafana/opensearch-datasource
3 participants
@iwysiu @kevinwcyu @rkarthikr