OTel Agent Feature Support #1559

mackjmr · 2024-12-11T16:55:45Z

What does this PR do?

Add support for Otel Agent as a Feature
Remove support for OTel Agent as a flag
Maintains support for OTel Agent as an annotation temporarily

Motivation

OTEL-2290

Additional Notes

Anything else we should know when reviewing?

Minimum Agent Versions

Are there minimum versions of the Datadog Agent and/or Cluster Agent required?

Agent: vX.Y.Z
Cluster Agent: vX.Y.Z

Describe your test plan

This PR adds support for deploying the otel-agent in a new manner. In order to confirm these changes work as expected, we need to validate that the otel agent runs without error, but also that it is able to receive data from OTel clients and forward it to Datadog.

The following docker image: mackjmr/app-all:v2 contains an app that sends OTLP traces every 5 seconds, which we can use to send data to the otel agent.

This app can be deployed using the following deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-all
  labels:
    app: app-all
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app-all
  template:
    metadata:
      labels:
        app: app-all
    spec:
      containers:
      - name: app-all
        image: mackjmr/app-all:v2
        env:
        - name: MY_NODE_IP
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
          value: http://$(MY_NODE_IP):4317
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        - name: OTEL_RESOURCE_ATTRIBUTES
          value: "k8s.pod.ip=$(POD_IP),service.name=test-operator-changes"

Noteworthy:

OTEL_EXPORTER_OTLP_ENDPOINT is sent to send traces to the NODE_IP:4317 (otel agent listens on 4317 and binds container port to host port)
The service name is set via env var OTEL_RESOURCE_ATTRIBUTES.

Known issues when testing:
Agent will fail to startup in some kubernetes environments (e.g. Kind cluster) if it fails to detect the hostname. You can prevent this from happening by setting the hostname yourself:

spec:
  override:
    nodeAgent:
      containers: 
        agent:
          env:
            - name: DD_HOSTNAME
              value: "test.node.name"

In Kind/ Minikube, you also need to disable TLS Verify:

  global:
    kubelet:
      tlsVerify: false

Scenarios to test:

1. Otel Agent Feature with collector config in configData:

Test 1: Default ports

Deploy the agent using the following manifest: https://github.com/DataDog/datadog-operator/blob/1a69194e7c958641288a3f0489372acb2cecfa34/examples/datadogagent/datadog-agent-with-otel-agent.yaml.

Ensure that the otel-agent starts up without issues/ errors.
Once above has been validated, deploy the app-all image mentionned above, and ensure that traces are making it into Datadog

Test 2: Non default ports:

Same steps as above, but in manifest add ports sections:

  features:
    otelCollector:
      enabled: true
      ports:
        - containerPort: 3333
          name: otel-grpc

and change collector config port:

    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:3333

and in app-all deployment change OTEL_EXPORTER_OTLP_ENDPOINT to http://$(MY_NODE_IP):3333.

2. Otel Agent Feature with collector config in configMap:

Deploy the agent using the following manifest: https://github.com/DataDog/datadog-operator/blob/1a69194e7c958641288a3f0489372acb2cecfa34/examples/datadogagent/datadog-agent-with-otel-agent-configmap.yaml.

Ensure that the otel-agent starts up without issues/ errors.
Once above has been validated, deploy the app-all image mentionned above, and ensure that traces are making it into Datadog

3. Otel Agent Feature without passing collector config:

When we don't pass a collector config, a default config is used.

Test 1: Default ports

Deploy the agent using the following:

apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
spec:
  global:
    credentials:
      apiKey: <DATADOG_API_KEY>
  features:
    otelCollector:
      enabled: true

Ensure that the otel-agent starts up without issues/ errors.
Once above has been validated, deploy the app-all image mentionned above, and ensure that traces are making it into Datadog

Test 2: Non default ports:

Deploy the agent using the following:

apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
spec:
  global:
    credentials:
      apiKey: <DATADOG_API_KEY>
  features:
    otelCollector:
      enabled: true
      ports:
        - containerPort: 3333
          name: otel-grpc

Ensure that the otel-agent starts up without issues/ errors.
Once above has been validated, deploy the app-all image mentionned above changing OTEL_EXPORTER_OTLP_ENDPOINT to http://$(MY_NODE_IP):3333, and ensure that traces are making it into Datadog

4. Otel Agent via Annotation:

We need to validate that deploying otel agent with annotations still works.

Deploy the agent using the following manifest: https://github.com/DataDog/datadog-operator/blob/1a69194e7c958641288a3f0489372acb2cecfa34/examples/datadogagent/datadog-agent-with-otel-agent-annotations.yaml.

Ensure that the otel-agent starts up without issues/ errors.
Once above has been validated, deploy the app-all image mentionned above, and ensure that traces are making it into Datadog

5. Flare:

Test 1: Core config disabled

Disable the core config:

  features:
    otelCollector:
      coreConfig: 
        enabled: false

Trigger a flare (you can check the flare locally, or if you want to submit it, you can use ticket #1970459), and ensure that in otel/otel-agent.log you see log:

'otelcollector.enabled' is disabled in the configuration

Test 2: Core config enabled

The otelcollector.enabled should be enabled by default when the otel collector feature is enabled, so you can leave the coreConfig empty (this config: https://github.com/DataDog/datadog-operator/blob/1a69194e7c958641288a3f0489372acb2cecfa34/examples/datadogagent/datadog-agent-with-otel-agent.yaml will do). Trigger a flare (you can check the flare locally, or if you want to submit it, you can use ticket #1970459), and ensure that the otel directory contains the following data:

Test 3: Core config other params

Add the following params:

  features:
    otelCollector:
      coreConfig: 
        enabled: true
        extension_timeout: 13
        extension_url: "https://localhost:7777"

describe the pod (kubectl describe pod -n <namespace> <pod_name>), and ensure the following env vars are set:

      DD_OTELCOLLECTOR_ENABLED:                              true
      DD_OTELCOLLECTOR_EXTENSION_TIMEOUT:                    13
      DD_OTELCOLLECTOR_EXTENSION_URL:                        https://localhost:7777

Checklist

PR has at least one valid label: bug, enhancement, refactoring, documentation, tooling, and/or dependencies
PR has a milestone or the qa/skip-qa label

levan-m · 2024-12-13T01:37:29Z

internal/controller/datadogagent/component/agent/default.go

 func volumeMountsForInitConfig() []corev1.VolumeMount {
 	return []corev1.VolumeMount{
-		common.GetVolumeMountForLogs(),


Why delete initContainer configs, is this even in otel agent code path?

This was a mistake, meant to remove for otel agent voluments. undid the change

levan-m · 2024-12-13T01:42:19Z

api/datadoghq/v2alpha1/datadogagent_types.go

@@ -44,6 +44,8 @@ type DatadogAgentSpec struct {
 type DatadogFeatures struct {
 	// Application-level features

+	// OTelAgent configuration.
+	OTelAgent *OTelAgentFeatureConfig `json:"otelAgent,omitempty"`


is otelAgent really best name for this feature? should this be otelCollector or just otel? other features don't mention 'Agent'.

Renamed feature from otelAgent to otelCollector.

codecov-commenter · 2024-12-13T11:45:57Z

Codecov Report

Attention: Patch coverage is 81.65138% with 40 lines in your changes missing coverage. Please review.

Project coverage is 48.96%. Comparing base (f517775) to head (137220d).

Files with missing lines	Patch %	Lines
...controller/datadogagent/component/agent/default.go	0.00%	17 Missing ⚠️
...ller/datadogagent/feature/otelcollector/feature.go	88.81%	13 Missing and 3 partials ⚠️
api/datadoghq/v2alpha1/test/builder.go	88.23%	4 Missing and 2 partials ⚠️
...ller/datadogagent/feature/enabledefault/feature.go	0.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1559      +/-   ##
==========================================
+ Coverage   48.56%   48.96%   +0.39%     
==========================================
  Files         226      227       +1     
  Lines       20435    20626     +191     
==========================================
+ Hits         9925    10100     +175     
- Misses       9989    10001      +12     
- Partials      521      525       +4

Flag	Coverage Δ
unittests	`48.96% <81.65%> (+0.39%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
api/datadoghq/v2alpha1/datadogagent_default.go	`91.10% <100.00%> (+0.14%)`	⬆️
api/datadoghq/v2alpha1/datadogagent_types.go	`100.00% <ø> (ø)`
cmd/main.go	`0.00% <ø> (ø)`
internal/controller/datadogagent/controller.go	`51.85% <ø> (-1.72%)`	⬇️
internal/controller/datadogagent/feature/types.go	`26.92% <ø> (ø)`
internal/controller/setup.go	`51.49% <ø> (-0.36%)`	⬇️
pkg/defaulting/images.go	`100.00% <ø> (ø)`
...ller/datadogagent/feature/enabledefault/feature.go	`36.62% <0.00%> (+0.14%)`	⬆️
api/datadoghq/v2alpha1/test/builder.go	`91.52% <88.23%> (-0.29%)`	⬇️
...ller/datadogagent/feature/otelcollector/feature.go	`88.81% <88.81%> (ø)`
... and 1 more

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f517775...137220d. Read the comment docs.

… instance to replace

…ations example with remove note

api/datadoghq/v2alpha1/datadogagent_types.go

api/datadoghq/v2alpha1/test/builder.go

cmd/main.go

config/crd/bases/v1/datadoghq.com_datadogagents.yaml

levan-m · 2024-12-16T19:19:26Z

internal/controller/datadogagent/component/agent/default.go

@@ -104,6 +104,11 @@ func agentImage() string {
 	return fmt.Sprintf("%s/%s:%s", v2alpha1.DefaultImageRegistry, v2alpha1.DefaultAgentImageName, defaulting.AgentLatestVersion)
 }

+func otelAgentImage() string {
+	// todo(mackjmr): make this dynamic once we have a non-dev image.
+	return "datadog/agent-dev:nightly-ot-beta-main"


What's the ETA of dev image? Will this todo be resolved in this PR?

Will this todo be resolved in this PR?

No, and defering to @dineshg13 for ETA of non-dev image.

can this be moved with the rest of images constants

datadog-operator/pkg/defaulting/images.go

Lines 25 to 37 in 756ed08

GCRContainerRegistry ContainerRegistry = "gcr.io/datadoghq"

// DockerHubContainerRegistry corresponds to the datadoghq docker.io registry

DockerHubContainerRegistry ContainerRegistry = "docker.io/datadog"

// PublicECSContainerRegistry corresponds to the datadoghq PublicECSContainerRegistry registry

PublicECSContainerRegistry ContainerRegistry = "public.ecr.aws/datadog"

// DefaultImageRegistry corresponds to the datadoghq containers registry

DefaultImageRegistry = GCRContainerRegistry // TODO: this is also defined elsewhere and not used; consolidate

// JMXTagSuffix prefix tag for agent JMX images

JMXTagSuffix = "-jmx"

agentImageName = "agent"

clusterAgentImageName = "cluster-agent"

)

Still discussing with Dinesh what image to use, I'll keep this in mind when I update it.

@levan-m moved to images.go in d1345c2

levan-m · 2024-12-16T21:31:42Z

internal/controller/datadogagent/feature/otelcollector/feature.go

+			},
+		}
+	}
+	o.ports = dda.Spec.Features.OtelCollector.Ports


Why not assign the configured ports' structure directly to o.ports? As a general practice, we don't modify DDA withing the features, instead save relevant config in feature struct otelCollectorFeature.

Makes sense, updated in: ff7868e.

levan-m

Reviewed latest commits and looks good to me. Please verify the test cluster state isn't caused by this change before merging.

wspurgin · 2025-01-03T14:52:32Z

How would one enable OTEL prior to this PR if using the operator? I see that there's a command line flag, but it's unclear from the docs how to use those in a datadog manifest.

Relatedly, is there any target release planned that will include this feature?

levan-m · 2025-01-03T15:16:16Z

Hi @wspurgin, command line flag was added to enable internal testing using overrides section in the DatadogAgent CRD. This PR delivers feature support and will be released with v1.12, ETA early February.

wspurgin · 2025-01-03T15:40:26Z

Thanks for the response @levan-m - so am I right in my understanding that there is not any way a user of the DD operator helm chart could enable the "beta" OTeL flag themselves?

Is there any concern with allowing us to opt in for the time being (for users of the operator from versions v1.8 -> v1.11.1)? Otherwise I'll just have to drop the operator entirely and configure the agents in Helm directly (the operator is the recommended way, so I'd prefer obviously not to undo my last week of work to do the non-recommended approach 😓)

levan-m · 2025-01-03T19:28:25Z

@wspurgin, there are ways to enabled it (render helm chart and edit manifest or use annotation added here #1475) however main challenge will be correctly configuring the OTEL agent.

I'd recommend waiting till next week for 1.12 release candidate and use the feature added with this PR. For the RC setup you will need to install CRDs manually as helm chart isn't getting updated until final release.

wspurgin · 2025-01-03T19:57:30Z

Thanks @levan-m - I'll just wait for that RC. I tried the annotation (but the agent would crash with an error saying otel-agent didn't exist) - like you said I'm probably missing some config for that to work properly. I'll be on the lookout for that RC once it's out! Thanks again 👍

This PR removes support for enabling otel-agent via annotations in favor of enabling otel agent vie otelCollector feature: #1559. The annotation use has already been removed in staging: DataDog/k8s-datadog-agent-ops#4602.

mackjmr added 2 commits December 11, 2024 17:53

OTel Agent Feature Support

94b0882

update test

c20dc8c

mackjmr added the enhancement New feature or request label Dec 12, 2024

mackjmr added 5 commits December 12, 2024 14:47

add port documentation

cb51682

remove flag support

b546ad5

add examples

d5a8cdf

Merge branch 'main' into mackjmr/otel-agent-feature-support

5cfa20b

fix

cc036f7

mackjmr added this to the v1.11.0 milestone Dec 12, 2024

mackjmr added 3 commits December 12, 2024 16:41

fix examples

7a522c5

fix

fe61d96

fix test

cf17884

levan-m reviewed Dec 13, 2024

View reviewed changes

mackjmr added 10 commits December 13, 2024 10:59

rename feature from otelAgent to otelCollector

2a962e8

rename

06a9f7d

rename directory/ pkg

719a485

undo change to InitConfig volumes

2945214

remove unrelated changes

85e150b

add empty line

838d361

Merge branch 'main' into mackjmr/otel-agent-feature-support

b0b5a46

add otel collector feature to Test_defaultFeatures

d7aaf6d

add otel agent feature to factory test

d8f55e2

add annotation enabled/ feature disabled test

3d59fac

mackjmr added 3 commits December 13, 2024 14:22

Use Replace with i=1 instead of ReplaceAll as we know there is only 1…

fc532dd

… instance to replace

fix examples

7f50ee1

update example ports to 0.0.0.0 (defaults to localhost) and add annot…

1a69194

…ations example with remove note

mackjmr marked this pull request as ready for review December 13, 2024 14:47

mackjmr requested review from a team as code owners December 13, 2024 14:47

aliciascott approved these changes Dec 13, 2024

View reviewed changes

dineshg13 reviewed Dec 16, 2024

View reviewed changes

api/datadoghq/v2alpha1/datadogagent_types.go Show resolved Hide resolved

api/datadoghq/v2alpha1/test/builder.go Show resolved Hide resolved

cmd/main.go Show resolved Hide resolved

config/crd/bases/v1/datadoghq.com_datadogagents.yaml Show resolved Hide resolved

levan-m modified the milestones: v1.11.0, v1.12.0 Dec 16, 2024

levan-m reviewed Dec 16, 2024

View reviewed changes

mackjmr and others added 9 commits December 17, 2024 12:53

update o.ports rather than DDA directly

ff7868e

add support for core config

1175a45

change env var path

d12e4e0

Move image to images.go per feedback

d1345c2

Merge branch 'main' into mackjmr/otel-agent-feature-support

1f75457

fix test after merge of main

4a490b7

add non default auth path volume mount, required for staging

eb7f68e

add vm logs

7d61660

Merge branch 'main' into mackjmr/otel-agent-feature-support

137220d

levan-m approved these changes Dec 19, 2024

View reviewed changes

change sync delay to 30s

a4f0b7c

mackjmr merged commit e2f08ba into main Dec 20, 2024
19 checks passed

mackjmr deleted the mackjmr/otel-agent-feature-support branch December 20, 2024 17:29

wspurgin mentioned this pull request Jan 3, 2025

datadog-opeartor No mechanism exists to enable OTeL Collector for operator versions ~v1.8.0<=x<=v1.11.1 DataDog/helm-charts#1657

Open

swang392 mentioned this pull request Jan 7, 2025

Fix API violation from OTel Agent #1607

Merged

2 tasks

mackjmr mentioned this pull request Jan 14, 2025

Remove support for otel annotation #1624

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OTel Agent Feature Support #1559

OTel Agent Feature Support #1559

mackjmr commented Dec 11, 2024 •

edited

Loading

levan-m Dec 13, 2024

mackjmr Dec 13, 2024

levan-m Dec 13, 2024

mackjmr Dec 13, 2024

codecov-commenter commented Dec 13, 2024 •

edited

Loading

levan-m Dec 16, 2024

mackjmr Dec 17, 2024 •

edited

Loading

levan-m Dec 17, 2024

mackjmr Dec 18, 2024

mackjmr Dec 19, 2024

levan-m Dec 16, 2024

mackjmr Dec 17, 2024

levan-m left a comment

wspurgin commented Jan 3, 2025

levan-m commented Jan 3, 2025

wspurgin commented Jan 3, 2025

levan-m commented Jan 3, 2025

wspurgin commented Jan 3, 2025 •

edited

Loading

	GCRContainerRegistry ContainerRegistry = "gcr.io/datadoghq"
	// DockerHubContainerRegistry corresponds to the datadoghq docker.io registry
	DockerHubContainerRegistry ContainerRegistry = "docker.io/datadog"
	// PublicECSContainerRegistry corresponds to the datadoghq PublicECSContainerRegistry registry
	PublicECSContainerRegistry ContainerRegistry = "public.ecr.aws/datadog"
	// DefaultImageRegistry corresponds to the datadoghq containers registry
	DefaultImageRegistry = GCRContainerRegistry // TODO: this is also defined elsewhere and not used; consolidate
	// JMXTagSuffix prefix tag for agent JMX images
	JMXTagSuffix = "-jmx"

	agentImageName = "agent"
	clusterAgentImageName = "cluster-agent"
	)

OTel Agent Feature Support #1559

OTel Agent Feature Support #1559

Conversation

mackjmr commented Dec 11, 2024 • edited Loading

What does this PR do?

Motivation

Additional Notes

Minimum Agent Versions

Describe your test plan

Scenarios to test:

1. Otel Agent Feature with collector config in configData:

2. Otel Agent Feature with collector config in configMap:

3. Otel Agent Feature without passing collector config:

4. Otel Agent via Annotation:

5. Flare:

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-commenter commented Dec 13, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

mackjmr Dec 17, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

levan-m left a comment

Choose a reason for hiding this comment

wspurgin commented Jan 3, 2025

levan-m commented Jan 3, 2025

wspurgin commented Jan 3, 2025

levan-m commented Jan 3, 2025

wspurgin commented Jan 3, 2025 • edited Loading

mackjmr commented Dec 11, 2024 •

edited

Loading

codecov-commenter commented Dec 13, 2024 •

edited

Loading

mackjmr Dec 17, 2024 •

edited

Loading

wspurgin commented Jan 3, 2025 •

edited

Loading