[opentelemetry] Add OTLP intake E2E system tests #976

songy23 · 2023-03-16T20:45:29Z

Description

Add basic E2E integration tests on OpenTelemetry Protocol (OTLP) intake endpoint.

JIRA tickets: OTEL-338 and OTEL-401.
Usage: ./build.sh java_otel && ./run.sh OTEL_TRACING_E2E

What's covered

Add java_otel application container: a Spring boot native HTTP server that implements weblog interfaces and instrumented with OTel-Java SDK + Docker file. For basic tests (GET /) it generates 1 fake publisher span and 1 server span that has a link to the publisher span.
Add a test runner that sends requests to the application container. After receiving the response, it retrieves traces from DD backend and validates them. For now it only works with java_otel.
OTel spans generated by the test runner and application container are submitted to DD via the DD OTLP intake endpoint.

What's next

Add agent ingestion and OTel Collector to OTel tests
Verify OTel generated 128 bits trace ID

Workflow

⚠️⚠️ Create your PR as draft (we're receiving lot of PR, it saves us lot of time) ⚠️⚠️
Follows the style guidelines of this project (See how to easily lint the code)
Work on you PR until the CI passes (if something not related to your task is failing, you can ignore it)
Mark it as ready for review

NOTE: By default in PR only default scenario tests will be launched. Please refer to the documentation to run all scenarios in your PR if needed.

Once your PR is reviewed, you can merge it ! ❤️

songy23 · 2023-03-17T20:11:48Z

/cc @lambrospetrou

tests/otel_tracing_e2e/test_e2e.py

utils/_context/_scenarios.py

lambrospetrou · 2023-03-20T14:37:43Z

utils/_context/_scenarios.py

+    # OpenTelemetry tracing end-to-end scenarios
+    otel_tracing_e2e_w3c = EndToEndScenario(
+        "OTEL_TRACING_E2E_W3C",
+        weblog_env={"DD_API_KEY": os.environ.get("DD_API_KEY"), "DD_SITE": os.environ.get("DD_SITE"),},


You don't need these, since they are already set.

It's actually needed - we set dd api key and dd site for Agent containers but not Weblog containers.

cbeauchesne · 2023-03-22T18:24:02Z

FI, seen with @songy23 : this PR will introduce lot of internal change, let just wait for the another big PR to be closed (#958)

- Associate OTel trace id with request id - Remove client side instrumentation - Remove trace context propagation - Support OTLP intake in proxy

songy23

~~Holding off this PR to wait for #958.~~

tests/otel_tracing_e2e/test_e2e.py

utils/_context/_scenarios.py

songy23 · 2023-03-27T13:58:02Z

utils/_context/_scenarios.py

+    # OpenTelemetry tracing end-to-end scenarios
+    otel_tracing_e2e_w3c = EndToEndScenario(
+        "OTEL_TRACING_E2E_W3C",
+        weblog_env={"DD_API_KEY": os.environ.get("DD_API_KEY"), "DD_SITE": os.environ.get("DD_SITE"),},


It's actually needed - we set dd api key and dd site for Agent containers but not Weblog containers.

songy23 · 2023-04-14T16:21:28Z

Chatted with @cbeauchesne and we agreed to submit this PR first, so posted for review again and removed do-not-merge

tests/otel_tracing_e2e/_validator.py

...java_otel/spring-boot-native/src/main/java/com/datadoghq/springbootnative/WebController.java

utils/build/docker/java_otel/spring-boot-native/pom.xml

...ld/docker/java_otel/spring-boot-native/src/main/java/com/datadoghq/springbootnative/App.java

cbeauchesne · 2023-04-20T17:03:08Z

@songy23 the adjustments needed on interfaces/scenarios are done.

It does not work because the weblog crashes with this issue :

Exception in thread "main" java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:95)
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
	at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65)
Caused by: java.lang.NullPointerException: Parameter specified as non-null is null: method okhttp3.Headers$Builder.add, parameter value
	at okhttp3.Headers$Builder.add(Headers.kt)
	at io.opentelemetry.exporter.internal.okhttp.OkHttpExporterBuilder.addHeader(OkHttpExporterBuilder.java:85)
	at io.opentelemetry.exporter.otlp.http.trace.OtlpHttpSpanExporterBuilder.addHeader(OtlpHttpSpanExporterBuilder.java:80)
	at com.datadoghq.springbootnative.App.main(App.java:30)
	... 8 more

songy23 · 2023-04-20T17:57:29Z

I got a different error with the latest changes:

Failed to export spans. The request could not be executed. Full error message: runner: Name or service not known

Looking

songy23 · 2023-04-21T17:20:17Z

@cbeauchesne I was able to run the otel test with my latest changes:

% ./run.sh OTEL_TRACING_E2E
============================================== test context ===============================================
Scenario: OTEL_TRACING_E2E
Logs folder: ./logs_otel_tracing_e2e
=========================================== test session starts ===========================================
collected 349 items / 348 deselected / 1 selected                                                         
Executing weblog warmup...
----------------------------------------------- tests setup -----------------------------------------------

tests/otel_tracing_e2e/test_e2e.py .

-------------------------------------- Wait for agent interface (5s) --------------------------------------
---------------------------------- Wait for Weblog stdout interface (0s) ----------------------------------
---------------------------- Wait for .Net tracer-managed logs interface (0s) -----------------------------

tests/otel_tracing_e2e/test_e2e.py .                                                                [100%]

----- generated xml file: /Users/yang.song/Github/system-tests/logs_otel_tracing_e2e/reportJunit.xml ------
==================================== 1 passed, 348 deselected in 9.05s ====================================

Could you give it another try when you get a chance? (Note you need to set DD_API_KEY and DD_APPLICATION_KEY because this is an E2E test)

cbeauchesne · 2023-04-24T08:46:13Z

strange, I have the issue again. I've added the job in the CI, we'll see if it's only my local setup or not.

songy23 · 2023-04-24T16:43:57Z

NVM this should be fixed in c143cd4

~~CI still failed (https://github.com/DataDog/system-tests/actions/runs/4788383870/jobs/8514919306?pr=976) because of~~

ocker.errors.ImageNotFound: 404 Client Error for http+docker://localhost/v1.41/images/create?tag=latest&fromImage=system_tests%2Fweblog: Not Found ("pull access denied for system_tests/weblog, repository does not exist or may require 'docker login': denied: requested access to the resource is denied")

~~Do I need to publish the new otel weblog image somewhere?~~

songy23 · 2023-04-24T21:07:17Z

Interestingly sometimes in CI super().__init__ in OpenTelemetryScenario isn't executed 80d089f#diff-3faa2d19131dbfc2dc8b0bbe64b4dbacb1229e5fcb77659b8eb724a2b5dcf708R420 and thus we need to copy some logic to self.__init__. Not sure if this is a known issue since EndToEndScenario does the same.

songy23 · 2023-04-24T22:40:42Z

@cbeauchesne test-the-tests-open-telemetry passed: https://github.com/DataDog/system-tests/actions/runs/4791447134/jobs/8521939498?pr=976

utils/proxy/core.py

utils/interfaces/_library/core.py

songy23 · 2023-04-25T15:05:31Z

Thanks for the review!

github-actions bot added the run-default-scenario label Mar 16, 2023

songy23 force-pushed the yang.song/otlp-intake-test branch 2 times, most recently from 3896647 to b944af9 Compare March 17, 2023 17:14

[opentelemetry] Add OTLP intake E2E system tests

f66b76a

songy23 force-pushed the yang.song/otlp-intake-test branch from b944af9 to f66b76a Compare March 17, 2023 17:27

songy23 changed the title ~~[WIP] [opentelemetry] Add OTLP intake E2E system tests~~ [opentelemetry] Add OTLP intake E2E system tests Mar 17, 2023

songy23 requested review from dineshg13 and a team March 17, 2023 20:11

songy23 marked this pull request as ready for review March 17, 2023 20:11

songy23 requested a review from a team as a code owner March 17, 2023 20:11

lambrospetrou reviewed Mar 20, 2023

View reviewed changes

cbeauchesne added the do-not-merge label Mar 22, 2023

songy23 marked this pull request as draft March 22, 2023 18:27

songy23 added 3 commits March 25, 2023 10:34

Refactor OTel system tests

50a251c

- Associate OTel trace id with request id - Remove client side instrumentation - Remove trace context propagation - Support OTLP intake in proxy

Merge branch 'main' into yang.song/otlp-intake-test

4843837

Format

ec52e30

songy23 commented Mar 27, 2023

View reviewed changes

songy23 mentioned this pull request Mar 28, 2023

[opentelemetry] Add DD Agent and OTel Collector ingestion tests to OTel system tests #1016

Merged

songy23 marked this pull request as ready for review April 14, 2023 14:18

Merge branch 'main' into yang.song/otlp-intake-test

558e7d3

songy23 removed the do-not-merge label Apr 14, 2023

songy23 requested a review from cbeauchesne April 14, 2023 16:20

dineshg13 requested changes Apr 17, 2023

View reviewed changes

Fix review comments

9c94045

dineshg13 reviewed Apr 17, 2023

View reviewed changes

utils/build/docker/java_otel/spring-boot-native/pom.xml Outdated Show resolved Hide resolved

dineshg13 reviewed Apr 17, 2023

View reviewed changes

...ld/docker/java_otel/spring-boot-native/src/main/java/com/datadoghq/springbootnative/App.java Outdated Show resolved Hide resolved

Remove unused deps

8ee4333

Merge branch 'main' into yang.song/otlp-intake-test

75e47f3

cbeauchesne mentioned this pull request Apr 20, 2023

Revamp and clean scenarios/containers #1098

Merged

Merge main

230216c

cbeauchesne force-pushed the yang.song/otlp-intake-test branch from 0a8da8f to 230216c Compare April 20, 2023 14:49

introduce open telemetry interface

80d089f

cbeauchesne force-pushed the yang.song/otlp-intake-test branch from 1cafa88 to 80d089f Compare April 20, 2023 15:47

Minor fixes

62ae0b3

songy23 added 4 commits April 20, 2023 17:47

Use correct proxy in exporter

3aec32d

More fixes on otel e2e test

5810006

Merge branch 'main' into yang.song/otlp-intake-test

1b79c07

Fix a merge conflict

e298580

Add otel in CI

90e665a

Fix build failure in run.sh

ec918cc

Fix CI failure

c143cd4

songy23 removed the run-default-scenario label Apr 24, 2023

Fix test-the-tests-open-telemetry CI config

d42fdbd

songy23 force-pushed the yang.song/otlp-intake-test branch from e2a9635 to d42fdbd Compare April 24, 2023 21:59

Merge branch 'main' into yang.song/otlp-intake-test

9dfbd40

cbeauchesne approved these changes Apr 25, 2023

View reviewed changes

utils/proxy/core.py Outdated Show resolved Hide resolved

utils/interfaces/_library/core.py Outdated Show resolved Hide resolved

Remove unused func

04f1547

songy23 merged commit ccd292a into main Apr 25, 2023

songy23 deleted the yang.song/otlp-intake-test branch April 25, 2023 16:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[opentelemetry] Add OTLP intake E2E system tests #976

[opentelemetry] Add OTLP intake E2E system tests #976

songy23 commented Mar 16, 2023 •

edited

Loading

songy23 commented Mar 17, 2023

lambrospetrou Mar 20, 2023

songy23 Mar 27, 2023

cbeauchesne commented Mar 22, 2023

songy23 left a comment •

edited

Loading

songy23 Mar 27, 2023

songy23 commented Apr 14, 2023

cbeauchesne commented Apr 20, 2023

songy23 commented Apr 20, 2023

songy23 commented Apr 21, 2023 •

edited

Loading

cbeauchesne commented Apr 24, 2023

songy23 commented Apr 24, 2023 •

edited

Loading

songy23 commented Apr 24, 2023

songy23 commented Apr 24, 2023

songy23 commented Apr 25, 2023

[opentelemetry] Add OTLP intake E2E system tests #976

[opentelemetry] Add OTLP intake E2E system tests #976

Conversation

songy23 commented Mar 16, 2023 • edited Loading

Description

What's covered

What's next

Workflow

songy23 commented Mar 17, 2023

lambrospetrou Mar 20, 2023

Choose a reason for hiding this comment

songy23 Mar 27, 2023

Choose a reason for hiding this comment

cbeauchesne commented Mar 22, 2023

songy23 left a comment • edited Loading

Choose a reason for hiding this comment

songy23 Mar 27, 2023

Choose a reason for hiding this comment

songy23 commented Apr 14, 2023

cbeauchesne commented Apr 20, 2023

songy23 commented Apr 20, 2023

songy23 commented Apr 21, 2023 • edited Loading

cbeauchesne commented Apr 24, 2023

songy23 commented Apr 24, 2023 • edited Loading

songy23 commented Apr 24, 2023

songy23 commented Apr 24, 2023

songy23 commented Apr 25, 2023

songy23 commented Mar 16, 2023 •

edited

Loading

songy23 left a comment •

edited

Loading

songy23 commented Apr 21, 2023 •

edited

Loading

songy23 commented Apr 24, 2023 •

edited

Loading