Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move labels and timestamp to model.APMEvent #5840

Merged
merged 5 commits into from
Aug 2, 2021

Conversation

axw
Copy link
Member

@axw axw commented Aug 2, 2021

Motivation/summary

  • Remove event-specific labels fields, and combine event-specific labels with "metadata" labels at event decoding time.
  • Remove event-specific timestamp fields, and add a timestamp field to model.APMEvent

Checklist

- [ ] Update CHANGELOG.asciidoc
- [ ] Documentation has been updated

How to test these changes

Non-functional change.

Related issues

#4120
#3565

@apmmachine
Copy link
Contributor

apmmachine commented Aug 2, 2021

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2021-08-02T02:10:35.597+0000

  • Duration: 41 min 10 sec

  • Commit: d9b777e

Test stats 🧪

Test Results
Failed 1
Passed 5679
Skipped 13
Total 5693

Trends 🧪

Image of Build Times

Image of Tests

Test errors 1

Expand to view the tests failures

Build and Test / ARM build-test / TestFleetFetch – sourcemap
    Expand to view the error details

     Failed 
    

    Expand to view the stacktrace

     ==================
    WARNING: DATA RACE
    Read at 0x00c000043550 by goroutine 25:
      github.com/elastic/apm-server/sourcemap.TestFleetFetch()
          /var/lib/jenkins/workspace/pm-server_apm-server-mbp_PR-5840/src/github.com/elastic/apm-server/sourcemap/fleet_store_test.go:87 +0x660
      testing.tRunner()
          /var/lib/jenkins/workspace/pm-server_apm-server-mbp_PR-5840/.gvm/versions/go1.16.6.linux.arm64/src/testing/testing.go:1193 +0x170
    
    Previous write at 0x00c000043550 by goroutine 41:
      github.com/elastic/apm-server/sourcemap.TestFleetFetch.func1()
          /var/lib/jenkins/workspace/pm-server_apm-server-mbp_PR-5840/src/github.com/elastic/apm-server/sourcemap/fleet_store_test.go:51 +0x170
      net/http.HandlerFunc.ServeHTTP()
          /var/lib/jenkins/workspace/pm-server_apm-server-mbp_PR-5840/.gvm/versions/go1.16.6.linux.arm64/src/net/http/server.go:2049 +0x40
      net/http.serverHandler.ServeHTTP()
          /var/lib/jenkins/workspace/pm-server_apm-server-mbp_PR-5840/.gvm/versions/go1.16.6.linux.arm64/src/net/http/server.go:2867 +0xa0
      net/http.(*conn).serve()
          /var/lib/jenkins/workspace/pm-server_apm-server-mbp_PR-5840/.gvm/versions/go1.16.6.linux.arm64/src/net/http/server.go:1932 +0x648
    
    Goroutine 25 (running) created at:
      testing.(*T).Run()
          /var/lib/jenkins/workspace/pm-server_apm-server-mbp_PR-5840/.gvm/versions/go1.16.6.linux.arm64/src/testing/testing.go:1238 +0x438
      testing.runTests.func1()
          /var/lib/jenkins/workspace/pm-server_apm-server-mbp_PR-5840/.gvm/versions/go1.16.6.linux.arm64/src/testing/testing.go:1511 +0x90
      testing.tRunner()
          /var/lib/jenkins/workspace/pm-server_apm-server-mbp_PR-5840/.gvm/versions/go1.16.6.linux.arm64/src/testing/testing.go:1193 +0x170
      testing.runTests()
          /var/lib/jenkins/workspace/pm-server_apm-server-mbp_PR-5840/.gvm/versions/go1.16.6.linux.arm64/src/testing/testing.go:1509 +0x510
      testing.(*M).Run()
          /var/lib/jenkins/workspace/pm-server_apm-server-mbp_PR-5840/.gvm/versions/go1.16.6.linux.arm64/src/testing/testing.go:1417 +0x2d0
      main.main()
          _testmain.go:135 +0x2ec
    
    Goroutine 41 (running) created at:
      net/http.(*Server).Serve()
          /var/lib/jenkins/workspace/pm-server_apm-server-mbp_PR-5840/.gvm/versions/go1.16.6.linux.arm64/src/net/http/server.go:2993 +0x4d0
      net/http/httptest.(*Server).goServe.func1()
          /var/lib/jenkins/workspace/pm-server_apm-server-mbp_PR-5840/.gvm/versions/go1.16.6.linux.arm64/src/net/http/httptest/server.go:308 +0x9c
    ==================
        testing.go:1092: race detected during execution of test 
    

Steps errors 1

Expand to view the steps failures

Test Sync
  • Took 3 min 29 sec . View more details on here
  • Description: ./.ci/scripts/sync.sh

Log output

Expand to view the last 100 lines of log output

[2021-08-02T02:51:41.970Z] === RUN   TestBreakdownMetrics
[2021-08-02T02:51:41.970Z] --- PASS: TestBreakdownMetrics (4.11s)
[2021-08-02T02:51:41.970Z] === RUN   TestApplicationMetrics
[2021-08-02T02:51:41.970Z] --- PASS: TestApplicationMetrics (5.72s)
[2021-08-02T02:51:41.970Z] === RUN   TestAPMServerMonitoring
[2021-08-02T02:51:41.970Z] --- PASS: TestAPMServerMonitoring (1.11s)
[2021-08-02T02:51:41.970Z] === RUN   TestAPMServerMonitoringBuiltinUser
[2021-08-02T02:51:41.970Z] --- PASS: TestAPMServerMonitoringBuiltinUser (1.99s)
[2021-08-02T02:51:41.970Z] === RUN   TestAPMServerOnboarding
[2021-08-02T02:51:41.970Z] --- PASS: TestAPMServerOnboarding (5.60s)
[2021-08-02T02:51:41.970Z] === RUN   TestOTLPGRPCTraces
[2021-08-02T02:51:41.970Z] --- PASS: TestOTLPGRPCTraces (4.16s)
[2021-08-02T02:51:41.970Z] === RUN   TestOTLPGRPCMetrics
[2021-08-02T02:51:41.970Z] --- PASS: TestOTLPGRPCMetrics (6.35s)
[2021-08-02T02:51:41.970Z] === RUN   TestOTLPGRPCAuth
[2021-08-02T02:51:41.970Z] --- PASS: TestOTLPGRPCAuth (4.08s)
[2021-08-02T02:51:41.970Z] === RUN   TestOTLPClientIP
[2021-08-02T02:51:41.970Z] --- PASS: TestOTLPClientIP (5.28s)
[2021-08-02T02:51:41.970Z] === RUN   TestOTLPAnonymous
[2021-08-02T02:51:41.970Z] --- PASS: TestOTLPAnonymous (0.18s)
[2021-08-02T02:51:41.970Z] === RUN   TestOTLPRateLimit
[2021-08-02T02:51:41.970Z] --- PASS: TestOTLPRateLimit (0.29s)
[2021-08-02T02:51:41.970Z] === RUN   TestRUMXForwardedFor
[2021-08-02T02:51:41.970Z] --- PASS: TestRUMXForwardedFor (4.39s)
[2021-08-02T02:51:41.970Z] === RUN   TestRUMAllowServiceNames
[2021-08-02T02:51:41.970Z] --- PASS: TestRUMAllowServiceNames (0.15s)
[2021-08-02T02:51:41.970Z] === RUN   TestRUMRateLimit
[2021-08-02T02:51:41.970Z] --- PASS: TestRUMRateLimit (0.90s)
[2021-08-02T02:51:41.970Z] === RUN   TestRUMCORS
[2021-08-02T02:51:41.970Z] --- PASS: TestRUMCORS (0.16s)
[2021-08-02T02:51:41.970Z] === RUN   TestKeepUnsampled
[2021-08-02T02:51:41.970Z] === RUN   TestKeepUnsampled/false
[2021-08-02T02:51:41.970Z] === RUN   TestKeepUnsampled/true
[2021-08-02T02:51:41.970Z] --- PASS: TestKeepUnsampled (9.54s)
[2021-08-02T02:51:41.970Z]     --- PASS: TestKeepUnsampled/false (5.40s)
[2021-08-02T02:51:41.970Z]     --- PASS: TestKeepUnsampled/true (4.14s)
[2021-08-02T02:51:41.970Z] === RUN   TestKeepUnsampledWarning
[2021-08-02T02:51:41.971Z] --- PASS: TestKeepUnsampledWarning (5.25s)
[2021-08-02T02:51:41.971Z] === RUN   TestTailSampling
[2021-08-02T02:51:41.971Z]     sampling_test.go:135: waiting for 100 "parent" transactions
[2021-08-02T02:51:41.971Z]     sampling_test.go:135: waiting for 100 "child" transactions
[2021-08-02T02:51:41.971Z] --- PASS: TestTailSampling (6.99s)
[2021-08-02T02:51:41.971Z] === RUN   TestTailSamplingUnlicensed
[2021-08-02T02:51:41.971Z] 2021/08/02 02:50:21 Starting container id: 8f518b6b203a image: docker.elastic.co/elasticsearch/elasticsearch:8.0.0-cf3e6139-SNAPSHOT
[2021-08-02T02:51:41.971Z] 2021/08/02 02:50:22 Waiting for container id 8f518b6b203a image: docker.elastic.co/elasticsearch/elasticsearch:8.0.0-cf3e6139-SNAPSHOT
[2021-08-02T02:51:41.971Z] 2021/08/02 02:50:40 Container is ready id: 8f518b6b203a image: docker.elastic.co/elasticsearch/elasticsearch:8.0.0-cf3e6139-SNAPSHOT
[2021-08-02T02:51:41.971Z] --- PASS: TestTailSamplingUnlicensed (29.96s)
[2021-08-02T02:51:41.971Z] === RUN   TestRUMErrorSourcemapping
[2021-08-02T02:51:41.971Z] --- PASS: TestRUMErrorSourcemapping (4.49s)
[2021-08-02T02:51:41.971Z] === RUN   TestRUMSpanSourcemapping
[2021-08-02T02:51:41.971Z] --- PASS: TestRUMSpanSourcemapping (4.51s)
[2021-08-02T02:51:41.971Z] === RUN   TestDuplicateSourcemapWarning
[2021-08-02T02:51:41.971Z] --- PASS: TestDuplicateSourcemapWarning (5.43s)
[2021-08-02T02:51:41.971Z] === RUN   TestNoMatchingSourcemap
[2021-08-02T02:51:41.971Z] --- PASS: TestNoMatchingSourcemap (4.61s)
[2021-08-02T02:51:41.971Z] === RUN   TestFetchLatestSourcemap
[2021-08-02T02:51:41.971Z] --- PASS: TestFetchLatestSourcemap (6.61s)
[2021-08-02T02:51:41.971Z] === RUN   TestSourcemapCaching
[2021-08-02T02:51:41.971Z] --- PASS: TestSourcemapCaching (6.70s)
[2021-08-02T02:51:41.971Z] === RUN   TestIndexTemplateCoverage
[2021-08-02T02:51:41.971Z] --- PASS: TestIndexTemplateCoverage (4.96s)
[2021-08-02T02:51:41.971Z] PASS
[2021-08-02T02:51:41.971Z] ok  	github.com/elastic/apm-server/systemtest	452.194s
[2021-08-02T02:51:41.971Z] === RUN   TestAPMServer
[2021-08-02T02:51:41.971Z] 2021/08/02 02:43:54 Building apm-server...
[2021-08-02T02:51:41.971Z] 2021/08/02 02:43:57 Built /var/lib/jenkins/workspace/pm-server_apm-server-mbp_PR-5840/src/github.com/elastic/apm-server/apm-server
[2021-08-02T02:51:41.971Z] --- PASS: TestAPMServer (2.61s)
[2021-08-02T02:51:41.971Z] === RUN   TestUnstartedAPMServer
[2021-08-02T02:51:41.971Z] --- PASS: TestUnstartedAPMServer (0.00s)
[2021-08-02T02:51:41.971Z] === RUN   TestAPMServerStartTLS
[2021-08-02T02:51:41.971Z] --- PASS: TestAPMServerStartTLS (0.11s)
[2021-08-02T02:51:41.971Z] === RUN   TestExpvar
[2021-08-02T02:51:41.971Z] --- PASS: TestExpvar (0.09s)
[2021-08-02T02:51:41.971Z] PASS
[2021-08-02T02:51:41.971Z] ok  	github.com/elastic/apm-server/systemtest/apmservertest	2.832s
[2021-08-02T02:51:41.971Z] ?   	github.com/elastic/apm-server/systemtest/benchtest	[no test files]
[2021-08-02T02:51:41.971Z] ?   	github.com/elastic/apm-server/systemtest/cmd/apmbench	[no test files]
[2021-08-02T02:51:41.971Z] ?   	github.com/elastic/apm-server/systemtest/estest	[no test files]
[2021-08-02T02:51:41.971Z] ?   	github.com/elastic/apm-server/systemtest/fleettest	[no test files]
[2021-08-02T02:51:41.971Z] + cleanup
[2021-08-02T02:51:41.971Z] + rm -rf /tmp/tmp.NyDrTbIj7V
[2021-08-02T02:51:41.971Z] + .ci/scripts/docker-get-logs.sh
[2021-08-02T02:51:41.971Z] It is not possible to grab the logs of 265890d813e2
[2021-08-02T02:51:43.009Z] Post stage
[2021-08-02T02:51:43.025Z] Running in /var/lib/jenkins/workspace/pm-server_apm-server-mbp_PR-5840/src/github.com/elastic/apm-server/build
[2021-08-02T02:51:43.053Z] Archiving artifacts
[2021-08-02T02:51:43.419Z] Recording test results
[2021-08-02T02:51:44.003Z] [Checks API] No suitable checks publisher found.
[2021-08-02T02:51:44.375Z] + tar --version
[2021-08-02T02:51:44.718Z] + tar --exclude=system-tests-linux-files.tgz -czf system-tests-linux-files.tgz system-tests
[2021-08-02T02:51:44.738Z] Archiving artifacts
[2021-08-02T02:51:45.149Z] Terminated
[2021-08-02T02:51:45.128Z] Terminated
[2021-08-02T02:51:45.376Z] Running on Jenkins in /var/lib/jenkins/workspace/pm-server_apm-server-mbp_PR-5840
[2021-08-02T02:51:45.435Z] [INFO] getVaultSecret: Getting secrets
[2021-08-02T02:51:45.490Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-08-02T02:51:46.272Z] + chmod 755 generate-build-data.sh
[2021-08-02T02:51:46.272Z] + ./generate-build-data.sh https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-server/apm-server-mbp/PR-5840/ https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-server/apm-server-mbp/PR-5840/runs/2 UNSTABLE 2470412
[2021-08-02T02:51:46.272Z] INFO: curl https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-server/apm-server-mbp/PR-5840/runs/2/steps/?limit=10000 -o steps-info.json
[2021-08-02T02:51:46.522Z] INFO: curl https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-server/apm-server-mbp/PR-5840/runs/2/tests/?status=FAILED -o tests-errors.json

axw added 5 commits August 2, 2021 10:10
Remove Transaction.Labels, Span.Labels, etc., and combine
global labels ("metadata labels") with event-specific labels
at decoding and model construction time.
Remove modeldecoder.Input.RequestTime. Instead, we will
have processor/stream initialise the base event's timestamp
to the request time and leave it unmodified in the event
that no timestamp is specified for the event.
@axw axw force-pushed the model-apmevent-timestamp branch from 000fd4f to d9b777e Compare August 2, 2021 02:10
@axw axw marked this pull request as ready for review August 2, 2021 03:03
@axw axw requested a review from a team August 2, 2021 03:03
@axw axw merged commit c041e85 into elastic:master Aug 2, 2021
@axw axw deleted the model-apmevent-timestamp branch August 2, 2021 05:34
mergify bot pushed a commit that referenced this pull request Aug 2, 2021
* Remove event-specific Labels fields

Remove Transaction.Labels, Span.Labels, etc., and combine
global labels ("metadata labels") with event-specific labels
at decoding and model construction time.

* model: move Timestamp to APMEvent

* model/modeldecoder: adapt to APMEvent.Timestamp

Remove modeldecoder.Input.RequestTime. Instead, we will
have processor/stream initialise the base event's timestamp
to the request time and leave it unmodified in the event
that no timestamp is specified for the event.

* processor/otel: adapt to APMEvent.Timestamp

* aggregation: adapt to APMEvent.Timestamp

(cherry picked from commit c041e85)
axw added a commit that referenced this pull request Aug 2, 2021
* Remove event-specific Labels fields

Remove Transaction.Labels, Span.Labels, etc., and combine
global labels ("metadata labels") with event-specific labels
at decoding and model construction time.

* model: move Timestamp to APMEvent

* model/modeldecoder: adapt to APMEvent.Timestamp

Remove modeldecoder.Input.RequestTime. Instead, we will
have processor/stream initialise the base event's timestamp
to the request time and leave it unmodified in the event
that no timestamp is specified for the event.

* processor/otel: adapt to APMEvent.Timestamp

* aggregation: adapt to APMEvent.Timestamp

(cherry picked from commit c041e85)

Co-authored-by: Andrew Wilkins <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants