Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Restructure model.Metricset #5739

Merged
merged 1 commit into from
Jul 16, 2021
Merged

Restructure model.Metricset #5739

merged 1 commit into from
Jul 16, 2021

Conversation

axw
Copy link
Member

@axw axw commented Jul 16, 2021

Motivation/summary

A few smallish changes to the model.Metricset type, for future-proofing. The goal is to make the type suitable for generating transformation to beat.Events logic later, while making the type better suited to protobuf definition in the short term.

model.Metricset.Samples is now a map, instead of a slice of named metrics, and the metrics fields are added as flat dotted field namess rather than hierarchical objects.

In the future we will send the samples map as-is (with values, type, and unit) to Elasticsearch and perform transformation to top-level metrics fields and dynamic templates completely in the ingest pipeline.

model.Sample is now called model.MetricsetSample, to clarify its relationship with model.Metricset.

We no longer set "_doc_count" in transformation; this is now part of the type and is set in transaction metrics aggregation
for "transaction.duration.histogram", the only metric where it is expected to be set.

Checklist

- [ ] Update CHANGELOG.asciidoc
- [ ] Documentation has been updated

How to test these changes

Non-functional change.

Related issues

#4120
#3565

@apmmachine
Copy link
Contributor

apmmachine commented Jul 16, 2021

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2021-07-16T06:59:46.513+0000

  • Duration: 42 min 42 sec

  • Commit: 9ab4631

Test stats 🧪

Test Results
Failed 1
Passed 5932
Skipped 14
Total 5947

Trends 🧪

Image of Build Times

Image of Tests

Test errors 1

Expand to view the tests failures

Build and Test / APM Integration Tests / test_rum – tests.agent.test_rum
    Expand to view the error details

     AssertionError: Expected done, got Timeout exceeded while waiting for event 
    

    Expand to view the stacktrace

     rum = <tests.fixtures.agents.Agent object at 0x7fd84d03de90>
    
        def test_rum(rum):
            elasticsearch = rum.apm_server.elasticsearch
            elasticsearch.clean()
            endpoint = Endpoint(rum.url, "run_integration_test", qu_str="echo=done", text="done")
        
            r = requests.get(endpoint.url)
    >       utils.check_request_response(r, endpoint)
    
    tests/agent/test_rum.py:12: 
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
    
    req = <Response [500]>
    endpoint = <tests.endpoint.Endpoint object at 0x7fd84d16ae10>
    
        def check_request_response(req, endpoint):
            msg = "Expected {}, got {}".format(endpoint.text, req.text)
    >       assert req.text.strip() == endpoint.text, msg
    E       AssertionError: Expected done, got Timeout exceeded while waiting for event
    
    tests/utils.py:38: AssertionError 
    

Steps errors 1

Expand to view the steps failures

Test Sync
  • Took 3 min 27 sec . View more details on here
  • Description: ./.ci/scripts/sync.sh

Log output

Expand to view the last 100 lines of log output

[2021-07-16T07:41:24.151Z] === RUN   TestBreakdownMetrics
[2021-07-16T07:41:24.151Z] --- PASS: TestBreakdownMetrics (4.04s)
[2021-07-16T07:41:24.151Z] === RUN   TestApplicationMetrics
[2021-07-16T07:41:24.151Z] --- PASS: TestApplicationMetrics (5.17s)
[2021-07-16T07:41:24.151Z] === RUN   TestAPMServerMonitoring
[2021-07-16T07:41:24.151Z] --- PASS: TestAPMServerMonitoring (0.57s)
[2021-07-16T07:41:24.151Z] === RUN   TestAPMServerMonitoringBuiltinUser
[2021-07-16T07:41:24.151Z] --- PASS: TestAPMServerMonitoringBuiltinUser (2.01s)
[2021-07-16T07:41:24.151Z] === RUN   TestAPMServerOnboarding
[2021-07-16T07:41:24.151Z] --- PASS: TestAPMServerOnboarding (3.83s)
[2021-07-16T07:41:24.151Z] === RUN   TestOTLPGRPCTraces
[2021-07-16T07:41:24.151Z] --- PASS: TestOTLPGRPCTraces (5.32s)
[2021-07-16T07:41:24.151Z] === RUN   TestOTLPGRPCMetrics
[2021-07-16T07:41:24.151Z] --- PASS: TestOTLPGRPCMetrics (4.23s)
[2021-07-16T07:41:24.151Z] === RUN   TestOTLPGRPCAuth
[2021-07-16T07:41:24.151Z] --- PASS: TestOTLPGRPCAuth (5.50s)
[2021-07-16T07:41:24.151Z] === RUN   TestOTLPClientIP
[2021-07-16T07:41:24.151Z] --- PASS: TestOTLPClientIP (3.56s)
[2021-07-16T07:41:24.151Z] === RUN   TestOTLPAnonymous
[2021-07-16T07:41:24.151Z] --- PASS: TestOTLPAnonymous (0.19s)
[2021-07-16T07:41:24.151Z] === RUN   TestOTLPRateLimit
[2021-07-16T07:41:24.151Z] --- PASS: TestOTLPRateLimit (0.25s)
[2021-07-16T07:41:24.151Z] === RUN   TestRUMXForwardedFor
[2021-07-16T07:41:24.151Z] --- PASS: TestRUMXForwardedFor (5.05s)
[2021-07-16T07:41:24.151Z] === RUN   TestRUMAllowServiceNames
[2021-07-16T07:41:24.151Z] --- PASS: TestRUMAllowServiceNames (0.18s)
[2021-07-16T07:41:24.151Z] === RUN   TestRUMRateLimit
[2021-07-16T07:41:24.151Z] --- PASS: TestRUMRateLimit (0.89s)
[2021-07-16T07:41:24.151Z] === RUN   TestRUMCORS
[2021-07-16T07:41:24.151Z] --- PASS: TestRUMCORS (0.19s)
[2021-07-16T07:41:24.151Z] === RUN   TestKeepUnsampled
[2021-07-16T07:41:24.151Z] === RUN   TestKeepUnsampled/false
[2021-07-16T07:41:24.151Z] === RUN   TestKeepUnsampled/true
[2021-07-16T07:41:24.151Z] --- PASS: TestKeepUnsampled (8.88s)
[2021-07-16T07:41:24.151Z]     --- PASS: TestKeepUnsampled/false (3.59s)
[2021-07-16T07:41:24.151Z]     --- PASS: TestKeepUnsampled/true (5.29s)
[2021-07-16T07:41:24.151Z] === RUN   TestKeepUnsampledWarning
[2021-07-16T07:41:24.151Z] --- PASS: TestKeepUnsampledWarning (3.78s)
[2021-07-16T07:41:24.151Z] === RUN   TestTailSampling
[2021-07-16T07:41:24.151Z]     sampling_test.go:135: waiting for 100 "parent" transactions
[2021-07-16T07:41:24.151Z]     sampling_test.go:135: waiting for 100 "child" transactions
[2021-07-16T07:41:24.151Z] --- PASS: TestTailSampling (8.36s)
[2021-07-16T07:41:24.151Z] === RUN   TestTailSamplingUnlicensed
[2021-07-16T07:41:24.151Z] 2021/07/16 07:40:04 Starting container id: 0c30525d4291 image: docker.elastic.co/elasticsearch/elasticsearch:8.0.0-52a5431f-SNAPSHOT
[2021-07-16T07:41:24.151Z] 2021/07/16 07:40:04 Waiting for container id 0c30525d4291 image: docker.elastic.co/elasticsearch/elasticsearch:8.0.0-52a5431f-SNAPSHOT
[2021-07-16T07:41:24.151Z] 2021/07/16 07:40:22 Container is ready id: 0c30525d4291 image: docker.elastic.co/elasticsearch/elasticsearch:8.0.0-52a5431f-SNAPSHOT
[2021-07-16T07:41:24.151Z] --- PASS: TestTailSamplingUnlicensed (29.65s)
[2021-07-16T07:41:24.151Z] === RUN   TestRUMErrorSourcemapping
[2021-07-16T07:41:24.151Z] --- PASS: TestRUMErrorSourcemapping (4.57s)
[2021-07-16T07:41:24.151Z] === RUN   TestRUMSpanSourcemapping
[2021-07-16T07:41:24.151Z] --- PASS: TestRUMSpanSourcemapping (5.06s)
[2021-07-16T07:41:24.151Z] === RUN   TestDuplicateSourcemapWarning
[2021-07-16T07:41:24.151Z] --- PASS: TestDuplicateSourcemapWarning (4.30s)
[2021-07-16T07:41:24.151Z] === RUN   TestNoMatchingSourcemap
[2021-07-16T07:41:24.151Z] --- PASS: TestNoMatchingSourcemap (5.14s)
[2021-07-16T07:41:24.151Z] === RUN   TestFetchLatestSourcemap
[2021-07-16T07:41:24.151Z] --- PASS: TestFetchLatestSourcemap (5.87s)
[2021-07-16T07:41:24.151Z] === RUN   TestSourcemapCaching
[2021-07-16T07:41:24.151Z] --- PASS: TestSourcemapCaching (5.62s)
[2021-07-16T07:41:24.151Z] === RUN   TestIndexTemplateCoverage
[2021-07-16T07:41:24.151Z] --- PASS: TestIndexTemplateCoverage (5.78s)
[2021-07-16T07:41:24.151Z] PASS
[2021-07-16T07:41:24.151Z] ok  	github.com/elastic/apm-server/systemtest	419.191s
[2021-07-16T07:41:24.151Z] === RUN   TestAPMServer
[2021-07-16T07:41:24.151Z] 2021/07/16 07:34:09 Building apm-server...
[2021-07-16T07:41:24.151Z] 2021/07/16 07:34:11 Built /var/lib/jenkins/workspace/pm-server_apm-server-mbp_PR-5739/src/github.com/elastic/apm-server/apm-server
[2021-07-16T07:41:24.151Z] --- PASS: TestAPMServer (2.44s)
[2021-07-16T07:41:24.151Z] === RUN   TestUnstartedAPMServer
[2021-07-16T07:41:24.151Z] --- PASS: TestUnstartedAPMServer (0.00s)
[2021-07-16T07:41:24.151Z] === RUN   TestAPMServerStartTLS
[2021-07-16T07:41:24.151Z] --- PASS: TestAPMServerStartTLS (0.14s)
[2021-07-16T07:41:24.151Z] === RUN   TestExpvar
[2021-07-16T07:41:24.151Z] --- PASS: TestExpvar (0.12s)
[2021-07-16T07:41:24.151Z] PASS
[2021-07-16T07:41:24.151Z] ok  	github.com/elastic/apm-server/systemtest/apmservertest	2.718s
[2021-07-16T07:41:24.151Z] ?   	github.com/elastic/apm-server/systemtest/benchtest	[no test files]
[2021-07-16T07:41:24.151Z] ?   	github.com/elastic/apm-server/systemtest/cmd/apmbench	[no test files]
[2021-07-16T07:41:24.151Z] ?   	github.com/elastic/apm-server/systemtest/estest	[no test files]
[2021-07-16T07:41:24.151Z] ?   	github.com/elastic/apm-server/systemtest/fleettest	[no test files]
[2021-07-16T07:41:24.151Z] + cleanup
[2021-07-16T07:41:24.151Z] + rm -rf /tmp/tmp.3LKodenXF9
[2021-07-16T07:41:24.151Z] + .ci/scripts/docker-get-logs.sh
[2021-07-16T07:41:24.151Z] It is not possible to grab the logs of 7016eee6ee89
[2021-07-16T07:41:25.294Z] Post stage
[2021-07-16T07:41:25.311Z] Running in /var/lib/jenkins/workspace/pm-server_apm-server-mbp_PR-5739/src/github.com/elastic/apm-server/build
[2021-07-16T07:41:25.338Z] Archiving artifacts
[2021-07-16T07:41:25.719Z] Recording test results
[2021-07-16T07:41:26.312Z] [Checks API] No suitable checks publisher found.
[2021-07-16T07:41:26.658Z] + tar --version
[2021-07-16T07:41:27.008Z] + tar --exclude=system-tests-linux-files.tgz -czf system-tests-linux-files.tgz system-tests
[2021-07-16T07:41:27.030Z] Archiving artifacts
[2021-07-16T07:41:27.626Z] Terminated
[2021-07-16T07:41:27.612Z] Terminated
[2021-07-16T07:41:27.789Z] Running on Jenkins in /var/lib/jenkins/workspace/pm-server_apm-server-mbp_PR-5739
[2021-07-16T07:41:27.849Z] [INFO] getVaultSecret: Getting secrets
[2021-07-16T07:41:27.905Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-07-16T07:41:28.800Z] + chmod 755 generate-build-data.sh
[2021-07-16T07:41:28.800Z] + ./generate-build-data.sh https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-server/apm-server-mbp/PR-5739/ https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-server/apm-server-mbp/PR-5739/runs/2 UNSTABLE 2501998
[2021-07-16T07:41:28.800Z] INFO: curl https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-server/apm-server-mbp/PR-5739/runs/2/steps/?limit=10000 -o steps-info.json
[2021-07-16T07:41:29.350Z] INFO: curl https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-server/apm-server-mbp/PR-5739/runs/2/tests/?status=FAILED -o tests-errors.json

A few smallish changes to the model.Metricset type, for
future-proofing. The goal is to make the type suitable for
generating transformation to beat.Events logic later, while
making the type better suited to protobuf definition in the
short term.

model.Metricset.Samples is now a map, instead of a slice of
named metrics, and the metrics fields are added as flat
dotted field namess rather than hierarchical objects.

In the future we will send the samples map as-is (with values,
type, and unit) to Elasticsearch and perform transformation
to top-level metrics fields and dynamic templates completely
in the ingest pipeline.

model.Sample is now called model.MetricsetSample.

We no longer set "_doc_count" in transformation; this is now
part of the type and is set in transaction metrics aggregation
for "transaction.duration.histogram", the only metric where it
is expected to be set.
@axw axw force-pushed the metricset-ingest branch from cf1f403 to 9ab4631 Compare July 16, 2021 06:59
@axw axw marked this pull request as ready for review July 16, 2021 07:45
@axw axw requested a review from a team July 16, 2021 07:46
Copy link
Contributor

@simitt simitt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changes make the coder easier to comprehend - thanks.

@axw axw merged commit 4ac8c60 into elastic:master Jul 16, 2021
@axw axw deleted the metricset-ingest branch July 16, 2021 09:53
mergify bot pushed a commit that referenced this pull request Jul 16, 2021
A few smallish changes to the model.Metricset type, for
future-proofing. The goal is to make the type suitable for
generating transformation to beat.Events logic later, while
making the type better suited to protobuf definition in the
short term.

model.Metricset.Samples is now a map, instead of a slice of
named metrics, and the metrics fields are added as flat
dotted field namess rather than hierarchical objects.

In the future we will send the samples map as-is (with values,
type, and unit) to Elasticsearch and perform transformation
to top-level metrics fields and dynamic templates completely
in the ingest pipeline.

model.Sample is now called model.MetricsetSample.

We no longer set "_doc_count" in transformation; this is now
part of the type and is set in transaction metrics aggregation
for "transaction.duration.histogram", the only metric where it
is expected to be set.

(cherry picked from commit 4ac8c60)
axw added a commit that referenced this pull request Jul 16, 2021
A few smallish changes to the model.Metricset type, for
future-proofing. The goal is to make the type suitable for
generating transformation to beat.Events logic later, while
making the type better suited to protobuf definition in the
short term.

model.Metricset.Samples is now a map, instead of a slice of
named metrics, and the metrics fields are added as flat
dotted field namess rather than hierarchical objects.

In the future we will send the samples map as-is (with values,
type, and unit) to Elasticsearch and perform transformation
to top-level metrics fields and dynamic templates completely
in the ingest pipeline.

model.Sample is now called model.MetricsetSample.

We no longer set "_doc_count" in transformation; this is now
part of the type and is set in transaction metrics aggregation
for "transaction.duration.histogram", the only metric where it
is expected to be set.

(cherry picked from commit 4ac8c60)

Co-authored-by: Andrew Wilkins <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants