Update apm-data: set a default `span.representative_count` #10792

axw · 2023-05-11T06:12:57Z

Motivation/summary

Update to apm-data changes:

Bring default RepresentativeCount for Spans in line with Transactions apm-data#30 (span.representative_count now defaults to 1)
model: Only write DSS source for metricsSets apm-data#33 (removes modelprocessor.DroppedSpansStatsDiscarder, now filtered in encoding)
model: introduce separate types for JSON encoding apm-data#35 (overhauled JSON encoding approach)

The metrics data structure has changed, so a new common ingest pipeline is added to handle these. We now send metrics as

"metricset.samples": [
  {"name": "metric_name1", "type": "histogram", "unit": "s", "values": [1, 2, 3], "counts": [4, 5, 6]},
  {"name": "metric_name2", "type": "summary", "value_count": 123, "sum": 456},
  {"name": "metric_name3", "type": "counter", "value": 123}
  {"name": "metric_name4", "value": 456}
]

The ingest pipeline takes care of mapping these to top-level metric fields as before.

Checklist

Update CHANGELOG.asciidoc
~~- [ ] Update package changelog.yml (only if changes to apmpackage have been made)~~
~~- [ ] Documentation has been updated~~

How to test these changes

The only functional change should be the span.representative_count one. This can be tested by sending spans, with destination fields set, to the server without sample_rate set, and checking that (a) the span documents indexed have representative_count: 1, and (b) service_destination metrics are produced.

Related issues

Closes #3565

mergify · 2023-05-11T06:13:31Z

This pull request does not have a backport label. Could you fix it @axw? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-7.x is the label to automatically backport to the 7.x branch.
backport-7./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

apmmachine · 2023-05-11T06:15:08Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-05-15T01:45:59.398+0000
Duration: 6 min 44 sec

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate and publish the docker images.
/test windows : Build & tests on Windows.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

simitt · 2023-05-11T09:45:39Z

I suggest to split the change for the metricset and the span.representative_count into two separate PRs as they are unrelated.

axw · 2023-05-11T10:07:35Z

I've pulled the apmpackage changes into #10793

apmmachine · 2023-05-12T04:02:23Z

📚 Go benchmark report

Diff with the main branch

goos: linux
goarch: amd64
pkg: github.com/elastic/apm-server/internal/agentcfg
cpu: 12th Gen Intel(R) Core(TM) i5-12500
                                  │ build/main/bench.out │             bench.out              │
                                  │        sec/op        │    sec/op     vs base              │
FetchAndAdd/FetchFromCache-12                42.17n ± 3%    46.07n ± 0%  +9.26% (p=0.002 n=6)
FetchAndAdd/FetchAndAddToCache-12            93.36n ± 2%   102.55n ± 2%  +9.84% (p=0.002 n=6)
geomean                                      62.75n         68.74n       +9.55%

                                  │ build/main/bench.out │             bench.out              │
                                  │         B/op         │    B/op     vs base                │
geomean                                                ²               +0.00%               ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                                  │ build/main/bench.out │             bench.out              │
                                  │      allocs/op       │ allocs/op   vs base                │
geomean                                                ²               +0.00%               ²
¹ all samples are equal
² summaries must be >0 to compute geomean

pkg: github.com/elastic/apm-server/internal/beater/request
                                             │ build/main/bench.out │              bench.out              │
                                             │        sec/op        │    sec/op     vs base               │
ContextResetContentEncoding/empty-12                   117.5n ±  3%   131.3n ±  2%  +11.79% (p=0.002 n=6)
ContextResetContentEncoding/uncompressed-12            140.1n ±  1%   156.7n ±  1%  +11.77% (p=0.002 n=6)
geomean                                                979.5n         906.9n         -7.41%

                                             │ build/main/bench.out │              bench.out               │
                                             │         B/op         │     B/op      vs base                │
geomean                                                           ²                 +0.00%               ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                                             │ build/main/bench.out │             bench.out              │
                                             │      allocs/op       │ allocs/op   vs base                │
geomean                                                           ²               +0.00%               ²
¹ all samples are equal
² summaries must be >0 to compute geomean

pkg: github.com/elastic/apm-server/internal/publish
             │ build/main/bench.out │          bench.out          │
             │        sec/op        │   sec/op    vs base         │

             │ build/main/bench.out │           bench.out            │
             │         B/op         │     B/op       vs base         │

             │ build/main/bench.out │           bench.out           │
             │      allocs/op       │  allocs/op    vs base         │

pkg: github.com/elastic/apm-server/x-pack/apm-server/aggregation/spanmetrics
                 │ build/main/bench.out │          bench.out           │
                 │        sec/op        │   sec/op     vs base         │

                 │ build/main/bench.out │            bench.out            │
                 │         B/op         │     B/op      vs base           │
¹ all samples are equal

                 │ build/main/bench.out │           bench.out           │
                 │      allocs/op       │ allocs/op   vs base           │
¹ all samples are equal

pkg: github.com/elastic/apm-server/x-pack/apm-server/aggregation/txmetrics
                        │ build/main/bench.out │          bench.out           │
                        │        sec/op        │   sec/op     vs base         │

                        │ build/main/bench.out │           bench.out           │
                        │         B/op         │    B/op     vs base           │
¹ all samples are equal

                        │ build/main/bench.out │           bench.out           │
                        │      allocs/op       │ allocs/op   vs base           │
¹ all samples are equal

pkg: github.com/elastic/apm-server/x-pack/apm-server/sampling
               │ build/main/bench.out │             bench.out              │
               │        sec/op        │    sec/op     vs base              │
geomean                  607.2n         619.0n        +1.95%

               │ build/main/bench.out │              bench.out               │
               │         B/op         │     B/op      vs base                │
geomean                             ²                 +0.19%               ²
¹ all samples are equal
² summaries must be >0 to compute geomean

               │ build/main/bench.out │             bench.out              │
               │      allocs/op       │ allocs/op   vs base                │
geomean                             ²               +0.00%               ²
¹ all samples are equal
² summaries must be >0 to compute geomean

pkg: github.com/elastic/apm-server/x-pack/apm-server/sampling/eventstorage
                                            │ build/main/bench.out │               bench.out               │
                                            │        sec/op        │    sec/op      vs base                │
WriteTransaction/json_codec-12                        4.139µ ±  5%   11.775µ ± 60%  +184.48% (p=0.002 n=6)
WriteTransaction/json_codec_big_tx-12                 5.056µ ±  4%   12.387µ ± 53%  +144.96% (p=0.002 n=6)
WriteTransaction/nop_codec_big_tx-12                  383.1n ± 87%    843.4n ± 68%  +120.17% (p=0.041 n=6)
ReadEvents/json_codec/0_events-12                     308.2n ±  5%    348.1n ±  8%   +12.94% (p=0.002 n=6)
ReadEvents/json_codec/10_events-12                    166.2µ ± 24%    123.6µ ± 23%   -25.60% (p=0.026 n=6)
ReadEvents/json_codec_big_tx/0_events-12              313.9n ± 10%    347.5n ±  7%   +10.69% (p=0.015 n=6)
ReadEvents/json_codec_big_tx/1_events-12              11.97µ ±  5%    12.83µ ±  8%    +7.21% (p=0.002 n=6)
ReadEvents/json_codec_big_tx/10_events-12             166.5µ ± 19%    139.1µ ± 14%   -16.44% (p=0.041 n=6)
ReadEvents/nop_codec/0_events-12                      304.6n ±  8%    346.6n ±  6%   +13.82% (p=0.004 n=6)
ReadEvents/nop_codec_big_tx/0_events-12               296.6n ±  6%    350.6n ±  8%   +18.22% (p=0.002 n=6)
IsTraceSampled/sampled-12                             69.69n ±  2%    77.11n ±  4%   +10.66% (p=0.002 n=6)
IsTraceSampled/unsampled-12                           71.98n ±  2%    79.60n ±  3%   +10.59% (p=0.002 n=6)
IsTraceSampled/unknown-12                             370.1n ±  2%    418.1n ±  9%   +12.98% (p=0.002 n=6)
geomean                                               28.92µ          31.43µ          +8.67%

                                            │ build/main/bench.out │               bench.out                │
                                            │         B/op         │      B/op       vs base                │
WriteTransaction/json_codec_big_tx-12                3.686Ki ±  0%    3.688Ki ±  0%  +0.05% (p=0.013 n=6)
ReadEvents/json_codec/100_events-12                  809.6Ki ±  0%    809.4Ki ±  0%  -0.02% (p=0.039 n=6)
geomean                                              31.34Ki          31.42Ki        +0.25%
¹ all samples are equal

                                            │ build/main/bench.out │              bench.out              │
                                            │      allocs/op       │  allocs/op   vs base                │
geomean                                                 144.7         144.7       +0.00%
¹ all samples are equal

report generated with https://pkg.go.dev/golang.org/x/perf/cmd/benchstat

axw · 2023-05-12T04:56:35Z

~~Waiting for elastic/kibana#157450 to be~~ merged.

mergify · 2023-05-12T09:08:29Z

This pull request is now in conflicts. Could you fix it @axw? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b update-apm-data upstream/update-apm-data
git merge upstream/main
git push upstream update-apm-data

mergify · 2023-05-12T15:26:49Z

This pull request is now in conflicts. Could you fix it @axw? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b update-apm-data upstream/update-apm-data
git merge upstream/main
git push upstream update-apm-data

axw · 2023-06-27T03:41:37Z

Testing this myself since I didn't actually make the representative_count change.

axw · 2023-06-27T03:44:16Z

Tested in conjunction with #10922. Verified with 8.9.0-BC1.

mergify bot added the backport-skip Skip notification from the automated backport with mergify label May 11, 2023

axw mentioned this pull request May 11, 2023

model: introduce separate types for JSON encoding elastic/apm-data#35

Merged

axw added the v8.9.0 label May 12, 2023

Update apm-data

8a2fda5

axw force-pushed the update-apm-data branch from fda69dc to 8a2fda5 Compare May 12, 2023 03:56

Update changelog

ac59880

axw changed the title ~~Update apm-data: set a default span.representative_count, handle new metrics structure~~ Update apm-data: set a default span.representative_count May 12, 2023

axw marked this pull request as ready for review May 12, 2023 08:43

axw requested a review from a team May 12, 2023 08:43

Merge branch 'main' into update-apm-data

bac54aa

Merge branch 'main' into update-apm-data

47bf3ab

marclop approved these changes May 15, 2023

View reviewed changes

axw merged commit 2ef4d39 into elastic:main May 15, 2023

axw deleted the update-apm-data branch May 15, 2023 03:57

kruskall added the test-plan label Jun 21, 2023

axw self-assigned this Jun 27, 2023

axw added the test-plan-ok label Jun 27, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update apm-data: set a default `span.representative_count` #10792

Update apm-data: set a default `span.representative_count` #10792

axw commented May 11, 2023 •

edited

Loading

mergify bot commented May 11, 2023

apmmachine commented May 11, 2023 •

edited

Loading

Build stats

simitt commented May 11, 2023

axw commented May 11, 2023

apmmachine commented May 12, 2023 •

edited

Loading

axw commented May 12, 2023 •

edited

Loading

mergify bot commented May 12, 2023

mergify bot commented May 12, 2023

axw commented Jun 27, 2023

axw commented Jun 27, 2023

Update apm-data: set a default span.representative_count #10792

Update apm-data: set a default span.representative_count #10792

Conversation

axw commented May 11, 2023 • edited Loading

Motivation/summary

Checklist

How to test these changes

Related issues

mergify bot commented May 11, 2023

apmmachine commented May 11, 2023 • edited Loading

💚 Build Succeeded

Build stats

🤖 GitHub comments

simitt commented May 11, 2023

axw commented May 11, 2023

apmmachine commented May 12, 2023 • edited Loading

📚 Go benchmark report

axw commented May 12, 2023 • edited Loading

mergify bot commented May 12, 2023

mergify bot commented May 12, 2023

axw commented Jun 27, 2023

axw commented Jun 27, 2023

Update apm-data: set a default `span.representative_count` #10792

Update apm-data: set a default `span.representative_count` #10792

axw commented May 11, 2023 •

edited

Loading

apmmachine commented May 11, 2023 •

edited

Loading

apmmachine commented May 12, 2023 •

edited

Loading

axw commented May 12, 2023 •

edited

Loading