Generate fields for APM package #4432

jalvz · 2020-11-17T07:39:00Z

As title. For more details, head to the diff in the Readme.md file.

Motivation/summary

Checklist

I have signed the Contributor License Agreement.
I have updated CHANGELOG.asciidoc

I have considered changes for:

documentation
logging (add log lines, choose appropriate log selector, etc.)
metrics and monitoring (create issue for Kibana team to add metrics to visualizations, e.g. Kibana#44001)
automated tests (add tests for the code changes, all unit tests pass locally)
telemetry
Elasticsearch Service (https://cloud.elastic.co)
Elastic Cloud Enterprise (https://www.elastic.co/products/ece)
Elastic Cloud on Kubernetes (https://www.elastic.co/elastic-cloud-kubernetes)

How to test these changes

run make update
check the contents of apmpackage/apm

Related issues

#4004

apmmachine · 2020-11-17T09:23:46Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: [Pull request #4432 updated]
Start Time: 2020-11-25T08:51:20.964+0000
Duration: 40 min 25 sec

Test stats 🧪

Test	Results
Failed	0
Passed	4754
Skipped	143
Total	4897

Steps errors

Expand to view the steps failures

`Compress`

Took 0 min 0 sec . View more details on here
Description: tar --exclude=coverage-files.tgz -czf coverage-files.tgz coverage

`Compress`

Took 0 min 0 sec . View more details on here
Description: tar --exclude=system-tests-linux-files.tgz -czf system-tests-linux-files.tgz system-tests

`Test Sync`

Took 3 min 49 sec . View more details on here
Description: ./.ci/scripts/sync.sh

simitt

Overall LGTM on a first review. My main question is how this integrates with any update workflow - will it be part of running make update?

Do you plan on addressing the TODOs before labeling this ready for review? (I believe fetching ECS info from github rather than expecting local installation would make sense).

apmpackage/genfields.go

apmpackage/model.go

jalvz · 2020-11-18T07:16:22Z

Overall LGTM on a first review. My main question is how this integrates with any update workflow - will it be part of running make update?

Exactly, it should be part of the target that generates the big fields.yml file.

Do you plan on addressing the TODOs before labeling this ready for review? (I believe fetching ECS info from github rather than expecting local installation would make sense).

Thanks for validating, I wasn't sure (I copied the loadECSFields method from the integrations repo, that expects a local install, that's why) So yeah, will do that

…ields

… into generate-package-fields

apmpackage/apm/0.1.0/data_stream/logs/fields/ecs.yml

apmpackage/apm/0.1.0/manifest.yml

axw · 2020-11-24T06:03:15Z

apmpackage/cmd/gen-package/main.go

+var packageVersion string
+
+func main() {
+	flag.StringVar(&packageVersion, "packageVersion", "0.1.0", "Package version")


Maybe add a TODO to get the default version from cmd/version.go (or move that into a package and make it an exported variable)?

done, lmk if i got it right

apmpackage/gendocs.go

apmpackage/model.go

tests/system/apmserver.py

go.mod

axw · 2020-11-24T07:09:33Z

A couple of comment which I forgot to add:

I wonder if there's value in checking in the generated files, vs. just checking in the script and static files. We can generate into the build directory, and copy from there into the integrations repo.
It would be nice to cache the (versioned) ecs_flat.yml files in the repo, so we don't have to download them every time we run make update.

jalvz · 2020-11-24T08:17:24Z

I wonder if there's value in checking in the generated files

I think there is. Eg. when new fields are added is easy to see in the diff the table generated for the readme file and check it is rendered properly.

I don't have a strong opinion thou.

It would be nice to cache the (versioned) ecs_flat.yml files in the repo, so we don't have to download them every time we run make update.

I can add a TODO for that, but not sure that is worth the trouble, tbh. It takes millisecs and we need an internet connection to push the changes anyways.

axw · 2020-11-24T08:43:41Z

I think there is. Eg. when new fields are added is easy to see in the diff the table generated for the readme file and check it is rendered properly.

I don't have a strong opinion thou.

Same, so keep it for now and maybe we'll revisit it later.

I can add a TODO for that, but not sure that is worth the trouble, tbh. It takes millisecs and we need and internet connection to push the changes anyways.

When we're updating the package, sure. I run make update all the time though, and I would prefer that it didn't reach out to the internet unless needed (most of the time it would not). Please add a TODO -- I'm happy to do the work since it may only annoy me ;)

…ields

codecov-io · 2020-11-24T11:09:28Z

Codecov Report

Merging #4432 (4d29307) into master (4727a8b) will decrease coverage by 0.02%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #4432      +/-   ##
==========================================
- Coverage   76.11%   76.09%   -0.03%     
==========================================
  Files         158      158              
  Lines        9782     9782              
==========================================
- Hits         7446     7444       -2     
- Misses       2336     2338       +2

Impacted Files	Coverage Δ
beater/jaeger/common.go	`75.00% <0.00%> (-7.15%)`	⬇️
...ack/apm-server/aggregation/txmetrics/aggregator.go	`93.36% <0.00%> (ø)`

jalvz · 2020-11-24T11:54:28Z

Wrote some actual docs in 2411077. Ill hand that off to Brandon for polishing and completing. I guess he will move that somewhere else and link instead; but it would be great a review now to make sure they are accurate and understandable.

axw

LGTM with the additional x-pack fields. Thanks!

I haven't been over it with a fine-tooth comb, but it looks good and safe - so we can adjust over time as we test if needed.

apmpackage/cmd/gen-package/genfields.go

Co-authored-by: Andrew Wilkins <[email protected]>

… into generate-package-fields

Adds an APM package for Fleet, with a script to generate fields.yml files and docs # Conflicts: # go.mod # include/fields.go

Adds an APM package for Fleet, with a script to generate fields.yml files and docs

simitt · 2020-12-16T16:36:01Z

@jalvz not sure this needs additional manual testing - please feel free to remove the labels in case you don't think it brings value.

simitt · 2020-12-23T20:35:09Z

Loooked into how to test this and decided to add a test-plan-skip label as everything that has been generated via make update has been committed and therefore approved already.

Generate fields for APM package

fb72b94

fix bits here and there

bb37e3e

simitt reviewed Nov 17, 2020

View reviewed changes

apmpackage/genfields.go Outdated Show resolved Hide resolved

apmpackage/model.go Outdated Show resolved Hide resolved

apmpackage/model.go Outdated Show resolved Hide resolved

apmpackage/model.go Outdated Show resolved Hide resolved

jalvz added 4 commits November 20, 2020 11:33

Merge remote-tracking branch 'elastic/master' into generate-package-f…

12a979e

…ields

Generate docs out of fields.yml files

3b19a24

update go.mod

1f57ac1

a few more bits

eec4e1c

jalvz marked this pull request as draft November 20, 2020 14:47

jalvz mentioned this pull request Nov 23, 2020

Integrate with Elastic Agent #4004

Closed

15 tasks

jalvz added 3 commits November 23, 2020 12:57

add base fields, sort fields, and unexpose types

edfdb79

add Makefile target and update readme

2786040

make fmt

6eb420e

jalvz requested review from simitt and axw November 23, 2020 14:43

jalvz marked this pull request as ready for review November 23, 2020 14:43

jalvz added 3 commits November 23, 2020 15:44

Merge branch 'master' into generate-package-fields

f02cd3b

fix lint

7fdb635

Merge branch 'generate-package-fields' of github.com:jalvz/apm-server…

efb1d73

… into generate-package-fields

axw requested changes Nov 24, 2020

View reviewed changes

axw mentioned this pull request Nov 24, 2020

make fmt breaks tests/system code #4454

Closed

jalvz added 2 commits November 24, 2020 11:20

code review

800b6f0

Merge remote-tracking branch 'elastic/master' into generate-package-f…

d3d23d2

…ields

update docs to something decent

2411077

axw approved these changes Nov 25, 2020

View reviewed changes

apmpackage/cmd/gen-package/genfields.go Outdated Show resolved Hide resolved

apmpackage/cmd/gen-package/genfields.go Outdated Show resolved Hide resolved

jalvz and others added 6 commits November 25, 2020 08:40

Update apmpackage/cmd/gen-package/genfields.go

1223ff2

Co-authored-by: Andrew Wilkins <[email protected]>

make update

68ecd29

Update apmpackage/cmd/gen-package/genfields.go

135d27d

Co-authored-by: Andrew Wilkins <[email protected]>

add handlebars template

b2b2c82

Merge branch 'generate-package-fields' of github.com:jalvz/apm-server…

099c3bb

… into generate-package-fields

Merge branch 'master' into generate-package-fields

4d29307

jalvz merged commit 576186d into elastic:master Nov 25, 2020

jalvz added a commit to jalvz/apm-server that referenced this pull request Dec 14, 2020

Generate fields for APM package (elastic#4432)

498b2af

Adds an APM package for Fleet, with a script to generate fields.yml files and docs # Conflicts: # go.mod # include/fields.go

jalvz mentioned this pull request Dec 14, 2020

[7.x] Generate fields for APM package (#4432) #4532

Merged

axw pushed a commit that referenced this pull request Dec 15, 2020

Generate fields for APM package (#4432) (#4532)

73d40c6

Adds an APM package for Fleet, with a script to generate fields.yml files and docs

simitt added test-plan v7.11.0 labels Dec 16, 2020

simitt added test-plan-skip and removed test-plan labels Dec 23, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generate fields for APM package #4432

Generate fields for APM package #4432

jalvz commented Nov 17, 2020 •

edited

Loading

apmmachine commented Nov 17, 2020 •

edited

Loading

Build stats

Test stats 🧪

`Compress`

`Compress`

`Test Sync`

simitt left a comment

jalvz commented Nov 18, 2020

axw Nov 24, 2020

jalvz Nov 24, 2020

axw commented Nov 24, 2020

jalvz commented Nov 24, 2020 •

edited

Loading

axw commented Nov 24, 2020

codecov-io commented Nov 24, 2020 •

edited

Loading

jalvz commented Nov 24, 2020

axw left a comment

simitt commented Dec 16, 2020

simitt commented Dec 23, 2020

Generate fields for APM package #4432

Generate fields for APM package #4432

Conversation

jalvz commented Nov 17, 2020 • edited Loading

Motivation/summary

Checklist

How to test these changes

Related issues

apmmachine commented Nov 17, 2020 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Steps errors

Compress

Compress

Test Sync

simitt left a comment

Choose a reason for hiding this comment

jalvz commented Nov 18, 2020

axw Nov 24, 2020

Choose a reason for hiding this comment

jalvz Nov 24, 2020

Choose a reason for hiding this comment

axw commented Nov 24, 2020

jalvz commented Nov 24, 2020 • edited Loading

axw commented Nov 24, 2020

codecov-io commented Nov 24, 2020 • edited Loading

Codecov Report

jalvz commented Nov 24, 2020

axw left a comment

Choose a reason for hiding this comment

simitt commented Dec 16, 2020

simitt commented Dec 23, 2020

jalvz commented Nov 17, 2020 •

edited

Loading

apmmachine commented Nov 17, 2020 •

edited

Loading

`Compress`

`Compress`

`Test Sync`

jalvz commented Nov 24, 2020 •

edited

Loading

codecov-io commented Nov 24, 2020 •

edited

Loading