Remove package transform #5729

axw · 2021-07-15T06:13:29Z

Motivation/summary

Remove package transform, which is no longer needed. Transformation to beat.Events longer requires parameterisation, with all customisable transformations moving to model processing. We could probably also remove the context parameter to the transformable interface, but I've left it there for now as it's not actively causing problems.

Transformation should now be more or less instant, and errors cannot occur, so it is no longer useful to instrument these operations.

Checklist

~~- [ ] Update CHANGELOG.asciidoc~~
~~- [ ] Documentation has been updated~~

How to test these changes

The only user-visible change is that spans are no longer produced for event transformation. This can be observed by enabling instrumentation (-E instrumentation.enabled=true), sending some events to the server, and then checking that no "Transform" spans are produced.

Related issues

#3565

This is no longer needed. Transformation to beat.Events longer requires parameterisation, with all customisable transformations moving to model processing.

Transformation should now be more or less instant, and errors cannot occur, so it is no longer useful to trace these operations.

apmmachine · 2021-07-15T06:53:14Z

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-07-15T06:13:43.416+0000
Duration: 39 min 22 sec
Commit: d4af5c4

Test stats 🧪

Test	Results
Failed	1
Passed	5932
Skipped	14
Total	5947

Trends 🧪

Test errors

Expand to view the tests failures

`Build and Test / APM Integration Tests / test_concurrent_req_dotnet_foobar – tests.agent.test_dotnet`

Expand to view the error details

 AssertionError: queried for [('processor.event', 'transaction'), ('service.name', ['dotnetapp', 'dotnetapp'])], expected 1375, got 1374

Expand to view the stacktrace

 dotnet = <tests.fixtures.agents.Agent object at 0x7f6ce326bd50>

    @pytest.mark.version
    @pytest.mark.dotnet
    def test_concurrent_req_dotnet_foobar(dotnet):
        foo = Concurrent.Endpoint(dotnet.foo.url,
                                  dotnet.app_name,
                                  ["foo"],
                                  "GET /foo",
                                  events_no=375)
        bar = Concurrent.Endpoint(dotnet.bar.url,
                                  dotnet.app_name,
                                  ["bar", "extra"],
                                  "GET /bar")
>       Concurrent(dotnet.apm_server.elasticsearch, [foo, bar], iters=1).run()

tests/agent/test_dotnet.py:43: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/agent/concurrent_requests.py:254: in run
    self.check_counts(it)
tests/agent/concurrent_requests.py:138: in check_counts
    assert_count([("processor.event", "transaction"), ("service.name", service_names)], transactions_count)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

terms = [('processor.event', 'transaction'), ('service.name', ['dotnetapp', 'dotnetapp'])]
expected = 1375

    def assert_count(terms, expected):
        """wait a bit for doc count to reach expectation"""
        @timeout_decorator.timeout(max_wait)
        def check_count(mut_actual):
            while True:
                rsp = self.es.count(index=self.index, body=self.elasticsearch.term_q(terms))
                mut_actual[0] = rsp["count"]
                if mut_actual[0] >= expected:
                    return
                time.sleep(backoff)
    
        mut_actual = [-1]  # keep actual count in this mutable
        try:
            check_count(mut_actual)
        except timeout_decorator.TimeoutError:
            pass
        actual = mut_actual[0]
>       assert actual == expected, err.format(terms, expected, actual)
E       AssertionError: queried for [('processor.event', 'transaction'), ('service.name', ['dotnetapp', 'dotnetapp'])], expected 1375, got 1374

tests/agent/concurrent_requests.py:132: AssertionError

Steps errors

Expand to view the steps failures

`Test Sync`

Took 3 min 29 sec . View more details on here
Description: ./.ci/scripts/sync.sh

Log output

Expand to view the last 100 lines of log output

[2021-07-15T06:50:28.617Z] --- PASS: TestAPMServerMonitoring (0.65s)
[2021-07-15T06:50:28.617Z] === RUN   TestAPMServerMonitoringBuiltinUser
[2021-07-15T06:50:28.617Z] --- PASS: TestAPMServerMonitoringBuiltinUser (2.03s)
[2021-07-15T06:50:28.617Z] === RUN   TestAPMServerOnboarding
[2021-07-15T06:50:28.617Z] --- PASS: TestAPMServerOnboarding (3.62s)
[2021-07-15T06:50:28.617Z] === RUN   TestOTLPGRPCTraces
[2021-07-15T06:50:28.617Z] --- PASS: TestOTLPGRPCTraces (4.18s)
[2021-07-15T06:50:28.617Z] === RUN   TestOTLPGRPCMetrics
[2021-07-15T06:50:28.617Z] --- PASS: TestOTLPGRPCMetrics (5.03s)
[2021-07-15T06:50:28.617Z] === RUN   TestOTLPGRPCAuth
[2021-07-15T06:50:28.617Z] --- PASS: TestOTLPGRPCAuth (5.40s)
[2021-07-15T06:50:28.617Z] === RUN   TestOTLPClientIP
[2021-07-15T06:50:28.617Z] --- PASS: TestOTLPClientIP (3.72s)
[2021-07-15T06:50:28.617Z] === RUN   TestOTLPAnonymous
[2021-07-15T06:50:28.617Z] --- PASS: TestOTLPAnonymous (0.26s)
[2021-07-15T06:50:28.617Z] === RUN   TestOTLPRateLimit
[2021-07-15T06:50:28.617Z] --- PASS: TestOTLPRateLimit (0.27s)
[2021-07-15T06:50:28.617Z] === RUN   TestRUMXForwardedFor
[2021-07-15T06:50:28.617Z] --- PASS: TestRUMXForwardedFor (5.25s)
[2021-07-15T06:50:28.617Z] === RUN   TestRUMAllowServiceNames
[2021-07-15T06:50:28.617Z] --- PASS: TestRUMAllowServiceNames (0.20s)
[2021-07-15T06:50:28.617Z] === RUN   TestRUMRateLimit
[2021-07-15T06:50:28.617Z] --- PASS: TestRUMRateLimit (0.91s)
[2021-07-15T06:50:28.617Z] === RUN   TestRUMCORS
[2021-07-15T06:50:28.617Z] --- PASS: TestRUMCORS (0.20s)
[2021-07-15T06:50:28.617Z] === RUN   TestKeepUnsampled
[2021-07-15T06:50:28.617Z] === RUN   TestKeepUnsampled/false
[2021-07-15T06:50:28.617Z] === RUN   TestKeepUnsampled/true
[2021-07-15T06:50:28.617Z] --- PASS: TestKeepUnsampled (7.67s)
[2021-07-15T06:50:28.617Z]     --- PASS: TestKeepUnsampled/false (3.50s)
[2021-07-15T06:50:28.617Z]     --- PASS: TestKeepUnsampled/true (4.17s)
[2021-07-15T06:50:28.617Z] === RUN   TestKeepUnsampledWarning
[2021-07-15T06:50:28.617Z] --- PASS: TestKeepUnsampledWarning (4.10s)
[2021-07-15T06:50:28.617Z] === RUN   TestTailSampling
[2021-07-15T06:50:28.617Z]     sampling_test.go:135: waiting for 100 "parent" transactions
[2021-07-15T06:50:28.617Z]     sampling_test.go:135: waiting for 100 "child" transactions
[2021-07-15T06:50:28.617Z] --- PASS: TestTailSampling (5.42s)
[2021-07-15T06:50:28.617Z] === RUN   TestTailSamplingUnlicensed
[2021-07-15T06:50:28.617Z] 2021/07/15 06:49:23 Starting container id: 0248285ccd12 image: docker.elastic.co/elasticsearch/elasticsearch:8.0.0-52a5431f-SNAPSHOT
[2021-07-15T06:50:28.617Z] 2021/07/15 06:49:23 Waiting for container id 0248285ccd12 image: docker.elastic.co/elasticsearch/elasticsearch:8.0.0-52a5431f-SNAPSHOT
[2021-07-15T06:50:28.617Z] 2021/07/15 06:49:42 Container is ready id: 0248285ccd12 image: docker.elastic.co/elasticsearch/elasticsearch:8.0.0-52a5431f-SNAPSHOT
[2021-07-15T06:50:28.617Z] --- PASS: TestTailSamplingUnlicensed (28.87s)
[2021-07-15T06:50:28.617Z] === RUN   TestRUMErrorSourcemapping
[2021-07-15T06:50:28.617Z] --- PASS: TestRUMErrorSourcemapping (4.39s)
[2021-07-15T06:50:28.617Z] === RUN   TestRUMSpanSourcemapping
[2021-07-15T06:50:28.617Z] --- PASS: TestRUMSpanSourcemapping (4.73s)
[2021-07-15T06:50:28.617Z] === RUN   TestDuplicateSourcemapWarning
[2021-07-15T06:50:28.617Z] --- PASS: TestDuplicateSourcemapWarning (4.92s)
[2021-07-15T06:50:28.617Z] === RUN   TestNoMatchingSourcemap
[2021-07-15T06:50:28.617Z] --- PASS: TestNoMatchingSourcemap (4.53s)
[2021-07-15T06:50:28.617Z] === RUN   TestFetchLatestSourcemap
[2021-07-15T06:50:28.617Z] --- PASS: TestFetchLatestSourcemap (5.93s)
[2021-07-15T06:50:28.617Z] === RUN   TestSourcemapCaching
[2021-07-15T06:50:28.617Z] --- PASS: TestSourcemapCaching (5.15s)
[2021-07-15T06:50:28.617Z] === RUN   TestIndexTemplateCoverage
[2021-07-15T06:50:28.617Z] --- PASS: TestIndexTemplateCoverage (5.81s)
[2021-07-15T06:50:28.617Z] PASS
[2021-07-15T06:50:28.617Z] ok  	github.com/elastic/apm-server/systemtest	415.745s
[2021-07-15T06:50:28.617Z] === RUN   TestAPMServer
[2021-07-15T06:50:28.617Z] 2021/07/15 06:43:29 Building apm-server...
[2021-07-15T06:50:28.617Z] 2021/07/15 06:43:32 Built /var/lib/jenkins/workspace/pm-server_apm-server-mbp_PR-5729/src/github.com/elastic/apm-server/apm-server
[2021-07-15T06:50:28.617Z] --- PASS: TestAPMServer (2.50s)
[2021-07-15T06:50:28.617Z] === RUN   TestUnstartedAPMServer
[2021-07-15T06:50:28.617Z] --- PASS: TestUnstartedAPMServer (0.00s)
[2021-07-15T06:50:28.617Z] === RUN   TestAPMServerStartTLS
[2021-07-15T06:50:28.617Z] --- PASS: TestAPMServerStartTLS (0.10s)
[2021-07-15T06:50:28.617Z] === RUN   TestExpvar
[2021-07-15T06:50:28.617Z] --- PASS: TestExpvar (0.10s)
[2021-07-15T06:50:28.617Z] PASS
[2021-07-15T06:50:28.617Z] ok  	github.com/elastic/apm-server/systemtest/apmservertest	2.732s
[2021-07-15T06:50:28.617Z] ?   	github.com/elastic/apm-server/systemtest/benchtest	[no test files]
[2021-07-15T06:50:28.617Z] ?   	github.com/elastic/apm-server/systemtest/cmd/apmbench	[no test files]
[2021-07-15T06:50:28.617Z] ?   	github.com/elastic/apm-server/systemtest/estest	[no test files]
[2021-07-15T06:50:28.617Z] ?   	github.com/elastic/apm-server/systemtest/fleettest	[no test files]
[2021-07-15T06:50:28.617Z] + cleanup
[2021-07-15T06:50:28.617Z] + rm -rf /tmp/tmp.iLTuUb4k9F
[2021-07-15T06:50:28.617Z] + .ci/scripts/docker-get-logs.sh
[2021-07-15T06:50:29.196Z] It is not possible to grab the logs of 96df093123f0
[2021-07-15T06:50:30.390Z] Post stage
[2021-07-15T06:50:30.415Z] Running in /var/lib/jenkins/workspace/pm-server_apm-server-mbp_PR-5729/src/github.com/elastic/apm-server/build
[2021-07-15T06:50:30.476Z] Archiving artifacts
[2021-07-15T06:50:30.906Z] Recording test results
[2021-07-15T06:50:31.440Z] [Checks API] No suitable checks publisher found.
[2021-07-15T06:50:31.843Z] + tar --version
[2021-07-15T06:50:32.217Z] + tar --exclude=system-tests-linux-files.tgz -czf system-tests-linux-files.tgz system-tests
[2021-07-15T06:50:32.510Z] Archiving artifacts
[2021-07-15T06:50:33.035Z] Terminated
[2021-07-15T06:50:33.022Z] Terminated
[2021-07-15T06:52:58.149Z] [INFO] For detailed information see: https://apm-ci.elastic.co/job/apm-integration-tests-selector-mbp/job/master/18777/display/redirect
[2021-07-15T06:53:01.243Z] Copied 27 artifacts from "APM Integration Test MBP Selector » master" build number 18777
[2021-07-15T06:53:02.691Z] Post stage
[2021-07-15T06:53:02.712Z] Recording test results
[2021-07-15T06:53:03.480Z] [Checks API] No suitable checks publisher found.
[2021-07-15T06:53:04.372Z] Running on Jenkins in /var/lib/jenkins/workspace/pm-server_apm-server-mbp_PR-5729
[2021-07-15T06:53:04.512Z] [INFO] getVaultSecret: Getting secrets
[2021-07-15T06:53:04.577Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-07-15T06:53:05.653Z] + chmod 755 generate-build-data.sh
[2021-07-15T06:53:05.653Z] + ./generate-build-data.sh https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-server/apm-server-mbp/PR-5729/ https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-server/apm-server-mbp/PR-5729/runs/1 UNSTABLE 2361959
[2021-07-15T06:53:05.653Z] INFO: curl https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-server/apm-server-mbp/PR-5729/runs/1/steps/?limit=10000 -o steps-info.json
[2021-07-15T06:53:05.904Z] INFO: curl https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-server/apm-server-mbp/PR-5729/runs/1/tests/?status=FAILED -o tests-errors.json

simitt

🎉

* Remove package transform This is no longer needed. Transformation to beat.Events longer requires parameterisation, with all customisable transformations moving to model processing. * publish: don't create spans for transformation Transformation should now be more or less instant, and errors cannot occur, so it is no longer useful to trace these operations. (cherry picked from commit e63fd60)

* Remove package transform This is no longer needed. Transformation to beat.Events longer requires parameterisation, with all customisable transformations moving to model processing. * publish: don't create spans for transformation Transformation should now be more or less instant, and errors cannot occur, so it is no longer useful to trace these operations. (cherry picked from commit e63fd60) Co-authored-by: Andrew Wilkins <[email protected]>

axw added 2 commits July 15, 2021 14:04

Remove package transform

5fc9725

This is no longer needed. Transformation to beat.Events longer requires parameterisation, with all customisable transformations moving to model processing.

publish: don't create spans for transformation

d4af5c4

Transformation should now be more or less instant, and errors cannot occur, so it is no longer useful to trace these operations.

axw added v7.15.0 backport-7.x labels Jul 15, 2021

axw marked this pull request as ready for review July 15, 2021 06:13

axw requested a review from a team July 15, 2021 06:13

simitt approved these changes Jul 16, 2021

View reviewed changes

axw merged commit e63fd60 into elastic:master Jul 16, 2021

axw deleted the remove-transform branch July 16, 2021 06:55

mergify bot mentioned this pull request Jul 16, 2021

[7.x] Remove package transform (backport #5729) #5740

Merged

simitt added the test-plan-skip label Aug 23, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove package transform #5729

Remove package transform #5729

axw commented Jul 15, 2021

apmmachine commented Jul 15, 2021

Build stats

Test stats 🧪

Trends 🧪

`Build and Test / APM Integration Tests / test_concurrent_req_dotnet_foobar – tests.agent.test_dotnet`

`Test Sync`

simitt left a comment

Remove package transform #5729

Remove package transform #5729

Conversation

axw commented Jul 15, 2021

Motivation/summary

Checklist

How to test these changes

Related issues

apmmachine commented Jul 15, 2021

💔 Tests Failed

Build stats

Test stats 🧪

Trends 🧪

Test errors

Build and Test / APM Integration Tests / test_concurrent_req_dotnet_foobar – tests.agent.test_dotnet

Steps errors

Test Sync

Log output

simitt left a comment

Choose a reason for hiding this comment

`Build and Test / APM Integration Tests / test_concurrent_req_dotnet_foobar – tests.agent.test_dotnet`

`Test Sync`