You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In the past, agent teams measured performance using the OATS system, which required a lot of hands-on attention to operate. Our agent repo contains benchmark tests that are runnable locally, and which could be easily incorporated in a regular workflow.
npm run bench runs the existing benchmark test suite. Currently some fail due to changes in the agent or changes to agent dependencies. Results are sent to stdout.
Anticipated related tickets:
Spike(s) to determine which tests are being run: which pass, which fail, which to keep, which failing ones to fix, which to remove, and which new tests to add.
Alter benchmark tests to ship results to New Relic, instead of stdout.
Create a dashboard for presenting bench results correlated to agent version as well as Node runtime
Create workflow for running benchmarks on PRs and nightly.
Acceptance Criteria
Failing benchmark tests should be fixed or removed
There should be a dashboard in NR we can view to graph performance over time, tagged with agent version and Node runtime.
There should be GHA workflows that run a matrix of currently supported Node runtimes on the current agent version.
Running benchmark tests should result in telemetry sent to NR, and that telemetry should be presented in that dashboard.
The text was updated successfully, but these errors were encountered:
In the past, agent teams measured performance using the OATS system, which required a lot of hands-on attention to operate. Our agent repo contains benchmark tests that are runnable locally, and which could be easily incorporated in a regular workflow.
npm run bench
runs the existing benchmark test suite. Currently some fail due to changes in the agent or changes to agent dependencies. Results are sent to stdout.Anticipated related tickets:
Acceptance Criteria
The text was updated successfully, but these errors were encountered: