Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ci-visibility] Change test.type in browser tests #3358

Merged
merged 8 commits into from
Jul 10, 2023

Conversation

ericlaz
Copy link
Contributor

@ericlaz ericlaz commented Jul 6, 2023

What does this PR do?

Adds a new test type: browser which is only enabled when testing using Cypress or Playwright.

Motivation

It adds more flexibility to test types

Plugin Checklist

@github-actions
Copy link

github-actions bot commented Jul 6, 2023

Overall package size

Self size: 4.79 MB
Deduped: 61.54 MB
No deduping: 61.63 MB

Dependency sizes

name version self size total size
@datadog/pprof 2.2.3 14.25 MB 15.13 MB
@datadog/native-iast-taint-tracking 1.5.0 14.86 MB 14.86 MB
@datadog/native-appsec 3.2.0 13.38 MB 13.39 MB
protobufjs 7.1.2 2.76 MB 6.55 MB
@datadog/native-iast-rewriter 2.0.1 2.09 MB 2.1 MB
@opentelemetry/core 1.14.0 872.87 kB 1.47 MB
@datadog/native-metrics 2.0.0 898.77 kB 1.3 MB
@opentelemetry/api 1.4.1 780.32 kB 780.32 kB
msgpack-lite 0.1.26 201.16 kB 281.59 kB
opentracing 0.14.7 194.81 kB 194.81 kB
semver 7.5.3 93.39 kB 123.79 kB
@datadog/sketches-js 2.1.0 109.9 kB 109.9 kB
lodash.sortby 4.7.0 75.76 kB 75.76 kB
lru-cache 7.14.0 74.95 kB 74.95 kB
ipaddr.js 2.0.1 59.52 kB 59.52 kB
int64-buffer 0.1.10 49.18 kB 49.18 kB
ignore 5.2.0 48.87 kB 48.87 kB
import-in-the-middle 1.3.5 34.34 kB 38.81 kB
istanbul-lib-coverage 3.2.0 29.34 kB 29.34 kB
retry 0.10.1 27.44 kB 27.44 kB
lodash.uniq 4.5.0 25.01 kB 25.01 kB
limiter 1.1.5 23.17 kB 23.17 kB
lodash.kebabcase 4.1.1 17.75 kB 17.75 kB
lodash.pick 4.4.0 16.33 kB 16.33 kB
node-abort-controller 3.0.1 14.33 kB 14.33 kB
crypto-randomuuid 1.0.0 11.18 kB 11.18 kB
diagnostics_channel 1.1.0 7.07 kB 7.07 kB
path-to-regexp 0.1.7 6.78 kB 6.78 kB
koalas 1.0.2 6.47 kB 6.47 kB
methods 1.1.2 5.29 kB 5.29 kB
module-details-from-path 1.0.3 4.47 kB 4.47 kB

🤖 This report was automatically generated by heaviest-objects-in-the-universe

@codecov
Copy link

codecov bot commented Jul 6, 2023

Codecov Report

Merging #3358 (d3f4b18) into master (b12a60d) will decrease coverage by 0.04%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master    #3358      +/-   ##
==========================================
- Coverage   84.18%   84.15%   -0.04%     
==========================================
  Files         205      205              
  Lines        8061     8064       +3     
  Branches       33       33              
==========================================
  Hits         6786     6786              
- Misses       1275     1278       +3     
Impacted Files Coverage Δ
packages/dd-trace/src/plugins/util/test.js 68.45% <0.00%> (-1.41%) ⬇️

... and 1 file with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

Copy link
Collaborator

@juan-fernandez juan-fernandez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good job! Most of the comments are just for moving certain things around, but the logic is good

ericlaz and others added 2 commits July 7, 2023 11:40
Co-authored-by: Juan Antonio Fernández de Alba <[email protected]>
@pr-commenter
Copy link

pr-commenter bot commented Jul 7, 2023

Benchmarks

Benchmark execution time: 2023-07-07 14:48:31

Comparing candidate commit d3f4b18 in PR branch eric-navarro/test-type-browser with baseline commit b12a60d in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 464 metrics, 28 unstable metrics.

Copy link
Collaborator

@juan-fernandez juan-fernandez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we're almost there :-)

additionally, cypress tests are failing because we need to update this https://github.com/DataDog/dd-trace-js/blob/master/packages/datadog-plugin-cypress/test/index.spec.js#L81 and this https://github.com/DataDog/dd-trace-js/blob/master/packages/datadog-plugin-cypress/test/index.spec.js#L106

we could also optionally expand this test integration-tests/playwright/playwright.spec.js to include TEST_TYPE assertion, but that's up to you 😄

ericlaz and others added 2 commits July 7, 2023 14:30
Co-authored-by: Juan Antonio Fernández de Alba <[email protected]>
@juan-fernandez juan-fernandez changed the title Add browser framework type checking [ci-visibility] Change test.type in browser tests Jul 7, 2023
Copy link
Collaborator

@juan-fernandez juan-fernandez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just noticed that we're missing a TEST_TYPE update in https://github.com/DataDog/dd-trace-js/blob/master/packages/dd-trace/src/plugins/util/test.js#L277.

The tests are correctly set but session, modules and suites are still being reported with test.type='test'

@juan-fernandez
Copy link
Collaborator

I just noticed that we're missing a TEST_TYPE update in https://github.com/DataDog/dd-trace-js/blob/master/packages/dd-trace/src/plugins/util/test.js#L277.

The tests are correctly set but session, modules and suites are still being reported with test.type='test'

that's why the integration test is failing: https://github.com/DataDog/dd-trace-js/actions/runs/5487302751/jobs/9998579728?pr=3358. Good thing we added that! 😄

@juan-fernandez juan-fernandez marked this pull request as ready for review July 7, 2023 15:18
@juan-fernandez juan-fernandez requested a review from a team as a code owner July 7, 2023 15:18
@ericlaz ericlaz merged commit 2fed3ce into master Jul 10, 2023
@ericlaz ericlaz deleted the eric-navarro/test-type-browser branch July 10, 2023 08:45
Qard pushed a commit that referenced this pull request Jul 11, 2023
* Add framework type checking

* Improve code readability

Co-authored-by: Juan Antonio Fernández de Alba <[email protected]>

* Fix imports

* Fix styling

Co-authored-by: Juan Antonio Fernández de Alba <[email protected]>

* Add cypress and playwright tests

* Change browser conditions to detect all playwright instances

* Fix include string

* Fix some suites still being reported as test type

---------

Co-authored-by: Juan Antonio Fernández de Alba <[email protected]>
Qard pushed a commit that referenced this pull request Jul 11, 2023
* Add framework type checking

* Improve code readability

Co-authored-by: Juan Antonio Fernández de Alba <[email protected]>

* Fix imports

* Fix styling

Co-authored-by: Juan Antonio Fernández de Alba <[email protected]>

* Add cypress and playwright tests

* Change browser conditions to detect all playwright instances

* Fix include string

* Fix some suites still being reported as test type

---------

Co-authored-by: Juan Antonio Fernández de Alba <[email protected]>
Qard pushed a commit that referenced this pull request Jul 11, 2023
* Add framework type checking

* Improve code readability

Co-authored-by: Juan Antonio Fernández de Alba <[email protected]>

* Fix imports

* Fix styling

Co-authored-by: Juan Antonio Fernández de Alba <[email protected]>

* Add cypress and playwright tests

* Change browser conditions to detect all playwright instances

* Fix include string

* Fix some suites still being reported as test type

---------

Co-authored-by: Juan Antonio Fernández de Alba <[email protected]>
This was referenced Jul 11, 2023
Qard pushed a commit that referenced this pull request Jul 11, 2023
* Add framework type checking

* Improve code readability

Co-authored-by: Juan Antonio Fernández de Alba <[email protected]>

* Fix imports

* Fix styling

Co-authored-by: Juan Antonio Fernández de Alba <[email protected]>

* Add cypress and playwright tests

* Change browser conditions to detect all playwright instances

* Fix include string

* Fix some suites still being reported as test type

---------

Co-authored-by: Juan Antonio Fernández de Alba <[email protected]>
Qard pushed a commit that referenced this pull request Jul 11, 2023
* Add framework type checking

* Improve code readability

Co-authored-by: Juan Antonio Fernández de Alba <[email protected]>

* Fix imports

* Fix styling

Co-authored-by: Juan Antonio Fernández de Alba <[email protected]>

* Add cypress and playwright tests

* Change browser conditions to detect all playwright instances

* Fix include string

* Fix some suites still being reported as test type

---------

Co-authored-by: Juan Antonio Fernández de Alba <[email protected]>
Qard pushed a commit that referenced this pull request Jul 11, 2023
* Add framework type checking

* Improve code readability

Co-authored-by: Juan Antonio Fernández de Alba <[email protected]>

* Fix imports

* Fix styling

Co-authored-by: Juan Antonio Fernández de Alba <[email protected]>

* Add cypress and playwright tests

* Change browser conditions to detect all playwright instances

* Fix include string

* Fix some suites still being reported as test type

---------

Co-authored-by: Juan Antonio Fernández de Alba <[email protected]>
Qard pushed a commit that referenced this pull request Jul 14, 2023
* Add framework type checking

* Improve code readability

Co-authored-by: Juan Antonio Fernández de Alba <[email protected]>

* Fix imports

* Fix styling

Co-authored-by: Juan Antonio Fernández de Alba <[email protected]>

* Add cypress and playwright tests

* Change browser conditions to detect all playwright instances

* Fix include string

* Fix some suites still being reported as test type

---------

Co-authored-by: Juan Antonio Fernández de Alba <[email protected]>
Qard pushed a commit that referenced this pull request Jul 14, 2023
* Add framework type checking

* Improve code readability

Co-authored-by: Juan Antonio Fernández de Alba <[email protected]>

* Fix imports

* Fix styling

Co-authored-by: Juan Antonio Fernández de Alba <[email protected]>

* Add cypress and playwright tests

* Change browser conditions to detect all playwright instances

* Fix include string

* Fix some suites still being reported as test type

---------

Co-authored-by: Juan Antonio Fernández de Alba <[email protected]>
Qard pushed a commit that referenced this pull request Jul 14, 2023
* Add framework type checking

* Improve code readability

Co-authored-by: Juan Antonio Fernández de Alba <[email protected]>

* Fix imports

* Fix styling

Co-authored-by: Juan Antonio Fernández de Alba <[email protected]>

* Add cypress and playwright tests

* Change browser conditions to detect all playwright instances

* Fix include string

* Fix some suites still being reported as test type

---------

Co-authored-by: Juan Antonio Fernández de Alba <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants