[ML] API integration tests - stabilize saved objects endpoint tests

[pheyos]

Summary

This PR stabilizes some saved objects endpoint tests by making sure a different order of objects in a response array is still accepted.

Additional details

The validation expect.eql "Checks if the obj sortof equals another" (quote from doc string), which includes a different order of items in an array, but it doesn't work well with nested objects. So in some cases we need to sort the objects in an array before comparison. In other cases it's enough to split response validation and check the values of the keys separately.

Closes #87269

[elasticmachine]

Pinging @elastic/ml-ui (:ml)

[pheyos]

Checking test stability in a flaky test runner job ... No ML failure in 50 runs (2 unrelated test failures in index management and audit log).

[pheyos]

Checking test stability again in a flaky test runner job ... no failure in 50 runs ✔️

[peteharverson]

Latest edits LGTM

[kibanamachine]

💛 Build succeeded, but was flaky

• continuous-integration/kibana-ci/pull-request
• Commit: fe1b747
• Flaky suites:
  • xpack-kibana-ciGroup7

---

Test Failures

Chrome X-Pack UI Plugin Functional Tests.x-pack/test/plugin_functional/test_suites/resolver.Resolver test app when the user is interacting with the node with ID: secondChild when the user hovers over the primary button when the user has clicked the primary button (which selects the node.) should render as expected

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has failed 5 times on tracked branches: https://github.com/elastic/kibana/issues/87425

[00:00:00]       │
[00:00:00]         └-: Resolver test app
[00:00:00]           └-> "before all" hook
[00:00:00]           └-> "before all" hook
[00:00:00]             │ debg navigating to resolverTest url: http://localhost:61221/app/resolverTest
[00:00:00]             │ debg navigate to: http://localhost:61221/app/resolverTest
[00:00:00]             │ debg browser[INFO] http://localhost:61221/login?next=%2Fapp%2FresolverTest%3F_t%3D1609951820441 341 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:00:00]             │
[00:00:00]             │ debg browser[INFO] http://localhost:61221/bootstrap.js 42:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:00:00]             │ debg ... sleep(700) start
[00:00:00]             │ debg ... sleep(700) end
[00:00:00]             │ debg returned from get, calling refresh
[00:00:01]             │ debg browser[INFO] http://localhost:61221/login?next=%2Fapp%2FresolverTest%3F_t%3D1609951820441 341 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:00:01]             │
[00:00:01]             │ debg browser[INFO] http://localhost:61221/bootstrap.js 42:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:00:01]             │ debg currentUrl = http://localhost:61221/login?next=%2Fapp%2FresolverTest%3F_t%3D1609951820441
[00:00:01]             │          appUrl = http://localhost:61221/app/resolverTest
[00:00:01]             │ debg TestSubjects.find(kibanaChrome)
[00:00:01]             │ debg Find.findByCssSelector('[data-test-subj="kibanaChrome"]') with timeout=60000
[00:00:03]             │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-18-tests-xxl-1609947208161604071] [.ds-ilm-history-5-2021.01.06-000001] creating index, cause [initialize_data_stream], templates [ilm-history], shards [1]/[0]
[00:00:03]             │ info [o.e.c.m.MetadataCreateDataStreamService] [kibana-ci-immutable-ubuntu-18-tests-xxl-1609947208161604071] adding data stream [ilm-history-5] with write index [.ds-ilm-history-5-2021.01.06-000001] and backing indices []
[00:00:03]             │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xxl-1609947208161604071] moving index [.ds-ilm-history-5-2021.01.06-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [ilm-history-ilm-policy]
[00:00:03]             │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-ubuntu-18-tests-xxl-1609947208161604071] current.health="GREEN" message="Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.ds-ilm-history-5-2021.01.06-000001][0]]])." previous.health="YELLOW" reason="shards started [[.ds-ilm-history-5-2021.01.06-000001][0]]"
[00:00:03]             │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xxl-1609947208161604071] moving index [.ds-ilm-history-5-2021.01.06-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] in policy [ilm-history-ilm-policy]
[00:00:03]             │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-ubuntu-18-tests-xxl-1609947208161604071] moving index [.ds-ilm-history-5-2021.01.06-000001] from [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-follow-shard-tasks"}] in policy [ilm-history-ilm-policy]
[00:00:03]             │ debg Found login page
[00:00:03]             │ debg TestSubjects.setValue(loginUsername, test_user)
[00:00:03]             │ debg TestSubjects.click(loginUsername)
[00:00:03]             │ debg Find.clickByCssSelector('[data-test-subj="loginUsername"]') with timeout=10000
[00:00:03]             │ debg Find.findByCssSelector('[data-test-subj="loginUsername"]') with timeout=10000
[00:00:03]             │ debg TestSubjects.setValue(loginPassword, changeme)
[00:00:03]             │ debg TestSubjects.click(loginPassword)
[00:00:03]             │ debg Find.clickByCssSelector('[data-test-subj="loginPassword"]') with timeout=10000
[00:00:03]             │ debg Find.findByCssSelector('[data-test-subj="loginPassword"]') with timeout=10000
[00:00:03]             │ debg TestSubjects.click(loginSubmit)
[00:00:03]             │ debg Find.clickByCssSelector('[data-test-subj="loginSubmit"]') with timeout=10000
[00:00:03]             │ debg Find.findByCssSelector('[data-test-subj="loginSubmit"]') with timeout=10000
[00:00:03]             │ debg Find.waitForDeletedByCssSelector('.kibanaWelcomeLogo') with timeout=10000
[00:00:03]             │ proc [kibana]   log   [16:50:24.388] [info][plugins][routes][security] Logging in with provider "basic" (basic)
[00:00:04]             │ debg Find.findByCssSelector('[data-test-subj="kibanaChrome"]') with timeout=60000
[00:00:04]             │ debg Find.findByCssSelector('[data-test-subj="kibanaChrome"] nav:not(.ng-hide)') with timeout=60000
[00:00:07]             │ debg browser[INFO] http://localhost:61221/app/resolverTest?_t=1609951820441 341 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:00:07]             │
[00:00:07]             │ debg browser[INFO] http://localhost:61221/bootstrap.js 42:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:00:07]             │ debg browser[INFO] http://localhost:61221/app/resolverTest?_t=1609951827471 341 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:00:07]             │
[00:00:07]             │ debg browser[INFO] http://localhost:61221/bootstrap.js 42:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:00:08]             │ debg Finished login process currentUrl = http://localhost:61221/app/resolverTest
[00:00:08]             │ debg ... sleep(501) start
[00:00:08]             │ debg ... sleep(501) end
[00:00:08]             │ debg in navigateTo url = http://localhost:61221/app/resolverTest
[00:00:08]             │ debg TestSubjects.exists(statusPageContainer)
[00:00:08]             │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="statusPageContainer"]') with timeout=2500
[00:00:11]             │ debg --- retry.tryForTime error: [data-test-subj="statusPageContainer"] is not displayed
[00:00:11]             │ debg ======browser======== setWindowSize 3840 2400
[00:00:12]             │ debg ======browser======== actual initial screenshot size width=1200, height=800
[00:00:12]             │ debg ======browser======== actual second screenshot size width= 600, height=400
[00:00:12]             │ debg ======browser======== calculated values xBorder= 0, yBorder=0, xScaling=1, yScaling=1
[00:00:12]             │ debg ======browser======== setting browser size to 3840 x 2400
[00:00:14]             │ debg ======browser======== final screenshot size width=3840, height=2400
[00:00:14]           └-> renders at least one node
[00:00:14]             └-> "before each" hook: global before each
[00:00:14]             │ debg TestSubjects.exists(resolver:node)
[00:00:14]             │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="resolver:node"]') with timeout=120000
[00:00:14]             └- ✓ pass  (74ms) "Resolver test app renders at least one node"
[00:00:14]           └-> renders a node list
[00:00:14]             └-> "before each" hook: global before each
[00:00:14]             │ debg TestSubjects.exists(resolver:node-list)
[00:00:14]             │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="resolver:node-list"]') with timeout=120000
[00:00:14]             └- ✓ pass  (35ms) "Resolver test app renders a node list"
[00:00:14]           └-> renders at least one edge line
[00:00:14]             └-> "before each" hook: global before each
[00:00:14]             │ debg TestSubjects.exists(resolver:graph:edgeline)
[00:00:14]             │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="resolver:graph:edgeline"]') with timeout=120000
[00:00:14]             └- ✓ pass  (81ms) "Resolver test app renders at least one edge line"
[00:00:14]           └-> renders graph controls
[00:00:14]             └-> "before each" hook: global before each
[00:00:14]             │ debg TestSubjects.exists(resolver:graph-controls)
[00:00:14]             │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="resolver:graph-controls"]') with timeout=120000
[00:00:14]             └- ✓ pass  (29ms) "Resolver test app renders graph controls"
[00:00:14]           └-: when the user is interacting with the node with ID: origin
[00:00:14]             └-> "before all" hook
[00:00:41]           └-: when the user is interacting with the node with ID: secondChild
[00:00:41]             └-> "before all" hook
[00:00:41]             └-> should render as expected
[00:00:41]               └-> "before each" hook: global before each
[00:00:41]               └-> "before each" hook
[00:00:41]               │ debg Find.findByCssSelector('[data-test-resolver-node-id="secondChild"]') with timeout=10000
[00:00:41]               │ debg compareAgainstBaseline
[00:00:41]               │ info Taking screenshot "/dev/shm/workspace/parallel/22/kibana/x-pack/test/plugin_functional/screenshots/session/second_child.png"
[00:00:42]               │ debg comparePngs: /dev/shm/workspace/parallel/22/kibana/x-pack/test/plugin_functional/screenshots/session/second_child.png vs /dev/shm/workspace/parallel/22/kibana/x-pack/test/plugin_functional/screenshots/baseline/second_child.png
[00:00:42]               │ debg calculating diff pixels...
[00:00:42]               │ debg percent different: 0.04694444444444444
[00:00:42]               └- ✓ pass  (1.3s) "Resolver test app when the user is interacting with the node with ID: secondChild should render as expected"
[00:00:42]             └-: when the user hovers over the primary button
[00:00:42]               └-> "before all" hook
[00:00:42]               └-> should render as expected
[00:00:42]                 └-> "before each" hook: global before each
[00:00:42]                 └-> "before each" hook
[00:00:42]                 └-> "before each" hook
[00:00:42]                   │ debg Find.findByCssSelector('[data-test-resolver-node-id="secondChild"]') with timeout=10000
[00:00:42]                 │ debg Find.findByCssSelector('[data-test-resolver-node-id="secondChild"]') with timeout=10000
[00:00:42]                 │ debg compareAgainstBaseline
[00:00:42]                 │ info Taking screenshot "/dev/shm/workspace/parallel/22/kibana/x-pack/test/plugin_functional/screenshots/session/second_child_with_primary_button_hovered.png"
[00:00:43]                 │ debg comparePngs: /dev/shm/workspace/parallel/22/kibana/x-pack/test/plugin_functional/screenshots/session/second_child_with_primary_button_hovered.png vs /dev/shm/workspace/parallel/22/kibana/x-pack/test/plugin_functional/screenshots/baseline/second_child_with_primary_button_hovered.png
[00:00:43]                 │ debg calculating diff pixels...
[00:00:43]                 │ debg percent different: 0.05201388888888889
[00:00:44]                 └- ✓ pass  (1.4s) "Resolver test app when the user is interacting with the node with ID: secondChild when the user hovers over the primary button should render as expected"
[00:00:44]               └-: when the user has clicked the primary button (which selects the node.)
[00:00:44]                 └-> "before all" hook
[00:00:44]                 └-> should render as expected
[00:00:44]                   └-> "before each" hook: global before each
[00:00:44]                   └-> "before each" hook
[00:00:44]                   └-> "before each" hook
[00:00:44]                     │ debg Find.findByCssSelector('[data-test-resolver-node-id="secondChild"]') with timeout=10000
[00:00:44]                   └-> "before each" hook
[00:00:45]                   │ debg Find.findByCssSelector('[data-test-resolver-node-id="secondChild"]') with timeout=10000
[00:00:45]                   │ debg compareAgainstBaseline
[00:00:45]                   │ info Taking screenshot "/dev/shm/workspace/parallel/22/kibana/x-pack/test/plugin_functional/screenshots/session/second_child_selected_with_primary_button_hovered.png"
[00:00:46]                   │ debg comparePngs: /dev/shm/workspace/parallel/22/kibana/x-pack/test/plugin_functional/screenshots/session/second_child_selected_with_primary_button_hovered.png vs /dev/shm/workspace/parallel/22/kibana/x-pack/test/plugin_functional/screenshots/baseline/second_child_selected_with_primary_button_hovered.png
[00:00:46]                   │ debg calculating diff pixels...
[00:00:46]                   │ debg percent different: 0.09585648148148149
[00:00:46]                   │ info Taking screenshot "/dev/shm/workspace/parallel/22/kibana/x-pack/test/plugin_functional/screenshots/failure/Resolver test app when the user is interacting with the node with ID_ secondChild when the user hovers over the primary button when the user has clicked the primary button _which selects the node_ should render as expected.png"
[00:00:47]                   │ info Current URL is: http://localhost:61221/app/resolverTest?resolver-test=(panelParameters%3A(nodeID%3AsecondChild)%2CpanelView%3AnodeDetail)
[00:00:47]                   │ info Saving page source to: /dev/shm/workspace/parallel/22/kibana/x-pack/test/plugin_functional/failure_debug/html/Resolver test app when the user is interacting with the node with ID_ secondChild when the user hovers over the primary button when the user has clicked the primary button _which selects the node_ should render as expected.html
[00:00:47]                   └- ✖ fail: Resolver test app when the user is interacting with the node with ID: secondChild when the user hovers over the primary button when the user has clicked the primary button (which selects the node.) should render as expected
[00:00:47]                   │      Error: expected 0.09585648148148149 to be below 0.09
[00:00:47]                   │       at Assertion.assert (/dev/shm/workspace/parallel/22/kibana/packages/kbn-expect/expect.js:100:11)
[00:00:47]                   │       at Assertion.lessThan.Assertion.below (/dev/shm/workspace/parallel/22/kibana/packages/kbn-expect/expect.js:336:8)
[00:00:47]                   │       at Function.lessThan (/dev/shm/workspace/parallel/22/kibana/packages/kbn-expect/expect.js:531:15)
[00:00:47]                   │       at Context.<anonymous> (test/plugin_functional/test_suites/resolver/index.ts:119:23)
[00:00:47]                   │       at Object.apply (/dev/shm/workspace/parallel/22/kibana/packages/kbn-test/src/functional_test_runner/lib/mocha/wrap_function.js:84:16)
[00:00:47]                   │ 
[00:00:47]                   │ 

Stack Trace

Error: expected 0.09585648148148149 to be below 0.09
    at Assertion.assert (/dev/shm/workspace/parallel/22/kibana/packages/kbn-expect/expect.js:100:11)
    at Assertion.lessThan.Assertion.below (/dev/shm/workspace/parallel/22/kibana/packages/kbn-expect/expect.js:336:8)
    at Function.lessThan (/dev/shm/workspace/parallel/22/kibana/packages/kbn-expect/expect.js:531:15)
    at Context.<anonymous> (test/plugin_functional/test_suites/resolver/index.ts:119:23)
    at Object.apply (/dev/shm/workspace/parallel/22/kibana/packages/kbn-test/src/functional_test_runner/lib/mocha/wrap_function.js:84:16)

---

Metrics [docs]

✅ unchanged

History

• 💔 Build #97270 failed f64de3d
• 💔 Build #97223 failed 86a695d
• 💚 Build #96909 succeeded 0d637e9

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream