Failing test: X-Pack API Integration Tests.x-pack/test/api_integration/apis/beats/assign_tags_to_beats·js - apis beats assign_tags_to_beats should return errors for non-existent beats

[kibanamachine]

A test failed on a tracked branch

        { Error: expected [ { status: 200, result: 'updated' } ] to sort of equal [ { status: 404, result: 'Beat bar not found' } ]
    at Assertion.assert (/var/lib/jenkins/workspace/elastic+kibana+7.x/JOB/x-pack-ciGroup5/node/immutable/kibana/packages/kbn-expect/expect.js:100:11)
    at Assertion.eql (/var/lib/jenkins/workspace/elastic+kibana+7.x/JOB/x-pack-ciGroup5/node/immutable/kibana/packages/kbn-expect/expect.js:235:8)
    at Context.eql (test/api_integration/apis/beats/assign_tags_to_beats.js:193:42)
  actual: '[\n  {\n    "result": "updated"\n    "status": 200\n  }\n]',
  expected:
   '[\n  {\n    "result": "Beat bar not found"\n    "status": 404\n  }\n]',
  showDiff: true }
      

First failure: Jenkins Build

[elasticmachine]

Pinging @elastic/kibana-test-triage

[elasticmachine]

Pinging @elastic/beats

[kibanamachine]

New failure: Jenkins Build

Branch: 7.0

00:20:06.344              └-> should return errors for non-existent beats
00:20:06.344                └-> "before each" hook: global before each
00:20:06.345                └-> "before each" hook: cleanup
00:20:06.387                └-> "before each" hook: load beats archive
00:20:07.462                └- ✖ fail: "apis beats assign_tags_to_beats should return errors for non-existent beats"
00:20:07.462                │ 
00:20:07.463                │       Error: expected [ { status: 200, result: 'updated' } ] to sort of equal [ { status: 404, result: 'Beat baz not found' } ]
00:20:07.464                │       + expected - actual
00:20:07.465                │ 
00:20:07.465                │        [
00:20:07.465                │          {
00:20:07.465                │       -    "result": "updated"
00:20:07.466                │       -    "status": 200
00:20:07.466                │       +    "result": "Beat baz not found"
00:20:07.467                │       +    "status": 404
00:20:07.467                │          }
00:20:07.468                │        ]
00:20:07.468                │       
00:20:07.468                │       at Assertion.assert (/var/lib/jenkins/workspace/elastic+kibana+7.0/JOB/x-pack-ciGroup5/node/immutable/kibana/packages/kbn-expect/expect.js:100:11)
00:20:07.469                │       at Assertion.eql (/var/lib/jenkins/workspace/elastic+kibana+7.0/JOB/x-pack-ciGroup5/node/immutable/kibana/packages/kbn-expect/expect.js:235:8)
00:20:07.470                │       at Context.it (test/api_integration/apis/beats/assign_tags_to_beats.js:193:42)
00:20:07.470                │ 
00:20:07.471                │ 

[kibanamachine]

New failure: Jenkins Build

[00:03:26]                 └- ✖ fail: "apis beats assign_tags_to_beats should return errors for non-existent beats"
[00:03:26]                 │ 
[00:03:26]                 │       Error: expected [ { success: true, result: { message: 'updated' } } ] to sort of equal [ { success: false,
[00:03:26]                 │     error: { code: 404, message: 'Beat bar not found' } } ]
[00:03:26]                 │       + expected - actual
[00:03:26]                 │ 
[00:03:26]                 │        [
[00:03:26]                 │          {
[00:03:26]                 │       -    "result": {
[00:03:26]                 │       -      "message": "updated"
[00:03:26]                 │       +    "error": {
[00:03:26]                 │       +      "code": 404
[00:03:26]                 │       +      "message": "Beat bar not found"
[00:03:26]                 │            }
[00:03:26]                 │       -    "success": true
[00:03:26]                 │       +    "success": false
[00:03:26]                 │          }
[00:03:26]                 │        ]
[00:03:26]                 │       
[00:03:26]                 │       at Assertion.assert (/var/lib/jenkins/workspace/elastic+kibana+master/JOB/x-pack-ciGroup6/node/immutable/kibana/packages/kbn-expect/expect.js:100:11)
[00:03:26]                 │       at Assertion.eql (/var/lib/jenkins/workspace/elastic+kibana+master/JOB/x-pack-ciGroup6/node/immutable/kibana/packages/kbn-expect/expect.js:235:8)
[00:03:26]                 │       at Context.eql (test/api_integration/apis/beats/assign_tags_to_beats.js:193:38)
[00:03:26]                 │ 

[spalger]

Skipped

master: 02dbf2d
7.x/7.2: 0d62f70
7.1: 6f99ebd
6.8: 1c37770

[mattapperson]

@spalger I would be surprised if skipping that test fixes anything. Look at the expect in the test and the failed test output... they don’t match up. Something is going sideways with mocha

[spalger]

The assertion: https://github.com/elastic/kibana/blob/master/x-pack/test/api_integration/apis/beats/assign_tags_to_beats.js#L194-L196

expect(apiResponse.results).to.eql([
  { success: false, error: { code: 404, message: `Beat ${nonExistentBeatId} not found` } },
]);

The error, reformatted:

expected
  [
    {
      success: true,
      result: { message: 'updated' }
    }
  ]
  to sort of equal
  [
    {
      success: false,
      error: {
        code: 404,
        message: 'Beat bar not found'
      }
    }
  ]

Looks right to me

[spalger]

I assume that since the test is using chance.word() it's just randomly choosing bar and baz sometimes which maybe actually exist? If you switch to chance.word({ length: 20 }) or something that should avoid the possibility.