[WIP][skip-ci] Retrieve index pattern from ingest manager app #60063

jonathan-buttner · 2020-03-12T20:07:03Z

This PR connects the endpoint and ingest apps. The endpoint app will rely on the ingest manager to retrieve the index pattern for querying elasticsearch. The index pattern is built using the contents of the endpoint package.

One issue I think I'm going to run into is that the ingest manager's setup doesn't happen until the app is navigated too in the UI. The endpoint package needs to be installed by default and be there before the endpoint app can start making queries to elasticsearch. I think the ingest manager will need to do install the default system packages earlier or the endpoint app will have to call the setup route first.

* plugin.start now does reactdom.render vs returning react element export plugin function from public/index * Move setClient call from plugin.start to plugin.setup * Use `useUiSetting$` from `useKibana` hooks

Can't use useKibana outside a React component. Reverting to prior approach since it's still NP. Can revisit context usage in a followup PR

* Support basic "click button -> show spinner -> installed" install flow * Remove incorrect comments. Add TS return types to data functions.

* Use NP feature_catalogue.register * Use type from NP plugin

…to feature-ingest

…ana into feature-ingest

This reverts commit d05f20d.

* Update (all remaining?) references from integrations_manager to EPM * Update path in i18n file

* change min max version to single range value * add elastic stack icon and change text * remove badge from version and use code font * remove euistyled * add back version component lost in merge * remove euiStyled * remove old file

Temporary work around until we know how the stack wants to add the permissions we need. Either adding to the kibana user or creating a new user.

This reverts commit f1020e4.

Temporary work around until we know how the stack wants to add the permissions we need. Either adding to the kibana user or creating a new user.

* Initialize es in test. * Add it(), no es.init() * Clean up.

* add confirmation modal, move install state to Header * update callout to use title * move components only used in detail view to detail dir * use better variable names * update to more descriptive variable names

@skh

…51252) Discussed this with @skh elastic#51112 (comment) & elastic#51112 (comment) as well as in a video call Also added some TS type annotations for data fetching functions to make the contracts more explicit

…stic#51414) * API only shows installable assets. Server types to own file. Restore enums * Fix type imports * Only return installable asset types (kibana for now) * server/types now only has code from hapi which shouldn't go to client * Add more restricted TS types to DisplayAssets object * Flip order of arguments to Extract In these cases it still works the same, but looking at https://www.typescriptlang.org/docs/handbook/advanced-types.html the signature is `Extract<T, U>` - Extract from `T` those types that are assignable to `U` so the larger set should be first

* Disable create/destroy CTAs if no write capability Use `core.application.capabilities.ingestManager.write` to test user permissions * Add -all & -read tags for HTTP routes * Update test .expect() to match description * Add useCapabilities hook. Fix two issues with hiding/disabling CTA. Co-authored-by: Elastic Machine <[email protected]>

…p-from-ingest

FTR tests failing to start ES with error ``` ERROR ERROR: setting [xpack.security.authc.api_key.enabled] already set, saw [true] and [true] ``` https://github.com/elastic/kibana/pull/59376/checks?check_run_id=503930031 & https://github.com/elastic/kibana/pull/59376/checks?check_run_id=503975576 etc It appears the xpack.security.authc.api_key.enabled flag was recently added to master in another part, so remove our instance of the setting to prevent the error

These should have failed when the routes were changed. Will go back and see what happened.

* Add revision to agent config and datasource saved objects, add delete datasource service and datasource * Add revision to full agent config output * PR feedback

kibanamachine · 2020-03-12T21:44:57Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: 0ab28d3

Test Failures

Kibana Pipeline / x-pack-intake-agent / X-Pack Jest Tests.x-pack/plugins/endpoint/server/routes/resolver/queries.related events query generates the correct legacy queries

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

Stack Trace

Error: expect(received).toStrictEqual(expected) // deep equality

- Expected
+ Received

@@ -1,14 +1,7 @@
  Object {
    "body": Object {
-     "aggs": Object {
-       "total": Object {
-         "value_count": Object {
-           "field": "endgame.serial_event_id",
-         },
-       },
-     },
      "query": Object {
        "bool": Object {
          "filter": Array [
            Object {
              "terms": Object {
@@ -17,36 +10,27 @@
                ],
              },
            },
            Object {
              "term": Object {
-               "agent.id": "awesome-id",
+               "agent.id": Object {
+                 "eventID": "foo",
+                 "size": 1,
+                 "timestamp": 1584047038998,
+               },
              },
            },
            Object {
              "bool": Object {
                "must_not": Object {
                  "term": Object {
                    "event.category": "process",
-                 },
                  },
                },
              },
-         ],
            },
-     },
-     "search_after": Array [
-       1584047038998,
-       "foo",
          ],
-     "size": 1,
-     "sort": Array [
-       Object {
-         "@timestamp": "asc",
        },
-       Object {
-         "endgame.serial_event_id": "asc",
      },
-     ],
    },
    "index": "endgame-*",
  }
    at Object.it (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/x-pack/plugins/endpoint/server/routes/resolver/queries/related_events.test.ts:14:7)
    at Object.asyncJestTest (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-jasmine2/build/jasmineAsyncInstall.js:102:37)
    at resolve (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-jasmine2/build/queueRunner.js:43:12)
    at new Promise (<anonymous>)
    at mapper (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-jasmine2/build/queueRunner.js:26:19)
    at promise.then (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-jasmine2/build/queueRunner.js:73:41)

Kibana Pipeline / x-pack-intake-agent / X-Pack Jest Tests.x-pack/plugins/endpoint/server/routes/resolver/queries.related events query generates the correct non-legacy queries

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

Stack Trace

Error: expect(received).toStrictEqual(expected) // deep equality

- Expected
+ Received

@@ -1,35 +1,24 @@
  Object {
    "body": Object {
-     "aggs": Object {
-       "total": Object {
-         "value_count": Object {
-           "field": "event.id",
-         },
-       },
-     },
      "query": Object {
        "bool": Object {
          "filter": Array [
            Object {
-             "bool": Object {
-               "should": Array [
-                 Object {
              "terms": Object {
-                     "endpoint.process.entity_id": Array [
+               "endgame.unique_pid": Array [
                  "baz",
                ],
              },
            },
            Object {
-                   "terms": Object {
-                     "process.entity_id": Array [
-                       "baz",
-                     ],
-                   },
+             "term": Object {
+               "agent.id": Object {
+                 "eventID": "bar",
+                 "size": 1,
+                 "timestamp": 1584047039007,
                },
-               ],
              },
            },
            Object {
              "bool": Object {
                "must_not": Object {
@@ -37,24 +26,11 @@
                    "event.category": "process",
                  },
                },
              },
            },
-         ],
-       },
-     },
-     "search_after": Array [
-       1584047039007,
-       "bar",
          ],
-     "size": 1,
-     "sort": Array [
-       Object {
-         "@timestamp": "asc",
        },
-       Object {
-         "event.id": "asc",
      },
-     ],
    },
-   "index": undefined,
+   "index": "endgame-*",
  }
    at Object.it (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/x-pack/plugins/endpoint/server/routes/resolver/queries/related_events.test.ts:55:7)
    at Object.asyncJestTest (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-jasmine2/build/jasmineAsyncInstall.js:102:37)
    at resolve (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-jasmine2/build/queueRunner.js:43:12)
    at new Promise (<anonymous>)
    at mapper (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-jasmine2/build/queueRunner.js:26:19)
    at promise.then (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-jasmine2/build/queueRunner.js:73:41)

Kibana Pipeline / x-pack-intake-agent / X-Pack Jest Tests.x-pack/plugins/endpoint/server/routes/resolver/queries.children events query generates the correct legacy queries

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

Stack Trace

Error: expect(received).toStrictEqual(expected) // deep equality

- Expected
+ Received

@@ -1,14 +1,7 @@
  Object {
    "body": Object {
-     "aggs": Object {
-       "total": Object {
-         "value_count": Object {
-           "field": "endgame.serial_event_id",
-         },
-       },
-     },
      "query": Object {
        "bool": Object {
          "filter": Array [
            Object {
              "terms": Object {
@@ -17,37 +10,28 @@
                ],
              },
            },
            Object {
              "term": Object {
-               "agent.id": "awesome-id",
+               "agent.id": Object {
+                 "eventID": "foo",
+                 "size": 1,
+                 "timestamp": 1584047044776,
+               },
              },
            },
            Object {
              "term": Object {
                "event.category": "process",
              },
            },
            Object {
              "term": Object {
                "event.type": "process_start",
-             },
              },
-         ],
            },
-     },
-     "search_after": Array [
-       1584047044776,
-       "foo",
          ],
-     "size": 1,
-     "sort": Array [
-       Object {
-         "@timestamp": "asc",
        },
-       Object {
-         "endgame.serial_event_id": "asc",
      },
-     ],
    },
    "index": "endgame-*",
  }
    at Object.it (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/x-pack/plugins/endpoint/server/routes/resolver/queries/children.test.ts:14:7)
    at Object.asyncJestTest (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-jasmine2/build/jasmineAsyncInstall.js:102:37)
    at resolve (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-jasmine2/build/queueRunner.js:43:12)
    at new Promise (<anonymous>)
    at mapper (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-jasmine2/build/queueRunner.js:26:19)
    at promise.then (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-jasmine2/build/queueRunner.js:73:41)

and 6 more failures, only showing the first 3.

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

jen-huang · 2020-03-13T01:19:45Z

FYI, feature-ingest was merged into master (#59376), so this work can be rebased against master now.

jonathan-buttner · 2020-03-13T13:06:18Z

Thanks @jen-huang I'll switch it over!

…p-from-ingest

jonathan-buttner · 2020-03-13T14:24:16Z

Closing because commits from feature-ingest aren't syncing up.

John Schulz and others added 30 commits November 15, 2019 14:10

[EPM] useKibana hook & render in plugin.start (elastic#50110)

d470bb9

* plugin.start now does reactdom.render vs returning react element export plugin function from public/index * Move setClient call from plugin.start to plugin.setup * Use `useUiSetting$` from `useKibana` hooks

Fix broken app due to bad hooks usage

1d8b931

Can't use useKibana outside a React component. Reverting to prior approach since it's still NP. Can revisit context usage in a followup PR

[EPM] Install package from detail view on button click (elastic#50735)

e7a7f74

* Support basic "click button -> show spinner -> installed" install flow * Remove incorrect comments. Add TS return types to data functions.

[EPM] Use NP feature_catalogue.register (elastic#50108)

232347f

* Use NP feature_catalogue.register * Use type from NP plugin

Merge branch 'feature-make-it-ingest' of github.com:elastic/kibana in…

570c1c7

…to feature-ingest

Merge branch 'feature-integrations-manager' of github.com:elastic/kib…

c7aa2ed

…ana into feature-ingest

fix linting

536d9f1

fix types

0831ef0

fix headers in Fleet

9447054

skipping test due to ES param change

d05f20d

Revert "skipping test due to ES param change"

9fc2f1c

This reverts commit d05f20d.

remove type field

0571584

remove unused import

8aa89d7

[EPM] Final(?) update from integrations_manager -> EPM (elastic#50976)

be9ca17

* Update (all remaining?) references from integrations_manager to EPM * Update path in i18n file

[EPM] update compatibility section (elastic#50975)

ece3570

* change min max version to single range value * add elastic stack icon and change text * remove badge from version and use code font * remove euistyled * add back version component lost in merge * remove euiStyled * remove old file

Merge branch 'master' of github.com:elastic/kibana into feature-ingest

646b340

Restore RequirementVersionRange type

590ccbe

Disable test for elasticsearch username.

f1020e4

Temporary work around until we know how the stack wants to add the permissions we need. Either adding to the kibana user or creating a new user.

Merge branch 'master' of github.com:elastic/kibana into feature-ingest

ef97a13

Revert "Disable test for elasticsearch username."

4720fd9

This reverts commit f1020e4.

Disable test for elasticsearch username.

37dad5c

Temporary work around until we know how the stack wants to add the permissions we need. Either adding to the kibana user or creating a new user.

Merge branch 'master' of github.com:elastic/kibana into feature-ingest

098c220

Fix EPM typing issues in register feature

9c158e5

Merge branch 'master' of github.com:elastic/kibana into feature-ingest

229745f

Fix typings after master merge

9f68a23

[EPM] CI fixes (elastic#51284)

c1512f8

* Initialize es in test. * Add it(), no es.init() * Clean up.

[EPM] add confirmation modal (elastic#51172)

e373cba

* add confirmation modal, move install state to Header * update callout to use title * move components only used in detail view to detail dir * use better variable names * update to more descriptive variable names

Restore prior response vaulues for install & delete package (elastic#…

0493795

…51252) Discussed this with @skh elastic#51112 (comment) & elastic#51112 (comment) as well as in a video call Also added some TS type annotations for data fetching functions to make the contracts more explicit

[Fleet] Enrollment api key UI (elastic#51495)

d47ab5d

jonathan-buttner and others added 12 commits March 11, 2020 17:01

exporting route variable

b9895e7

Adding new datastreams file

42e9f52

[Fleet] fix output rename api_token => api_key (elastic#60001)

27d85cf

Missed one in e12a8ad

352e5f6

Use package icon as default when no other can be found (elastic#60025)

1ee3c1e

Adding datastream get request

e89f675

Merge branch 'feature-ingest' of github.com:elastic/kibana into get-i…

693ecab

…p-from-ingest

Merge remote-tracking branch 'upstream/master' into feature-ingest

e83b67b

Update EPM file tests to use /packages/{pkgkey}

4a9af8d

These should have failed when the routes were changed. Will go back and see what happened.

Pulling in index pattern service into endpoint app

0ab28d3

jonathan-buttner changed the title ~~[WIP] Retrieve index pattern from ingest manager app~~ [WIP][skip-ci] Retrieve index pattern from ingest manager app Mar 12, 2020

jonathan-buttner added v7.7.0 v8.0.0 WIP Work in progress labels Mar 12, 2020

jonathan-buttner and others added 5 commits March 12, 2020 16:09

removing reference to datastream service

471646d

switch saved objects field name to patterns

6b76a48

removing route to data_stream

9694bfb

removing more references to datastreams

4a5f99c

[Ingest] Add revision to agent configs & data sources (elastic#59848)

755ba3a

* Add revision to agent config and datasource saved objects, add delete datasource service and datasource * Add revision to full agent config output * PR feedback

jonathan-buttner changed the base branch from feature-ingest to master March 13, 2020 13:06

jonathan-buttner added 2 commits March 13, 2020 09:07

Merge branch 'feature-ingest' of github.com:elastic/kibana into get-i…

d25e7a2

…p-from-ingest

Fixing merge conflicts

2fe9fb5

jonathan-buttner closed this Mar 13, 2020

jonathan-buttner deleted the get-ip-from-ingest branch April 1, 2020 18:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP][skip-ci] Retrieve index pattern from ingest manager app #60063

[WIP][skip-ci] Retrieve index pattern from ingest manager app #60063

jonathan-buttner commented Mar 12, 2020

kibanamachine commented Mar 12, 2020

Standard Out

Stack Trace

Standard Out

Stack Trace

Standard Out

Stack Trace

jen-huang commented Mar 13, 2020

jonathan-buttner commented Mar 13, 2020

jonathan-buttner commented Mar 13, 2020

[WIP][skip-ci] Retrieve index pattern from ingest manager app #60063

[WIP][skip-ci] Retrieve index pattern from ingest manager app #60063

Conversation

jonathan-buttner commented Mar 12, 2020

kibanamachine commented Mar 12, 2020

💔 Build Failed

Test Failures

Standard Out

Stack Trace

Standard Out

Stack Trace

Standard Out

Stack Trace

jen-huang commented Mar 13, 2020

jonathan-buttner commented Mar 13, 2020

jonathan-buttner commented Mar 13, 2020