Failing ES Promotion: x-pack/test/fleet_api_integration/config.ts / Fleet Endpoints EPM Endpoints

[mistic]

Fleet Endpoints EPM Endpoints
x-pack/test/fleet_api_integration/apis/*

apis/epm
apis/fleet_setup
package_policy/create
package_policy/update
package_policy/get

This failure is preventing the promotion of the current Elasticsearch nightly snapshot.

For more information on the Elasticsearch snapshot promotion process including how to reproduce using the unverified nightly ES build: https://www.elastic.co/guide/en/kibana/master/development-es-snapshots.html

• Failed promotion job

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[mistic]

Skipped.

main: 5feeb17

[kpollich]

Seems the endpoint package failed to install here:

 proc [kibana] [2023-04-10T14:41:36.277+00:00][ERROR][plugins.fleet] Failed to fetch latest version of endpoint from registry: Error connecting to package registry: request to http://localhost:6104/search?package=endpoint&prerelease=false&kibana.version=8.8.0 failed, reason: connect ECONNREFUSED 127.0.0.1:6104

lists all limited packages from the registry
--
  | └-> "before each" hook: global before each for "lists all limited packages from the registry"
  | └-> "before each" hook: beforeSetupWithDockerRegistry for "lists all limited packages from the registry"
  | └- ✖ fail: Fleet Endpoints EPM Endpoints EPM - list list api tests lists all limited packages from the registry
  | │       Error: expected [] to sort of equal [ 'endpoint' ]
  | │       + expected - actual
  | │
  | │       -[]
  | │       +[
  | │       +  "endpoint"
  | │       +]
  | │
  | │       at Assertion.assert (expect.js:100:11)
  | │       at Assertion.eql (expect.js:244:8)
  | │       at Context.<anonymous> (list.ts:58:46)
  | │       at runMicrotasks (<anonymous>)
  | │       at processTicksAndRejections (node:internal/process/task_queues:96:5)
  | │       at Object.apply (wrap_function.js:73:16)
  | │

Seems like the EPR container did come up as expected, though

info [docker:registry] pulling docker image "docker.elastic.co/package-registry/distribution:lite"
--
  | info [o.e.c.m.MetadataCreateIndexService] [ftr] [.ds-.logs-deprecation.elasticsearch-default-2023.04.10-000001] creating index, cause [initialize_data_stream], templates [.deprecation-indexing-template], shards [1]/[1]
  | info [o.e.c.m.MetadataCreateDataStreamService] [ftr] adding data stream [.logs-deprecation.elasticsearch-default] with write index [.ds-.logs-deprecation.elasticsearch-default-2023.04.10-000001], backing indices [], and aliases []
  | info [o.e.c.r.a.AllocationService] [ftr] updating number_of_replicas to [0] for indices [.ds-.logs-deprecation.elasticsearch-default-2023.04.10-000001]
  | info [o.e.x.i.IndexLifecycleTransition] [ftr] moving index [.ds-.logs-deprecation.elasticsearch-default-2023.04.10-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [.deprecation-indexing-ilm-policy]
  | info [o.e.x.i.IndexLifecycleTransition] [ftr] moving index [.ds-.logs-deprecation.elasticsearch-default-2023.04.10-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] in policy [.deprecation-indexing-ilm-policy]
  | info [o.e.c.r.a.AllocationService] [ftr] current.health="GREEN" message="Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.ds-.logs-deprecation.elasticsearch-default-2023.04.10-000001][0]]])." previous.health="YELLOW" reason="shards started [[.ds-.logs-deprecation.elasticsearch-default-2023.04.10-000001][0]]"
  | info [o.e.x.i.IndexLifecycleTransition] [ftr] moving index [.ds-.logs-deprecation.elasticsearch-default-2023.04.10-000001] from [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] to [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] in policy [.deprecation-indexing-ilm-policy]
  | info [o.e.c.m.MetadataMappingService] [ftr] [.ds-.logs-deprecation.elasticsearch-default-2023.04.10-000001/rdfXdGVXQqaOy0PyC5mrGw] update_mapping [_doc]
  | info [docker:registry] running image "docker.elastic.co/package-registry/distribution:lite"
  | info [docker:registry] Waiting for log line containing "package manifests loaded"
  | info [docker:registry] {"log.level":"info","@timestamp":"2023-04-10T14:41:50.532Z","log.origin":{"file.name":"package-registry/main.go","file.line":304},"message":"Packages paths: /packages/package-storage, /packages/test-packages, /packages/signed-test-packages","ecs.version":"1.6.0"}
  | info [docker:registry] {"log.level":"info","@timestamp":"2023-04-10T14:41:50.532Z","log.origin":{"file.name":"package-registry/main.go","file.line":305},"message":"Cache time for /: 10s","ecs.version":"1.6.0"}
  | info [docker:registry] {"log.level":"info","@timestamp":"2023-04-10T14:41:50.532Z","log.origin":{"file.name":"package-registry/main.go","file.line":306},"message":"Cache time for /index.json: 10s","ecs.version":"1.6.0"}
  | info [docker:registry] {"log.level":"info","@timestamp":"2023-04-10T14:41:50.532Z","log.origin":{"file.name":"package-registry/main.go","file.line":307},"message":"Cache time for /search: 10m0s","ecs.version":"1.6.0"}
  | info [docker:registry] {"log.level":"info","@timestamp":"2023-04-10T14:41:50.532Z","log.origin":{"file.name":"package-registry/main.go","file.line":308},"message":"Cache time for /categories: 10m0s","ecs.version":"1.6.0"}
  | info [docker:registry] {"log.level":"info","@timestamp":"2023-04-10T14:41:50.532Z","log.origin":{"file.name":"package-registry/main.go","file.line":309},"message":"Cache time for all others: 10m0s","ecs.version":"1.6.0"}
  | info [docker:registry] {"log.level":"info","@timestamp":"2023-04-10T14:41:50.532Z","log.origin":{"file.name":"package-registry/main.go","file.line":140},"message":"Package registry started","ecs.version":"1.6.0"}
  | info [docker:registry] {"log.level":"info","@timestamp":"2023-04-10T14:41:50.637Z","log.origin":{"file.name":"packages/packages.go","file.line":227},"message":"Searching packages in /packages/package-storage","ecs.version":"1.6.0"}
  | info [docker:registry] {"log.level":"info","@timestamp":"2023-04-10T14:41:54.104Z","log.origin":{"file.name":"packages/packages.go","file.line":227},"message":"Searching packages in /packages/test-packages","ecs.version":"1.6.0"}
  | info [docker:registry] {"log.level":"info","@timestamp":"2023-04-10T14:41:54.111Z","log.origin":{"file.name":"packages/packages.go","file.line":227},"message":"Searching packages in /packages/signed-test-packages","ecs.version":"1.6.0"}
  | info [docker:registry] {"log.level":"info","@timestamp":"2023-04-10T14:41:54.116Z","log.origin":{"file.name":"packages/packages.go","file.line":227},"message":"Searching packages in /packages/package-storage","ecs.version":"1.6.0"}
  | info [docker:registry] {"log.level":"warn","@timestamp":"2023-04-10T14:41:54.116Z","log.origin":{"file.name":"packages/packages.go","file.line":130},"message":"ignoring unexpected file","file.path":"/packages/test-packages/endpoint/endpoint-8.6.1.zip","ecs.version":"1.6.0"}
  | info [docker:registry] {"log.level":"info","@timestamp":"2023-04-10T14:41:54.117Z","log.origin":{"file.name":"packages/packages.go","file.line":227},"message":"Searching packages in /packages/test-packages","ecs.version":"1.6.0"}
  | info [docker:registry] {"log.level":"info","@timestamp":"2023-04-10T14:41:54.152Z","log.origin":{"file.name":"packages/packages.go","file.line":227},"message":"Searching packages in /packages/signed-test-packages","ecs.version":"1.6.0"}
  | info [docker:registry] {"log.level":"info","@timestamp":"2023-04-10T14:41:54.152Z","log.origin":{"file.name":"package-registry/main.go","file.line":324},"message":"407 local package manifests loaded.","ecs.version":"1.6.0"}

[kevinlog]

@kpollich we've got some skipped tests due to promotion as well. I'm seeing problems with permissions on transform install.

EndpointDataLoadingError: security_exception
	Root causes:
		security_exception: Cannot create transform [endpoint.metadata_united-default-8.8.0-preview-2] because user kibana_system lacks the required permissions [.fleet-agents*:[read, view_index_metadata], .metrics-endpoint.metadata_united_default:[], metrics-endpoint.metadata_current_default*:[]]
    at installOrUpgradeEndpointFleetPackage (setup_fleet_for_endpoint.ts:119:13)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at Context.<anonymous> (index.ts:35:7)
    at Object.apply (wrap_function.js:73:16) {
  meta: [
    {
      name: 'endpoint',
      statusCode: 500,
      error: 'security_exception\n' +
        '\tRoot causes:\n' +
        '\t\tsecurity_exception: Cannot create transform [endpoint.metadata_united-default-8.8.0-preview-2] because user kibana_system lacks the required permissions [.fleet-agents*:[read, view_index_metadata], .metrics-endpoint.metadata_united_default:[], metrics-endpoint.metadata_current_default*:[]]'
    }
  ]
}

issue: #154741

I'm trying to reproduce

[kevinlog]

Reproduced our error with the latest source which I think is likely an upstream issue to Fleet.

Comment in out own issue: #154741 (comment)

[kevinlog]

@kpollich @jen-huang - to update the issue on our side, we worked with the ES team to merge this fixv elastic/elasticsearch#94724 .

We've since worked with Kibana ops to push out another ES snapshot and we successfully re-enabled our tests.

More details:
#154741

Test unskips:
#154863

[jen-huang]

@kevinlog Thanks Kevin! We'll work on unskipping this test and other ones that were affected.