[Fleet] Improve support for air-gapped environments without an accessible EPR instance

[roboneci]

Fleet should be able to offer a limited experience around browsing and installing packages even if an instance of EPR isn't accessible on the network. Related to #112095

✅ Task List

• [Fleet] Prevent registry network errors during Fleet setup's bulk install operation from failing the installation of bundled packages #125097
• [Fleet] Remove dependency on reachable EPR from "Installed Integrations" tab in integrations UI #134326
• Remove dependency on reachable EPR from "Integration Details" screen. Similar to the above, we should be able to serve this page using data from a bundled package so long as it's installed and has an active integration policy. #134325

Context

In #122297, we added support for bundling packages along with Kibana to remove Fleet's dependency on EPR during its setup process. We no longer need to have an accessible instance of EPR in order to install packages so long as they are bundled with Kibana. This allows us to preconfigure things like APM and Fleet Server integrations and avoid costly or potentially brittle network calls during Kibana's boot process.

Now that this functionality is in place, we should do a pass of Fleet's logic to account for cases in which EPR is inaccessible, and serve up a limited Fleet/Integrations experience.

Goals

• Fleet Setup should succeed without an accessible EPR instance
• A user should be able to view the "browse integrations" screens without an accessible EPR instance - Already works! 🎉
• A user should be able to view the "Installed Integrations" tab of the integrations UI and see any integrations for which they have active package policies
• A user should be able to view the "details" page for their installed integrations without an accessible EPR instance

Current State

Fleet Setup

Currently, if the EPR is inaccessible, Fleet Setup will succeed with a non-fatal error and will not install a package even if it's bundled. e.g.

This should be a relatively quick fix to catch this error and allow bundled install to continue. I believe this error comes from Fleet attempting to query the Registry for the latest version here:

kibana/x-pack/plugins/fleet/server/services/epm/packages/bulk_install_packages.ts

Lines 40 to 45 in 73cc08e

	const packagesResults = await Promise.allSettled(
	packagesToInstall.map((pkg) => {
	if (typeof pkg === 'string') return Registry.fetchFindLatestPackage(pkg);
	return Promise.resolve(pkg);
	})
	);

"Browse Integrations" view

This view already supports inaccessible EPR environments and will display a CTA for setting up an accessible instance while continuing to display non-agent integrations cards like Beats modules and tutorials.

Installed Integrations

This view is not accessible without a reachable EPR instance

Integration details

The integration details page is not accessible without a reachable EPR instance, even in the case of a bundled package installed at boot time.

---

Original Description

**Describe the feature:** Hello, i want to try your elastic agent in out air gap environment. I know that this is experimental thing. I follow this manual: https://www.elastic.co/guide/en/ingest-management/current/ingest-management-getting-started.html

But i'm stuck at point:

Verify that ingest management has been correctly enabled:
Log in to Kibana and go to Management > Ingest Manager.

Kibana told me that:
Unable to initialize Ingest Manager
Error connecting to package registry: request to https://epr-experimental.elastic.co/search?package=base&internal_true ...

Of course our air gap environment doesn't have connection to internet.

Describe a specific use case for the feature:

[roboneci]

From:
https://github.com/elastic/kibana/blob/master/x-pack/plugins/ingest_manager/README.md

it should be possible for license Gold+. How to missing and why we should run another web server? I'm sure we can use existing kibana instance.

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[ruflin]

@roboneci Great to see you tried out ingest manager. I assume you are looking for a way to run the registry on prem and not use a proxy.

You are correct that it is possible to run the registry on prem as we use this for testing at the moment but we don't support it yet. The reasons is that this brings one additional variable to the versions. If users connect to our registry, we know exactly the version and control it. So before we start supporting it, we want to have good recommendations on how to do it, order of upgrades etc. If you are interested to playing around with it, the registry with the packages is available under docker.elastic.co/package-registry/distribution:production but please keep in mind, this is not supported and might break at any time.

[mostlyjason]

We published a list of known limitations for ingest manager here https://www.elastic.co/guide/en/ingest-management/current/ingest-management-limitations.html. That includes requiring access to the internet. I hope we can add official support for this in a later release.

[mbarretta]

@mostlyjason how is this progressing? We've been talking about supporting air-gapped users for a year, and while I know Agent and Fleet aren't GA, they soon will be. As it is, these users can't even evaluate Agent or how they might need to change their environment before 8.0

[scottdfedorov]

I know that to-date the discussion here has been around the package registry, but hope that "Support air-gap environment" also include the ability to upgrade agents.

I'm an ECE, with a platinum support contract. We worked with our killer support engineer Robbie (@n0othing) on some beta testing of Fleet. We found we cannot upgrade agents without access to the internet for each agent, which breaks all our security best practices.

We can set the agent.download.sourceURI field in the elastic-agent.yml file to point to a locally hosted http server to have the agents download updates. This works but currently requires us to update every file to add the field. Robbie opened an issue on our behalf #100413 to add the ability for this field to be set via the Kibana GUI so we don't have to manually update all agent's files.

I was hoping this could be added to consideration for "support air gap env".

Also, as an aside, there's a related bug: the code suggests you should be able to use a file server rather than http server, but that was failing in our testing Open issue here: elastic/beats#25855

[mostlyjason]

@mbarretta We are moving forward with requirements definition for Offline EPR elastic/integrations#1178. @akshay-saraswat is the PM leading this on the integrations team. This will likely be a post-GA enhancement.

++ Yes @scottdfedorov we are tracking this item on our roadmap thanks for filing the issues. This will likely come after GA as well.

[akshay-saraswat]

We announced the beta for Offline EPR. https://www.elastic.co/guide/en/integrations-developer/current/air-gapped.html
GA requirements are under discussion.

[kpollich]

Hello all. I am taking over this issue and preparing it for the Fleet team's iteration work in an upcoming release. The original issue description is preserved at the bottom of the new description. Thanks!

[mostlyjason]

@kpollich what about updating the browse tab? It'd be nice to show the cards for bundled packages here as well. If we do that, we should update the warning text to say something more generic like "Kibana is not able to retrieve integrations from the Elastic Package Registry." The reason is that EPR is no longer the only source of Elastic Agent integrations.

[kpollich]

@kpollich what about updating the browse tab? It'd be nice to show the cards for bundled packages here as well. If we do that, we should update the warning text to say something more generic like "Kibana is not able to retrieve integrations from the Elastic Package Registry." The reason is that EPR is no longer the only source of Elastic Agent integrations.

@joshdover and I had an offline conversation about this functionality with @ruflin recently and the general consensus was that we don't want to introduce a second "source of truth" to the integrations browsing experience. Bundled packages are intended more like a fallback cache of package assets, and the sole source of integrations data should remain the registry. I could see an argument for altering this behavior in air-gapped mode, though.

[mostlyjason]

Thanks! I'm trying to think through how this will affect the UX for APM and Fleet Server. When the user is in an air-gapped environment without an on-prem EPR set up, what happens if they click Integrations > APM > Manage APM integration in Fleet? They will have an integration server running, which I assume preconfigures the APM package and integration? One option is to show an error if they try to view/edit/add APM agents. @simitt do you have any input on this UX?

Also, what happens when the user tries to scale their Fleet Server by clicking Fleet > Policies > Elastic Cloud agent policy > Edit Fleet Server? Again, we could show an error here if they try to view/edit their Fleet Server integration policy. @nimarezainia do you have any input on that UX?

[elasticmachine]

Pinging @elastic/fleet (Feature:EPM)

[joshdover]

As the other problems are solved, I'm closing this in favor of the one remaining issue: #134326