Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fleet] Add space ID to Fleet system indices #182717

Closed
4 tasks done
kpollich opened this issue May 6, 2024 · 3 comments · Fixed by elastic/elasticsearch#108363
Closed
4 tasks done

[Fleet] Add space ID to Fleet system indices #182717

kpollich opened this issue May 6, 2024 · 3 comments · Fixed by elastic/elasticsearch#108363
Assignees
Labels
Team:Elastic-Agent-Control-Plane Team:Fleet Team label for Observability Data Collection Fleet team

Comments

@kpollich
Copy link
Member

kpollich commented May 6, 2024

#181860 details the process of moving Fleet's saved objects to a new single-space affinity model. In addition to this work, we need to make Fleet's non-saved-object data segregated by space.

Relevant dev docs:

## `.fleet-*` Indices
For any data that needs to be accessible by Fleet Service instances to push updates to, we write and read data
directly to a handful of `.fleet-` Elasticsearch indices. Fleet Server instances are configured with an API key that
has access only to these indices.
In prior alpha versions of Fleet, this data was also stored in Saved Objects because Elastic Agent instances were
communicating directly with Kibana for policy updates. Once Fleet Server was introduced, that data was migrated to these
Elasticsearch indices to be readable by Fleet Server.
_Note: All of these system indices are plain indices, and not data streams._
### `.fleet-agents` index
Each document in this index tracks an individual Elastic Agent's enrollment in the Fleet, which policy it is current
assigned to, its check in status, which packages are currently installed, and other metadata about the Agent.
All of the code that interacts with this index is currently located in
[`x-pack/plugins/fleet/server/services/agents/crud.ts`](../server/services/agents/crud.ts) and the schema of these
documents is maintained by the `FleetServerAgent` TypeScript interface.
- Cleanup model: N/A
### `.fleet-actions` index
Each document in this index represents an action that was initiated by a user and needs to be processed by Fleet Server
and sent to any agents that it applies to. Actions can apply to one or more agents. There are different types of actions
that can be created such as policy changes, unenrollments, upgrades, etc. See the `AgentActionType` type for a complete
list.
The total schema for actions is represented by the `FleetServerAgentAction` type.
- Cleanup model: Fleet Server considers actions expired after 30 days, and will remove them via an hourly process
- [Source](https://github.com/elastic/fleet-server/blob/9af3b2176b42a0de34c5583b5430558c03792dd0/internal/pkg/gc/schedules.go#L29-L33)
### `.fleet-actions-results`
- Cleanup model: N/A
### `.fleet-servers`
- Cleanup model: N/A
### `.fleet-artifacts`
- Cleanup model: N/A
### `.fleet-enrollment-api-keys`
- Cleanup model: Deleteable via Fleet UI/API, deleted when an agent policy is deleted
- [Source](https://github.com/elastic/kibana/blob/7a35748cb43f2c73623ffda6fa02b91c3cb4c689/x-pack/plugins/fleet/server/services/api_keys/enrollment_api_key.ts#L102)
### `.fleet-policies`
- Cleanup model: Deleted when a corresponding agent policy is deleted in the Fleet UI or API
- [Source](https://github.com/elastic/kibana/blob/976b1b2331371f4a1325f6947d38d1f4de7a7254/x-pack/plugins/fleet/server/services/agent_policy.ts#L699-L701)
### `.fleet-policies-leader`
- Cleanup model: N/A

Implementation

  • Add space ID to .fleet-enrollment-api-keys documents
  • Add space ID to .fleet-fileds-* indices - Link
  • Add space ID to .fleet-actions and .fleet-actions-results
  • Add space ID to .fleet-policies

It's unclear whether the following system indices are still in use or not, and whether or not they should be updated

  • .fleet-servers
  • .fleet-artifacts
  • .fleet-policies-leader
@kpollich kpollich added the Team:Fleet Team label for Observability Data Collection Fleet team label May 6, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/fleet (Team:Fleet)

@nchaulet
Copy link
Member

nchaulet commented May 6, 2024

@kpollich I am about to start that one as it will unblock a bunch of issues, what do you think of using the namespaces: keyword mapping here?, consistent with SO, (and it could potentially allow us to support multiple namespace in a long future)

@kpollich
Copy link
Member Author

kpollich commented May 6, 2024

namespaces: keyword for the mapping type w/ an array of keywords for the space ID makes sense to me 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Team:Elastic-Agent-Control-Plane Team:Fleet Team label for Observability Data Collection Fleet team
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants