[APM] Agent API Key Management

[graphaelli]

Metadata support added in elastic/elasticsearch#48182

Summary

APM has supported API Key authentication between agents and server for awhile. So far, the easiest way to manage keys is using the apm-server apikey subcommand. This is particularly burdensome for our ESS users. Here, we introduce a UI for managing those keys, matching the functionality provided by the current tool for creating and invalidating keys.

For the initial UI, only immediate invalidation is needed - scheduled expiration is not needed.
For creation, keys should be able to include a scheduled expiration and include any combination equivalent of these 3 permissions:

      --ingest              give the {event event:write} privilege to this key, required for ingesting events
      --sourcemap           give the {sourcemap sourcemap:write} privilege to this key, required for uploading sourcemaps
      --agent-config        give the {agentConfig config_agent:read} privilege to this key, required for agents to read configuration remotely

Design

This UI could fit in with settings, like:

though the key names should not actually be links.

The create key button triggers a flyout similar to the one for create alert including:

• a text box for Name, required, default to key-yyyy-mm-dd-hh-mm-ss
• 3 checkbox - ingest, sourcemap, agent-config - at least one required, all checked by default
• Expiration date setting, optional (and not required for MVP), default to never

@formgeist has signed off on the initial design and should be tagged for review of the final flyout.

Future

We can tackle integration with the getting started guide as a follow up - secret_token is still supported and will be for some time. Alerts/UI notification for soon to expire keys would also be a nice follow up to MVP.

Security

To create a key, users must have the manage_api_key cluster privilege to create keys - failure to create the key due to permissions should mention this requirement.

API Keys are only available per these conditions:

the Elasticsearch API key service ... is automatically enabled when you configure TLS on the HTTP interface. Alternatively, you can explicitly enable the xpack.security.authc.api_key.enabled setting. When you are running in production mode, a bootstrap check prevents you from enabling the API key service unless you also enable TLS on the HTTP interface.

Elasticsearch API Calls

Create

Before creating a key, apm-server does a has_privileges check as explained here, adjusting the privileges as needed and ensuring the returned has_all_requested value is true:

GET /_security/user/_has_privileges
{
  "application": [
    {
      "application": "apm",
      "privileges": [
        "event:write",
        "sourcemap:write"
        "config_agent:read"
      ],
      "resources": [
        "-"
      ]
    }
  ]
}

To create the key, again adjusting for privileges specified:

POST /_security/api_key
{
  "expiration": "1s",
  "name": "key-2020-09-14-003",
  "role_descriptors": {
    "apm": {
      "applications": [
        {
          "application": "apm",
          "privileges": [
            "event:write",
            "sourcemap:write",
            "config_agent:read"
          ],
          "resources": [
            "*"
          ]
        }
      ]
    }
  }
}

When the key is created the user should be shown a message stating that they should save it as it can't be displayed again. The value shown should be base64 encoding of the id + ":" + key. This requirement may be lifted by the time this is implemented so be sure to verify it.

Invalidate

To invalidate a key:

DELETE /_security/api_key
{"id":"pk6fvnUBTgvLvGN2XNn5"}

List

API Keys can be retrieved by convention via metadata using the paginated get api key api.

{
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "metadata.application": "apm"
          }
        }
      ]
    }
  }
}

returns

    {
      "creation": 1629852761785,
      "id": "Z2PLensBk-RCONrKfi-5",
      "invalidated": false,
      "metadata": {
        "application": "apm"
      },
      "name": "key-2021-08-24-1629852761",
      "realm": "file1",
      "username": "admin"
    }
  ],
  "count": 10,
  "total": 16
}

Tasks:

• Add Agent Keys in APM settings with support for viewing and invalidating agent keys [APM] Add Agent Keys in APM settings - Agent key table #119543
• Add support for creating agent keys [APM] Add Agent Keys in APM settings - Create agent keys #120373

[elasticmachine]

Pinging @elastic/apm-ui (Team:apm)

[formgeist]

@graphaelli Looks good to me - I can very quickly do a mock of the create flyout for creating keys and add it to this issue so it's ready for implementation.

[cyrille-leclerc]

FYI we have been involved by the Stack / Elasticsearch team in a work to homogeneize API keys.

[cyrille-leclerc]

FYI our initiative to expose a dedicated view to manage Agent API Keys is well aligned with the transversal initiative on API keys.

[ruflin]

I assume in addition to the UI, there will be an API to manage these keys? So instead of automating it with apm-server today, a user could automate it by talking to the Kibana APM API.

[sorenlouv]

@ruflin Yes, sounds reasonable that we should document the API and thereby making it publicly accessible.

[graphaelli]

I have updated description based on refinement to:

• list the elastisearch api calls for managing keys
• specify permissions needed
• remove links from API Key names

This raises an issue that I thought was already resolved - there doesn't appear to be a way to query keys by permission (all keys with APM application access) or to add metadata keys so they can be found later. I'd rather not use a hack like naming them in a certain way just to be able to find them later. Labeling this issue blocked for the moment while we resolve this.

[cjcenizal]

Hey team, I just wanted to make sure y’all were aware of the API Key Management UI under Stack Management. How does that UI relate to this one from a product and user perspective? Should there be cross-linking between the two? Is there any reusable code from that UI that can be used for this UI?

[sorenlouv]

Hey @cjcenizal,

Yes we are aware of the existing API Management UI. The UI we are building is tailor-made for creating API keys to be used between APM agents and APM server. From what I can tell there is no UI for creating API keys - is that correct?

If it's possible to tag API keys (eg. "apm") and the existing UI supports linking to this, this could make sense. Example of the url I'm thinking of:

/app/management/security/api_keys?tag=apm

[cjcenizal]

Thanks @sqren! I'm going to tag in the @elastic/kibana-security team to answer your questions since they own that plugin.

[legrego]

👋 hey folks - thanks for the ping, CJ.

Allowing API Keys to be "tagged" would likely be something that we'd want to implement within ES. One possible solution is to add support for generic metadata (elastic/elasticsearch#48182), which Kibana and other solutions could take advantage of.

From what I can tell there is no UI for creating API keys - is that correct?

It's funny you ask. @thomheymann started work on a PR for this just recently (#82005). We had a discussion earlier today with @arisonl and @bytebilly, where we decided to postpone work on this.

API Keys are used for a number of different things, and we want to take a step back and think about this more holistically before adding more UIs around them. I'd suggest doing the same here until we have a better idea of what we want to implement. Our hope is to have more concrete thoughts around this in the next couple of weeks. It's possible that we'll introduce new constructs or change existing ones around API Keys, so tailoring a ux around today's API Keys may not work for us in the near future.

[simitt]

Cross linking #78222 to ensure efforts are aligned (or conciously kept separated).

[sorenlouv]

Thanks @simitt. Even with a centralized UI for managing API keys in Kibana, I still think it makes sense to have a dedicated APM view that only shows apm api keys.

[bytebilly]

We had some discussion in the context of the broader plan for API keys, and we agreed it would be better to have a separate UI for APM API keys rather than considering this feature in the scope of #78222.

Prioritization of metadata (or other alternatives to identify keys) will be discussed in the next few weeks. We can then synchronize on timelines.

[sorenlouv]

Sounds good. Thanks for the update @bytebilly

[sorenlouv]

With elastic/elasticsearch#70292 merged I'll mark this as no longer blocked.

[graphaelli]

#78222 has been closed by #92610 which might enable some reusable pieces to implement this

[graphaelli]

related to elastic/elasticsearch#73705, should make this possible.

[bytebilly]

I don't have full visibility on the use of API keys and metadata that you are envisioning, so I want to point out that relying on metadata should not be considered a security paradigm since anyone with proper permissions can create API keys with arbitrary metadata, potentially the very same as what is set for APM.

You could rely on the API key creator in conjunction with metadata to make it safer. This works well if the creator is an APM service account, rather than the end user interacting with the system. We already implemented that for Fleet, so there may be some similarities. Happy to discuss more if you will.

[sorenlouv]

@bytebilly I don't think this is going to be a security issue but I'll explain our use case and then you can tell me how I'm wrong :D

Without the ability to filter by metadata we would have to refer the user to the general API keys view, where they'd have to manually sift through the list of possible 1000s of keys to find the ones relevant to APM.

By filtering by metadata, we can show them just the relevant APM keys. If somebody has created a non-apm key and tagged it with "apm" (accidentally or maliciously) it will show up. This might create some confusion but I don't see any security concerns here. The only action the user can take is to revoke the key if they don't recognise it.

Also, if a malicious actor succeeds in creating an API key it will show up in the general API key UI anyway, which should already be cause for concern, no?

[bytebilly]

@sqren I agree with you. My concern was not specific to the view of keys, so I don't think it's a blocker here.

[graphaelli]

with elastic/elasticsearch#75335 this is getting a lot closer to feasible. We still need pagination support which is on the way with elastic/elasticsearch#76144.

GET /_security/_query/api_key
{
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "metadata.application": "apm"
          }
        }
      ]
    }
  }
}

returns

{
  "api_keys" : [
    {
      "id" : "6HsOGHsB8G4U9ZuEp9nD",
      "name" : "key-2021-08-05-1628196218",
      "creation" : 1628196218821,
      "invalidated" : false,
      "username" : "admin",
      "realm" : "file1",
      "metadata" : {
        "application" : "apm"
      }
    }
    ...

[graphaelli]

pagination has landed, description updated accordingly - this is no longer blocked.

[gbamparop]

Closing this issue. Issues have been created for documenting agent key management in APM Server and adding docs for the public APM app API.

[gbamparop]

In order to create an APM agent key, users must have at least the manage_own_api_key cluster privilege and the required application privileges. For example, a user with the privileges shown in the example below will be able to create an APM agent key with all the currently supported privileges (sourcemap:write, event:write, config_agent:read). After discussing with @graphaelli and @sqren these privileges will not be added to an existing built-in-role for APM.

Example (part of GET /_security/user/_has_privileges response):

...

"cluster" : {
    "manage_own_api_key" : true
},
"application" : {
    "apm" : {
      "*" : {
        "sourcemap:write" : true,
        "event:write" : true,
        "config_agent:read" : true
      }
    }

...

Apart from providing appropriate error messages in the UI when certain privileges are missing, documentation will also be added for the APM Server and for the public APM app API.