Role-Based Access Control

Created on 2022-12-23 by Paweł Kosiec (@pkosiec)

Status
`PROPOSED`

Overview

Botkube exposes an ability to run arbitrary commands, especially those related to Kubernetes clusters. Currently, to execute any kubectl command, Botkube uses ClusterRole which comes with an installation. This ClusterRole is bound to a ServiceAccount used by a Botkube Pod. This raises security concerns.

Also, while Botkube supports per-channel kubectl execution configuration, users should be allowed to set per-user or group mapping for more complex scenarios. While initially we could stick to channel-based grouping, it would be great to have an extensible mechanism to cover future use cases.

Goal
Assumptions

Executors
Sources
Reference
Alternatives
- Configure RBAC mapping
- Executing commands with proper access rights
Consequences

Goal

Stop using ClusterRole assigned to the Botkube pod for executing commands.
Implement ability to configure command execution permissions for different user groups.
Add ability to map users from the communication platforms to Kubernetes permissions.
Ensure that the proposed approach can be used with external identity providers.
Support plugins.

Assumptions

RBAC support is plugin-specific. Not all executors are based on Kubernetes API.

Executors

This section describes how RBAC will be implemented for executors.

User and group identification

To identify users and groups, we have multiple options, which varies between communication platforms. Summarizing the options, the proposal suggests the following approaches:

User identification:
- User email
- Static value for a given configuration
Group identification:
- Channel name
- User groups (Slack, Mattermost), Roles (Discord). Not supported by MS Teams.
- Static value for a given configuration

We can support just a few options initially, and later add more based on user feedback. See the Consequences paragraph for more details.

To get user email, channel name or user group name, we would need to do additional API calls in bot logic. However, we can cache the fetched values.

Implementation details

Slack:

We need to add two new scopes to the Slack app: users:read and users:read.email
We need to do additional API calls to get email (user info for a given ID) and channel name (conversation info for a given ID)
Unfortunately Slack user groups are in paid plan, so I couldn't test them. Probably we will need one additional call to get user group details to get its name.

Mattermost:

We need to do additional API call to get user email based on its ID
Based on user ID we can get the groups. They are not included in the message data.

Discord:

We need to query user data as they are missing from the message object. Quoting the docs: "The field user won't be included in the member object attached to MESSAGE_CREATE and MESSAGE_UPDATE gateway events."

For each message we get role IDs.

&discordgo.MessageCreate{
  Message: &discordgo.Message{
    // ...
    Member: &discordgo.Member{
      // ...
      Roles: []string{
        "976789858670497805", // <-- this is what we can use
      },
    },
  },
}

Based on that we can query for a full role object to get its name.

b.api.AddHandler(func(s *discordgo.Session, m *discordgo.MessageCreate) {
    msg := discordMessage{
      Event: m,
    }
    roles, _ := b.api.GuildRoles(m.GuildID)
    // manually filter role by ID - unfortunately there's just list capability - couldn't find get
        // we can cache the roles as they are server-wide (or guild-wide, depends what term is used)
    // get role.Name - more: https://discord.com/developers/docs/topics/permissions#role-object

MS Teams:

We need to do additional API calls to get the user name email. Firstly, we need to retrieve user ID and/or channel ID, then get user details.
There are no user groups or roles in MS Teams we can use.

Configure RBAC mapping

Cluster Admin creates Roles and ClusterRoles, as well as the RoleBindings and ClusterRoleBindings. For example:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: read-only
subjects:
  - kind: Group # group binding
    name: developers
    apiGroup: rbac.authorization.k8s.io
  - kind: User # user binding
    name: foo@example.com
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: read-only
  apiGroup: rbac.authorization.k8s.io

Next, Botkube needs to be aware of such configuration. In order to achieve this, we'll extend our execution configuration. We would support multiple types of mapping:

user-email based (allows per-user access rights configuration)
channel-based (allows per-channel access rights configuration)
user-group based (for supported platforms we'll use user groups or roles; for MS Teams, we'll fallback to channel-based mapping)
static (allows to impersonate a given user or group every time a given executor is executed regardless the context)

NOTE: Username and groups prefix could be useful in the clusters where OIDC is configured. For example, see this blog post.

executors:
  'kubectl-read-only':
    botkube/kubectl:
      enabled: true
      # New field "context" which will be used by Botkube Default Executor
      context:
        rbac: &rbacCtx

            # Option 1: user impersonation using only email address
            user:
              type: Email # user email will be used as identifying subject without group
              prefix: "" # prefix added to the user email fetched from comm platform
            group:
              type: "" # equal to `type: Disabled` - no group mapping
              prefix: ""

            # Option 2: user impersonation using email address and channel name as a group
            user:
              type: Email # user email will be used as user name
              prefix: "" # prefix added to the user email fetched from comm platform
            group:
              type: ChannelName # channel name will be used as identifying subject
              prefix: "" # added to the channel name

            # Option 3: user impersonation using email address and communication platform user group as a group
            user:
              type: Email # user email will be used as user name
              prefix: "" # prefix added to the user email fetched from comm platform
            group:
              type: UserGroupName # channel name will be used as identifying subject
              prefix: "" # added to the user group name

            # Option 4 (default): static impersonation for a given username and groups
            user:
              type: Static # impersonate as a given user every time a given executor is executed
              static:
                value: "default"
            group: # Optional, if a static group impersonation should be used
              type: Static # impersonate as a given group every time a given executor is executed
              static:
                value: [ "developers" ] # groups to impersonate

      config:
        defaultNamespace: default
        restrictAccess: false

  'kubectl-pods-rw':
    botkube/kubectl:
      enabled: true
      context:
        rbac: &rbacCtx # no need to specify it again if it's the same as above
  'kubectl-deploy-rw':
    botkube/kubectl@v1.0.0:
      enabled: true
      context:
        rbac: &rbacCtx # no need to specify it again if it's the same as above

A few remarks:

Merging different mapping configuration won't besupported and will result in an error. The configuration will be validated during Botkube startup.
As Kubernetes impersonation requires an username when using group impersonation, the default user name for mapping will be set as a static default value, as shown in Option 4 above.
If the rbac context is not provided (as a result, rbac.user|group.type is empty), Botkube won't create a temporary Kubeconfig for a given plugin.
By default, the static mapping will be used, to keep the current Botkube behavior. During Botkube installation, we will create ClusterRole and ClusterRoleBinding resources for it. If users would like to use channel-based access rights, they will need to create Kubernetes resources manually and configure mappings in Botkube configuration.
At later point we can think of facilitating access rights configuration by managing Roles, ClusterRoles, ClusterRoleBindings and RoleBindings directly from Botkube.

Executing commands with proper access rights

To ensure compatibility with our plugin system, we'll use K8s API user impersonation to create a special short-living Kubeconfig to use for a given execution. This will be handled globally by Botkube for all plugins.

We'll extend our gPRC API for executors in the following way:

message ExecuteRequest {
  // Commands represents the exact command that was specified by the user.
  string command = 1;
  // Configs is a list of Executor configurations specified by users.
  repeated Config configs = 2;

  // New field
  ExecuteContext context = 3;
}

message ExecuteContext {
  string kubeconfig_path = 1;
}

This path may be used by the plugin to access Kubernetes cluster in a restricted way. Later we can introduce more fields to the context, such as user details extracted for a given command execution.

Plugin isolation

To ensure no plugin has access to the sensitive configuration data, like Slack or Discord tokens, we will modify the plugin manager to run command with in an isolated environment in the following way.

For each executor plugin, We will use chroot to limit the access to the filesystem. A given plugin will have an isolated directory with all dependencies (e.g. helm or kubectl binary). For each command execution Botkube plugin manager will:

create a temporary subdirectory inside the isolated directory,
create a temporary Kubeconfig file in the subdirectory,
pass the Kubeconfig path via gRPC API,
delete the temporary subdirectory after command execution.

This is possible as we have full control over how the command is run:

	for key, path := range bins {
    // ...

		//nolint:gosec // warns us about 'Subprocess launching with variable', but we are the one that created that variable.
		cmd := exec.Command(path)
		cmd.SysProcAttr = &syscall.SysProcAttr{
      Chroot:     tmpExecutionDir, // change root directory
			Credential: &syscall.Credential{
        // change user / group - if we considered to have also another user/group for plugins
        Uid: ...,
        Gid: ...,
      },
		}

		cli := plugin.NewClient(&plugin.ClientConfig{
			Cmd:              cmd,
      // ...
		})

    // ...
  }

For such temporary Kubeconfig we still use the same token as the Botkube installation. That means, user needs to trust each plugin before execution, as it can behave maliciously, and e.g. impersonate a cluster-admin with a modified Kubeconfig. Fortunately, Cluster Admin can prevent this, by defining tighter Botkube ClusterRole:

- apiGroups: [""]
  resources: ["groups"]
  verbs: ["impersonate"]
  resourceNames: ["developers", "testers"] # group names
- apiGroups: [""]
  resources: ["users"]
  verbs: ["impersonate"]
  resourceNames: ["dev@example.com"] # user names

Still, a harmful plugin can use any of the group names / users defined in the ClusterRole (if it somehow guesses them) and we cannot prevent this, if we pass the token with ability to impersonate.

Sources

For source plugins, we will use the same approach as for executors.

NOTE: Before you read this paragraph, get familiar with Executors section.

Configure RBAC mapping

Cluster Admin creates Roles, ClusterRoles, RoleBindings and ClusterRoleBindings. Then, to configure the mapping, Cluster Admin uses the same syntax as for Executor plugins:

sources:
  'plugin-based':
    botkube/cm-watcher:
      enabled: true
      context:
        rbac:

          # Option 1: static user impersonation and using channel name as a group
          user:
            type: Static # impersonate as a given user when starting source plugin
            static:
              value: "default"
          group:
            type: ChannelName # channel name will be used as identifying subject
            prefix: "" # added to the channel name

          # Option 2 (default): static impersonation for a given username and groups
          user:
            type: Static # impersonate as a given user when starting source plugin
            static:
              value: "default"
          group:
            type: Static # impersonate as a given group when starting source plugin
            static:
              value: [ "developers" ] # groups to impersonate

Initially, we will support only static mapping, with the channel-based group mapping as a nice-to-have feature.

Running source plugins with proper access rights

Same as for Executors, we will use Kubernetes user impersonation to create Kubeconfig for each source plugin startup. This will be handled globally by Botkube for all plugins.

We'll extend our gPRC API for sources in the following way:

message StreamRequest {
	// Configs is a list of Source configurations specified by users.
	repeated Config configs = 1;

  // New field
  SourceContext context = 3;
}

message SourceContext {
  string kubeconfig_path = 1;
}

This path may be used by the plugin to access Kubernetes cluster in a restricted way.

Plugin isolation

The source plugin isolation is the same as for executors. We will run the subprocess in an isolated environment, with a Kubeconfig file per each configuration.

Reference

See the RBAC Proof of concept to understand what code changes are needed to implement the production solution.

Alternatives

This section covers all alternative solutions that were considered during the design process.

Configure RBAC mapping

Initial mapping config was proposed in the following way:

executors:
  'kubectl-read-only':
    botkube/kubectl:
      enabled: true
      # New field "context" which will be used by Botkube Default Executor
      context:
        kubeconfig: &kubeconfigCtx

            # Option 1: user impersonation using only email address
            mapping:
              type: UserEmail # user email will be used as identifying subject without group
              usernamePrefix: "" # prefix added to the user email fetched from comm platform
              groupsPrefix: "" # not used

            # Option 2: user impersonation using email address and channel name as a group
            mapping:
              type: ChannelName # channel name will be used as identifying subject
              usernamePrefix: "" # prefix added to the user email fetched from comm platform
              groupsPrefix: "" # added to the channel name

            # Option 3: user impersonation using email address and communication platform user group as a group
            mapping:
              type: UserGroupName # user group name will be used as identifying subject
              usernamePrefix: "" # prefix added to the user email fetched from comm platform
              groupsPrefix: "" # added to the user group name

            # Option 4 (default): static impersonation
            mapping:
              type: Static # impersonate as a given group or user every time a given executor is executed
              username: "any" # username to impersonate
              groups: [ "developers" ] # groups to impersonate

      config:
        defaultNamespace: default
        restrictAccess: false

  'kubectl-pods-rw':
    botkube/kubectl:
      enabled: true
      context:
        kubeconfig: &kubeconfigCtx # no need to specify it again if it's the same as above
  'kubectl-deploy-rw':
    botkube/kubectl@v1.0.0:
      enabled: true
      context:
        kubeconfig: &kubeconfigCtx # no need to specify it again if it's the same as above

However, allowing mapping user and group separately gives us more flexibility. For example, we will be able to implement username mapping for users (for those who don't want to share email addresses with Botkube), and still getting group name from the channel name.

Executing commands with proper access rights

After identifying the user and/or group, instead of introspection, we could run executor as a short-living separate Pod with a dedicated ServiceAccount and (Cluster)RoleBinding. This way we could use Kubernetes RBAC to restrict access to the cluster, with the isolation provided by Kubernetes. However, this is not possible as our plugin system is not compatible:

While the plugin system is over RPC, it is currently only designed to work over a local reliable network. Plugins over a real network are not supported and will lead to unexpected behavior.

Source: https://github.com/hashicorp/go-plugin

Consequences

Once the proposal is accepted, the following changes will be made:

Update configuration:
- Introduce executors[name][pluginname]context and source[name][pluginname]context fields.
- Initially support for executors:
  - static mapping for both users and groups,
  - channel-based group mapping.
  Create a follow-up task to support additional mappings and monitor the community demand for it:
  - email-based and username-based user mapping,
  - user-group-based group mapping.
- For sources, initially support static user and group mapping. Nice to have: chabnel-based group mapping if the implementation won't be hard.
- Validate executor and source bindings configuration and return error if kubeconfig RBAC mapping is different for the same bound plugins.
- Update defaults to use static mapping.
- Enable creating optional ClusterRole for all users to keep previous behavior.
- Allow restricting access to specific groups and/or users for impersonation in Helm chart.
Handle Kubeconfig passing for executors:
- Generate and pass Kubeconfig as a part of Context for each execution of a Kubernetes-related executor.
- Modify plugin manager to run plugins in a restricted environment.
Refactor kubectl executor:
- Remove commands property and our custom policy engine
- Ensure the kubectl auth can-i command can be executed from a communication platform.
This can be done as a part of #841 issue.
Refactor Kubernetes source:
- Ensure we use proper Kubeconfig passed via gRPC API.

This can be done as a part of #840 issue.

Update documentation with the latest RBAC changes:
- Add required scopes to the communication apps (e.g. Slack) to identify users by email.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

2022-12-23-rbac.md

2022-12-23-rbac.md

Role-Based Access Control

Overview

Goal

Assumptions

Executors

User and group identification

Implementation details

Configure RBAC mapping

Executing commands with proper access rights

Plugin isolation

Sources

Configure RBAC mapping

Running source plugins with proper access rights

Plugin isolation

Reference

Alternatives

Configure RBAC mapping

Executing commands with proper access rights

Consequences

Files

2022-12-23-rbac.md

Latest commit

History

2022-12-23-rbac.md

File metadata and controls

Role-Based Access Control

Overview

Goal

Assumptions

Executors

User and group identification

Implementation details

Configure RBAC mapping

Executing commands with proper access rights

Plugin isolation

Sources

Configure RBAC mapping

Running source plugins with proper access rights

Plugin isolation

Reference

Alternatives

Configure RBAC mapping

Executing commands with proper access rights

Consequences