January 23, 2023
- Background
- How is each user credentials persisted
- How is each token generated
- How should client side configure credentials
- How is the credentials and token exchanged/transported
- How is the token used and managed
- Possible minor improvement
etcd needs to handle token/credentials only when auth is enabled.
This post describes how etcd generates and manages the tokens. There are two kinds of tokens, which are simple and JWT. The simple token is designed
for development testing, so it isn't recommended to use simple token in production use cases. Please use JWT token in production. Use the flag --auth-token
to specify the token type, and the valid options are "simple" or "jwt".
No matter which token type you are going to use, you need to setup RBAC firstly.
Creating users using command etcdctl user add or client SDK. Specifically,
you need to set both username and password when creating a user.
etcd uses bcrypt.GenerateFromPassword
(see also v3_server.go#L490)
to generate a bcrypt hash of the password at the given cost specified in --bcrypt-cost. Eventually the hashed password is persisted in the
bbolt database.
Note if using TLS CN based auth, then no password is needed; accordingly nothing to save in this case.
etcd server generates a token when it successfully authenticates a user. The client side needs to provide both username and password to authenticate a user.
When authenticating a user, etcd server uses bcrypt.CompareHashAndPassword (see also store.go#L375) to compare a bcrypt hashed password with the given plaintext password. If both the username and password are correct, then etcd server generates a token per the token type you choose.
If simple token is configured, etcd server generates a simple token
in the format fmt.Sprintf("%s.%d", simpleTokenPrefix, index).
The simpleTokenPrefix is a random string of 16 bytes, and
the index is just the current consistent_index.
Note simple tokens are not cryptographically signed.
Note you need to start etcd with flags something like below to use JWT token,
--auth-token=jwt,ttl=300s,pub-key=/srv/jwt_RS256.pub,priv-key=/srv/jwt_RS256,sign-method=RS256
If JWT token is configured, etcd server generates a JWT token, which is cryptographically signed by the provided private key. Note etcd depends on golang-jwt/jwt to generate JWT tokens.
Refer to #signing-methods-and-key-types to learn more about Signing Methods and Key Types.
There are two ways for the client side to configure the credentials. The first way is to configure username and password. Note that it's independent of how the token is generated
on server side. In other words, the server side can generate a simple token or JWT based on the value configured for --auth-token.
The second way is to use TLS Common Name with the option --client-cert-auth=true. In this case, the client doesn't need a password for a user, accordingly the client doesn't need to
authenticate the user to get a token either. The server side will try to get the username from the field of Common Name (CN) from the client's certificate.
When adding a user, the client side populates AuthUserAddRequest, and the request is marshaled at client side and unmarshalled at server side by gRPC automatically.
When authenticating a user, the client side populates AuthenticateRequest, and the request is marshaled at client side and unmarshalled at server side by gRPC automatically. If the user is successfully authenticated, then the generated token is returned to the client side via AuthenticateResponse, which is marshaled at server side and unmarshalled at client side by gRPC automatically.
The transport layer is secured by TLS if configured, see client.go#L352 and client.go#L234.
When auth is enabled and the client already gets a valid token, it needs to get the token included in the context for each following gRPC request. See credentials.go#L118 and client.go#L296 for client side. See store.go#L1044-L1058 for server side.
Note if using TLS CN based auth, etcd server gets the username from the Common Name in peer's certificate.
etcd server parses and manages the token differently per the token type.
Note the simple tokens are stateful,
etcd caches all token-username mappings in memory.
The lifetime of each simple token is specified by --auth-token-ttl, which defaults to 300 seconds. etcd removes a token from the cache when it expires.
When etcd server receives a simple token, it just checks whether the token exists in the cache, and regards it as valid if it exists.
The JWT tokens are stateless, and etcd depends on jwt.Parse to
parse, validate, verify the signature of a JWT token and return the parsed token. Each JWT token's lifetime is specified in
--auth-token=jwt,ttl=300s,..., and defaults to 300 seconds.
Note --auth-token-ttl is only used by simple tokens. Personally I think it should be used by the JWT token as well. In order to be
backward compatible, JWT should use TTL below in descending priority order,
--auth-token=jwt,ttl=300s,...--auth-token-ttl- The default value 300s.
Please anyone feel free to raise an issue in etcd community and gets it resolved.