Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add resource version section to api-concepts documentation #16910

Merged
merged 6 commits into from
Dec 3, 2019
Merged

Add resource version section to api-concepts documentation #16910

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
68ecd24
Add Resource Version semantics section to api concepts
jpbetz Oct 14, 2019
611b675
Clarify risks of going back in time, add details about compaction and…
jpbetz Oct 15, 2019
dc0244f
Apply suggestions from liggitt
jpbetz Oct 27, 2019
ba3cd31
remove pesudocode, apply feedback
jpbetz Oct 27, 2019
d04cb63
Fix typo
jpbetz Nov 17, 2019
10fabcd
Clarify equality rules
jpbetz Dec 2, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 62 additions & 1 deletion content/en/docs/reference/using-api/api-concepts.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ The verbs supported for each subresource will differ depending on the object - s

## Efficient detection of changes

To enable clients to build a model of the current state of a cluster, all Kubernetes object resource types are required to support consistent lists and an incremental change notification feed called a **watch**. Every Kubernetes object has a `resourceVersion` field representing the version of that resource as stored in the underlying database. When retrieving a collection of resources (either namespace or cluster scoped), the response from the server will contain a `resourceVersion` value that can be used to initiate a watch against the server. The server will return all changes (creates, deletes, and updates) that occur after the supplied `resourceVersion`. This allows a client to fetch the current state and then watch for changes without missing any updates. If the client watch is disconnected they can restart a new watch from the last returned `resourceVersion`, or perform a new collection request and begin again.
To enable clients to build a model of the current state of a cluster, all Kubernetes object resource types are required to support consistent lists and an incremental change notification feed called a **watch**. Every Kubernetes object has a `resourceVersion` field representing the version of that resource as stored in the underlying database. When retrieving a collection of resources (either namespace or cluster scoped), the response from the server will contain a `resourceVersion` value that can be used to initiate a watch against the server. The server will return all changes (creates, deletes, and updates) that occur after the supplied `resourceVersion`. This allows a client to fetch the current state and then watch for changes without missing any updates. If the client watch is disconnected they can restart a new watch from the last returned `resourceVersion`, or perform a new collection request and begin again. See [Resource Version Semantics](#resource-versions) for more detail.

For example:

Expand Down Expand Up @@ -617,3 +617,64 @@ Server Side Apply is a beta feature, so it is enabled by default. To turn this
you need to include the `--feature-gates ServerSideApply=false` flag when
starting `kube-apiserver`. If you have multiple `kube-apiserver` replicas, all
should have the same flag setting.

## Resource Versions

Resource versions are strings that identify the server's internal version of an object. Resource versions can be used by clients to determine when objects have changed, or to express data consistency requirements when getting, listing and watching resources. Resource versions must be treated as opaque by clients and passed unmodified back to the server. For example, clients must not assume resource versions are numeric, and may only compare two resource version for equality (i.e. must not compare resource versions for greater-than or less-than relationships).

### ResourceVersion in metadata

Clients find resource versions in resources, including the resources in watch events, and list responses returned from the server:

[v1.meta/ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.16/#objectmeta-v1-meta) - The `metadata.resourceVersion` of a resource instance identifies the resource version the instance was last modified at.

[v1.meta/ListMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.16/#listmeta-v1-meta) - The `metadata.resourceVersion` of a resource collection (i.e. a list response) identifies the resource version at which the list response was constructed.

### The ResourceVersion Parameter

The get, list and watch operations support the `resourceVersion` parameter.

The exact meaning of this parameter differs depending on the operation and the value of the resource version.

For get and list, the semantics of resource version are:

**Get:**

| resourceVersion unset | resourceVersion="0" | resourceVersion="{non-zero version}" |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: s/non-zero version/value other than 0/ to reinforce these are not necessarily numbers

|-----------------------|---------------------|--------------------------------------|
| Most Recent | Any | Not older than |
Copy link
Member

@liggitt liggitt Dec 3, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the "not older than" semantic for individual items needs clarification. If the server version is at rv=10, and an individual item is at rv=5, what will a get of the individual item with ?resourceVersion=8 do? Return the item at rv=5, because the server is at rv=10? Wait/return a "too large resource version" because the item is at rv=5? It's hard to tell from this description which to expect (it's especially confusing to request an individual item with resourceVersion=10 and get back a successful response of an item with resourceVersion=5)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll try adding this to the "not older than" semantic:" ... Note that this ensures only that the objects returned are no older than they were at the time of the provided resource version. The resource version in the ObjectMeta of individual object may be older than the provide resource version so long it is for the latest modification to the object at the time of the provided resource version."


**List:**

| paging | resourceVersion unset | resourceVersion="0" | resourceVersion="{non-zero version}" |
|-----------|-----------------------|---------------------|--------------------------------------|
| no limit | Most Recent | Any | Not older than |
| limit="n" | Most Recent | Any | Exact |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it worth including a row for the continue token behavior, since you can't set resourceVersion separately, and the version you get back is based on the resourceVersion embedded in the continue token?

limit unset, continue unset
limit="n", continue unset
limit="n", continue="<token>"



The meaning of the get and list semantics are:

- **Most Recent:** Return data at the most recent resource version. The returned data must be consistent (i.e. served from etcd via a quorum read).
- **Any:** Return data at any resource version. The newest available resource version is preferred, but strong consistency is not required; data at any resource version may be served. It is possible for the request to return data at a much older resource version that the client has previously observed, particularly in high availabiliy configurations, due to partitions or stale caches. Clients that cannot tolerate this should not use this semantic.
- **Not older than:** Return data at least as new as the provided resource version. The newest available resource version is preferred, but any data not older than this resource version may be served.
Copy link
Member

@liggitt liggitt Dec 3, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

data at least as new as the provided resource version ... any data not older than this resource version may be served

this specifically needs clarification for the individual get case

- **Exact:** Return data at the exact resource version provided.

For watch, the semantics of resource version are:

**Watch:**

| resourceVersion unset | resourceVersion="0" | resourceVersion="{non-zero version}" |
|-------------------------------------|----------------------------|--------------------------------------|
| Get State and Start at Most Recent | Get State and Start at Any | Start at Exact |

The meaning of the watch semantics are:

- **Get State and Start at Most Recent:** Start a watch at the most recent resource version, which must be consistent (i.e. served from etcd via a quorum read). To establish initial state, the watch begins with synthetic “Added” events of all resources instances that exist at the starting resource version. All following watch events are for all changes that occured after the resource version the watch started at.
- **Get State and Start at Any:** Warning: Watches initialize this way may return arbitrarily stale data! Please review this semantic before using it, and favor the other semantics where possible. Start a watch at any resource version, the most recent resource version available is preferred, but not required; any starting resource version is allowed. It is possible for the watch to start at a much older resource version that the client has previously observed, particularly in high availabiliy configurations, due to partitions or stale caches. Clients that cannot tolerate this should not start a watch with this semantic. To establish initial state, the watch begins with synthetic “Added” events for all resources instances that exist at the starting resource version. All following watch events are for all changes that occured after the resource version the watch started at.
- **Start at Exact:** Start a watch at an exact resource version. The watch events are for all changes after the provided resource version. Unlike "Get State and Start at Most Recent" and "Get State and Start at Any", the watch is not started with synthetic "Added" events for the provided resource version. The client is assumed to already have the initial state at the starting resource version since the client provided the resource version.

### "410 Gone" responses
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what does the server return if you get/list a version newer than the server has available? I guess the only way you would have done that is if you constructed a numeric resourceVersion yourself or used one across resources served from different storage backends, which you're not supposed to do

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Current behavior is:

I'll document.


Servers are not required to serve all older resource versions and may return a HTTP `410 (Gone)` status code if a client requests a resourceVersion older than the server has retained. Clients must be able to tolerate `410 (Gone)` responses. See [Efficient detection of changes](#efficient-detection-of-changes) for details on how to handle `410 (Gone)` responses when watching resources.

For example, the kube-apiserver periodically compacts old resource versions from etcd based on its `--etcd-compaction-interval` setting. Also, the kube-apiserver's watch cache keeps `--watch-cache-sizes` resource versions in each resource cache. It depends on if a request is served from cache on which one of these limits applies, but if a resource version is unavailable in the one that applies, a `410 (Gone)` will be returned by the kube-apiserver.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It depends on if a request is served from cache on which one of these limits applies, but if a resource version is unavailable in the one that applies

nit: consider rephrasing to something like "If a resourceVersion is requested outside the applicable limit (depending on whether a request is served from cache or not), a 410 (Gone) will be returned by the kube-apiserver."