diff --git a/README.md b/README.md index df2b4c8..6ddb80a 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,14 @@ Experimenting with Kubernetes protobufs. -## Protobufs +## Build Dependencies -### Download +- [gron](https://github.com/tomnomnom/gron) +- [just](https://github.com/casey/just) +- [sd](https://github.com/chmln/sd) +- [jq](https://stedolan.github.io/jq/) -Get protos by extracting them from Kubernetes releases: +## Protobufs +We get protos by extracting them from pinned Kubernetes releases: - https://github.com/kubernetes/api/releases - https://github.com/kubernetes/apimachinery/releases @@ -12,33 +16,19 @@ Get protos by extracting them from Kubernetes releases: - https://github.com/kubernetes/kube-aggregator/releases - https://github.com/kubernetes/metrics/releases -```bash -# In `protos/` -VERSION=1.22.0 -for x in api apimachinery apiextensions-apiserver kube-aggregator metrics; do - mkdir ./$x -p; - curl -sSL https://github.com/kubernetes/$x/archive/refs/tags/kubernetes-$VERSION.tar.gz | tar xzf - -C ./$x/ --strip-components=1; - fd -e proto -x sh -c "mkdir -p k8s.io/'{//}'; mv '{}' k8s.io/'{}'" ';' . ./$x; - rm -rf ./$x; -done -``` +We then do minor transforms on top of that to prepare for building. +Results of this step is committed already. But to run, invoke `just protos` -### Patch +## Openapi +To complement the protos with generic information, we also download the swagger schema, patch it, and transform it as described below. -Removing `k8s.io.`: +Results of this step is committed already. But to run, invoke `just swagger`. -```bash -fd -e proto -x sd 'k8s\.io\.(.+);' '$1;' {} -fd -e proto -x sd 'import "k8s\.io/(.+)";' 'import "$1";' {} -mv protos/k8s.io/* protos/ -rmdir protos/k8s.io/ -``` -### Generate -Refresh input paths to generate, then build: +## Building +With all dependent `swagger` and `protos` files built, run: ```bash -fd -e proto | sort > protos.list cargo build ``` @@ -68,7 +58,7 @@ See [`prost_build`](https://github.com/tokio-rs/prost/blob/32bc87cd0b7301f6af1a3 [`FileDescriptorSet`]: https://github.com/tokio-rs/prost/blob/32bc87cd0b7301f6af1a338e9afd7717d0f42ca9/prost-types/src/protobuf.rs#L1-L7 -## OpenAPI +## OpenAPI Strategy We need to use `swagger.json` to fill in some information. @@ -91,28 +81,6 @@ We should be able to find the following: - May also have paths for all namespaces for some verbs (e.g., `list` all pods) - Subresource if path contains `/{name}/` (`/` after `{name}`) -### Download - -In `openapi/` - -```bash -VERSION=1.22.0 -curl -sSL -o swagger.json \ - https://raw.githubusercontent.com/kubernetes/kubernetes/v$VERSION/api/openapi-spec/swagger.json -``` - -### Bug Fix - -Fix path operation annotated with a `x-kubernetes-group-version-kind` that references a type that doesn't exist in the schema. (See [`k8s-openapi`](https://github.com/Arnavion/k8s-openapi/blob/445e89ec444ebb1c68e61361e64eec4c4a3f4785/k8s-openapi-codegen/src/fixups/upstream_bugs.rs#L9)). - -```bash -gron swagger.json \ -| perl -pe 's/(?<=kind = ")(Pod|Node|Service)(?:Attach|Exec|PortForward|Proxy)Options(?=")/$1/' \ -| gron -u \ -> swagger-patched.json -mv swagger-patched.json swagger.json -``` - ### Transforming Transform `swagger.json` to something easier to explore. diff --git a/justfile b/justfile new file mode 100644 index 0000000..c6fa61f --- /dev/null +++ b/justfile @@ -0,0 +1,62 @@ +VERSION := "1.22.0" + +default: + @just --list + +# download proto schemas from upstream +protos-sync: + #!/usr/bin/env bash + set -exuo pipefail + rm -rf protos && mkdir protos && cd protos + for x in api apimachinery apiextensions-apiserver kube-aggregator metrics; do + mkdir ./$x -p + curl -sSL https://github.com/kubernetes/$x/archive/refs/tags/kubernetes-{{VERSION}}.tar.gz | tar xzf - -C ./$x/ --strip-components=1 + fd -e proto -x sh -c "mkdir -p k8s.io/'{//}'; mv '{}' k8s.io/'{}'" ';' . ./$x + rm -rf ./$x + done + +# fix import paths in downloaded schemas +protos-patch: + #!/usr/bin/env bash + set -exuo pipefail + fd -e proto -x sd 'k8s\.io\.(.+);' '$1;' {} + fd -e proto -x sd 'import "k8s\.io/(.+)";' 'import "$1";' {} + mv protos/k8s.io/* protos/ + rmdir protos/k8s.io/ + +# Generate path list for prost +protos-list: + fd -e proto | sort > protos.list + +# Download and generate all protos dependent files +protos: protos-sync protos-patch protos-list + +# Download swagger +swagger-sync: + #!/usr/bin/env bash + set -exuo pipefail + curl -sSL -o openapi/swagger.json \ + https://raw.githubusercontent.com/kubernetes/kubernetes/v{{VERSION}}/api/openapi-spec/swagger.json + +# Fix patch operation +swagger-patch: + #!/usr/bin/env bash + set -exuo pipefail + cd openapi + # Fix path operation annotated with a `x-kubernetes-group-version-kind` that references a type that doesn't exist in the schema. + # See https://github.com/Arnavion/k8s-openapi/blob/445e89ec444ebb1c68e61361e64eec4c4a3f4785/k8s-openapi-codegen/src/fixups/upstream_bugs.rs#L9 + gron swagger.json \ + | perl -pe 's/(?<=kind = ")(Pod|Node|Service)(?:Attach|Exec|PortForward|Proxy)Options(?=")/$1/' \ + | gron -u \ + > swagger-patched.json + mv swagger-patched.json swagger.json + +# Generate api-resources from patched swagger +swagger-transform: + #!/usr/bin/env bash + set -exuo pipefail + cd openapi + jq -f list-resources.jq < swagger.json > api-resources.json + +# Download and generate all swagger dependent files +swagger: swagger-sync swagger-patch swagger-transform