Skip to content

Commit

Permalink
move scripts into something that tracks script dependencies
Browse files Browse the repository at this point in the history
trying just here as well because we are already doing cool stuff with sd
and gron.

Signed-off-by: clux <[email protected]>
  • Loading branch information
clux committed Aug 22, 2021
1 parent d121f44 commit 9d83b1e
Show file tree
Hide file tree
Showing 2 changed files with 77 additions and 47 deletions.
62 changes: 15 additions & 47 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,44 +1,34 @@
Experimenting with Kubernetes protobufs.

## Protobufs
## Build Dependencies

### Download
- [gron](https://github.com/tomnomnom/gron)
- [just](https://github.com/casey/just)
- [sd](https://github.com/chmln/sd)
- [jq](https://stedolan.github.io/jq/)

Get protos by extracting them from Kubernetes releases:
## Protobufs
We get protos by extracting them from pinned Kubernetes releases:

- https://github.com/kubernetes/api/releases
- https://github.com/kubernetes/apimachinery/releases
- https://github.com/kubernetes/apiextensions-apiserver/releases
- https://github.com/kubernetes/kube-aggregator/releases
- https://github.com/kubernetes/metrics/releases

```bash
# In `protos/`
VERSION=1.22.0
for x in api apimachinery apiextensions-apiserver kube-aggregator metrics; do
mkdir ./$x -p;
curl -sSL https://github.com/kubernetes/$x/archive/refs/tags/kubernetes-$VERSION.tar.gz | tar xzf - -C ./$x/ --strip-components=1;
fd -e proto -x sh -c "mkdir -p k8s.io/'{//}'; mv '{}' k8s.io/'{}'" ';' . ./$x;
rm -rf ./$x;
done
```
We then do minor transforms on top of that to prepare for building.
Results of this step is committed already. But to run, invoke `just protos`

### Patch
## Openapi
To complement the protos with generic information, we also download the swagger schema, patch it, and transform it as described below.

Removing `k8s.io.`:
Results of this step is committed already. But to run, invoke `just swagger`.

```bash
fd -e proto -x sd 'k8s\.io\.(.+);' '$1;' {}
fd -e proto -x sd 'import "k8s\.io/(.+)";' 'import "$1";' {}
mv protos/k8s.io/* protos/
rmdir protos/k8s.io/
```

### Generate
Refresh input paths to generate, then build:
## Building
With all dependent `swagger` and `protos` files built, run:

```bash
fd -e proto | sort > protos.list
cargo build
```

Expand Down Expand Up @@ -68,7 +58,7 @@ See [`prost_build`](https://github.com/tokio-rs/prost/blob/32bc87cd0b7301f6af1a3
[`FileDescriptorSet`]: https://github.com/tokio-rs/prost/blob/32bc87cd0b7301f6af1a338e9afd7717d0f42ca9/prost-types/src/protobuf.rs#L1-L7


## OpenAPI
## OpenAPI Strategy

We need to use `swagger.json` to fill in some information.

Expand All @@ -91,28 +81,6 @@ We should be able to find the following:
- May also have paths for all namespaces for some verbs (e.g., `list` all pods)
- Subresource if path contains `/{name}/` (`/` after `{name}`)

### Download

In `openapi/`

```bash
VERSION=1.22.0
curl -sSL -o swagger.json \
https://raw.githubusercontent.com/kubernetes/kubernetes/v$VERSION/api/openapi-spec/swagger.json
```

### Bug Fix

Fix path operation annotated with a `x-kubernetes-group-version-kind` that references a type that doesn't exist in the schema. (See [`k8s-openapi`](https://github.com/Arnavion/k8s-openapi/blob/445e89ec444ebb1c68e61361e64eec4c4a3f4785/k8s-openapi-codegen/src/fixups/upstream_bugs.rs#L9)).

```bash
gron swagger.json \
| perl -pe 's/(?<=kind = ")(Pod|Node|Service)(?:Attach|Exec|PortForward|Proxy)Options(?=")/$1/' \
| gron -u \
> swagger-patched.json
mv swagger-patched.json swagger.json
```

### Transforming

Transform `swagger.json` to something easier to explore.
Expand Down
62 changes: 62 additions & 0 deletions justfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
VERSION := "1.22.0"

default:
@just --list

# download proto schemas from upstream
protos-sync:
#!/usr/bin/env bash
set -exuo pipefail
rm -rf protos && mkdir protos && cd protos
for x in api apimachinery apiextensions-apiserver kube-aggregator metrics; do
mkdir ./$x -p
curl -sSL https://github.com/kubernetes/$x/archive/refs/tags/kubernetes-{{VERSION}}.tar.gz | tar xzf - -C ./$x/ --strip-components=1
fd -e proto -x sh -c "mkdir -p k8s.io/'{//}'; mv '{}' k8s.io/'{}'" ';' . ./$x
rm -rf ./$x
done
# fix import paths in downloaded schemas
protos-patch:
#!/usr/bin/env bash
set -exuo pipefail
fd -e proto -x sd 'k8s\.io\.(.+);' '$1;' {}
fd -e proto -x sd 'import "k8s\.io/(.+)";' 'import "$1";' {}
mv protos/k8s.io/* protos/
rmdir protos/k8s.io/
# Generate path list for prost
protos-list:
fd -e proto | sort > protos.list

# Download and generate all protos dependent files
protos: protos-sync protos-patch protos-list

# Download swagger
swagger-sync:
#!/usr/bin/env bash
set -exuo pipefail
curl -sSL -o openapi/swagger.json \
https://raw.githubusercontent.com/kubernetes/kubernetes/v{{VERSION}}/api/openapi-spec/swagger.json
# Fix patch operation
swagger-patch:
#!/usr/bin/env bash
set -exuo pipefail
cd openapi
# Fix path operation annotated with a `x-kubernetes-group-version-kind` that references a type that doesn't exist in the schema.
# See https://github.com/Arnavion/k8s-openapi/blob/445e89ec444ebb1c68e61361e64eec4c4a3f4785/k8s-openapi-codegen/src/fixups/upstream_bugs.rs#L9
gron swagger.json \
| perl -pe 's/(?<=kind = ")(Pod|Node|Service)(?:Attach|Exec|PortForward|Proxy)Options(?=")/$1/' \
| gron -u \
> swagger-patched.json
mv swagger-patched.json swagger.json
# Generate api-resources from patched swagger
swagger-transform:
#!/usr/bin/env bash
set -exuo pipefail
cd openapi
jq -f list-resources.jq < swagger.json > api-resources.json
# Download and generate all swagger dependent files
swagger: swagger-sync swagger-patch swagger-transform

0 comments on commit 9d83b1e

Please sign in to comment.