diff --git a/docs/design-docs/01-oci-support.md b/docs/design-docs/01-oci-support.md new file mode 100644 index 0000000000..11d925d04a --- /dev/null +++ b/docs/design-docs/01-oci-support.md @@ -0,0 +1,474 @@ +# OCI Support + +* Author(s): Louis Dejardin, @loudej +* Approver: \ + +> Every feature will need design sign off an PR approval from a core +> maintainer. If you have not got in touch with anyone yet, you can leave +> this blank and we will try to line someone up for you. + +## Why + +Systems that deal with packaging software or bundling configuration often have +an atomic, versionable artifact. This artifact can exist as a source of +truth independant from the source controlled content from which it was built. + +It is also very common for those packages to have an associated feed or repository +which can receive those packages as they are published, and make them available for +download as needed. In many companies, using `git` as the as the repository and source +of truth for production configuration comes with challenges. + +This design document proposes to add `OCI` as an alternative to `git` for publishing and +distributing Kpt config packages. As a packaging format, it is well understood and documented. +As a repository format, it leverages existing container registries for pushing and pulling +config as image content. For security and production configuration management, if a +company has practices for managing Docker container images in private registries, then +the same practices and security model can be applied to config package images in private +registries. + +https://github.com/GoogleContainerTools/kpt/issues/2300 + +## Design + +### Design Assumptions + +The first stage of OCI support comes from adding support for `oci` in the places +where `git` appears today. + +An image tag is used in the same way a git branch or tag would be used. + +An image digest is used in the same way a git commit would be used. + +The scope of a single image is one root package, with any number of optional sub-packages. + +The structure of the image is a single tar layer. The root `Kptfile` is in the base directory from the tar layer's point of view. It contains only the Kpt package files, no entrypoint or executables. + +A package image should not be confused with a container image. Container images are executable by software, like `Docker`, and package images are purely configuration data. + +### Config chages + +The `Kptfile` structures for `upstream` and `upstreamLock` have `oci` in addition to `git` properties. The `type` property also has the string `oci` added as an accepted value. + +```yaml +upstream: + type: oci + oci: + image: 'IMAGE:TAG' +upstreamLock: + type: oci + oci: + image: 'IMAGE:DIGEST' +``` + +New verions of `kpt` will support existing `Kptfile`. The file structures and `git` functionality is unchanged. + +Existing versions of `kpt` will support `Kptfile` with `upstream` based on `git` for the same reason. The structure and meaning of existing fields is not changed. + +Existing versions of `kpt` will not support `Kptfile` with `upstream` based on `oci`. The `type` value, and missing `git` information will fail validation. The `kpt` binary used will need to be upgraded. + +### Command changes + +### `kpt pkg get` + +The argument that determines upstream today is parsed into `repo`, `ref`, and `path`, and is implicitly a `git` location. + +To support `oci`, it will be necessary to extract different values in a way that's unambiguous. Unfortunately, OCI image names have no Uri prefix, and are indistinguishable from a valid path or file name. + +To solve this, using [Helm](https://helm.sh/docs/topics/registries/#other-subcommands) as an example, the prefix `oci://` can be used. This ensures that selecting `oci` protocol isn't accidental, and it won't collide with other location formats that may be added. + +```shell +# clone package as new folder +kpt pkg get oci://us-docker.pkg.dev/the-project-id/the-repo-name/the-package:v3 my-package +``` + +Because OCI image reference already has a convention for `image:tag` references, using `:v3` should be used instead of `@v3` for version. It will be more intuitive how it relates to the registry, and easier to cut and paste values. + +### `kpt pkg get` sub-packages + +It is possible to use `kpt pkg get` to add sub-packages to a target location. + +Syntax for a sub-package target location is unchanged, it's a normal filesystem path. + +Syntax for an OCI sub-package source location requires the ability to tell when an image name ends and a sub-package path inside that image begins. In `git` this requires an explicit `.git` extension at the transition, and in `.oci` this requires double slash. + +```shell +# clone sub-package as new sub-folder +kpt pkg get oci://us-docker.pkg.dev/the-project-id/the-repo-name/the-package//simple/example:v3 my-package/simple/my-example +``` + +### `kpt pkg update` + +The command for update is not changed, but when the `upstream` is `oci` then the `@VERSION` is used to change the `upstream` image's `tag` or `digest` value. + +To update to an image tag, `kpt pkg update @v14` and `kpt pkg update DIR@v14` will assign the `:v14` tag onto the upstream image. + +```yaml +upstream: + type: oci + oci: + image: us-docker.pkg.dev/the-project-id/the-repo-name/the-package:v14 +``` + +To update to an upstream digest, `kpt pkg update @sha256:{SHA256_HEX}` and `kpt pkg update DIR@sha256:{SHA256_HEX}` will assign `@sha256:{SHA256_HEX}` as the new upstream image digest. + +```yaml +upstream: + type: oci + oci: + image: us-docker.pkg.dev/the-project-id/the-repo-name/the-package@sha256:8815143a333cb9d2cb341f10b984b22f3b8a99fe +``` + +Calling `kpt pkg update` and `kpt pkg update DIR` will perform an update without changing the upstream image name. + +At that point, if the `upstream` is an `image:tag` that is to discover the current `image:digest` for tag, otherwise the `upstream` value for `image:digest` is used. In either case, the `upstreamLock` is changed to point at that new `image:digest`. + +The package contents of the old and new `upstreamLock` image digest are fetched to temp folders, and are the basis of the 3-way merge to update the target package. + +### `kpt pkg diff` + +The `kpt pkg diff` command is identical to `kpt pkg update` in the way that `[PKG_PATH@VERSION]` argument is mapped to OCI concepts. + +### Command additions + +Although it is possible to create and push an OCI image using a combination of commands like `tar` and `gcrane`, that +doesn't provide a very complete end to end experience. Because kpt would already be built with the same OCI go module used +by `gcrane`, it is not difficult to support additional commands to move pull and push package contents from local folders +to remote images and back. + +### ` kpt pkg pull` + +``` +Usage: kpt pkg pull oci://{IMAGE[:TAG|@sha256:DIGEST]} [DIR] + DIR Destination folder for image contents. Default folder name is the last part of the IMAGE path. + IMAGE[:TAG|@sha256:DIGEST] Name of image to pull contents from, with optional TAG or DIGEST. Default TAG is `Latest` +``` + +This command is the reverse of push. An image can be pulled from a repository to a local folder, modified, and pushed +back to the same location, same location with different TAG, or entirely different location. + +The target DIR is optional, following the conventions of `kpg pkg get`, and will default to the final image/path segment. + +`kpt pkg pull` works on git uri as well. This may be used, for example, to mirror a set of known blueprints into a private +oci registry. + +### ` kpt pkg push` + +``` +Usage: kpt pkg push [DIR@VERSION] [--origin oci://{IMAGE[:TAG]}] [--increment] + DIR@VERSION Folder containing package root Kptfile. Default is current directory. + Optional @VERSION changes tag or branch to push onto. Default is most recently pulled or pushed tag. + --origin Name of image to push contents onto, with optional TAG to assign to resulting commit. + Default is to use most recently pulled/pushed image. Required if Kptfile does not have an origin. + --increment Increase the version by 1 while pushing. Default is to leave the origin's TAG or DIR@VERSION unchanged. + The Kptfile's image TAG is also updated to the new value. +``` + +This command will `tar` the contents of the package into a single image layer, and push it into the OCI repository. For +Google Artifact Registry and Google Container Registry, the current `gcloud auth` SSO credentials are used. + +The simplest form of the command is `kpt pkg push` or `kpt pkg push DIR` which will push the current contents back to +the IMAGE:TAG location that was saved when `kpt pkg pull` was run. + +The synxax `kpt pkg push @VERSION` or `kpt pkg push DIR@VERSION` will push back to the image location it came from, but with a new TAG name or version. Examples are `kpt pkt push @draft` or `kpt pkg push @v4` + +If the Kptfile was not obtained by `kpt pkg pull` - for example it's a new package from `kpt pkg init` or `kpt pkg get` - then +the first `kpt pkg push` will require an `--origin IMAGE:TAG` option to provide the target location. It is only necessary on the first +call. + +Finally, if the IMAGE's TAG value is a valid version number, the `--increment` switch can be used to add 1 to the current value before pushing. + +In the simplest case a `v1` is changed to `v2`, and `1` is changed to `2`, but any TAG that is a valid semver (with optional leading 'v') will have the smallest part of the number incremented. So `v1.0` becomes `v1.1`, `v1.0.0` becomes `v1.0.1`, and `v4.1.9-alpha` becomes `v4.1.10-alpha` + +#### Comparison of `pkg get` and `pkg pull` + +Starting with a simple root package, and an orange variant with root as the upstream: + +``` +-- root + \-- orange {upstream: root} +``` + +The purpose of `kpt get` is to create a new leaf node. This is done by creating the initial copy of the new leaf package in a +local folder. This has the side-effects of altering the kptfile name, the upstream values to point at the source, and makes appropriate +changes to sub-package metadata. + +As an example, after running `kpt pkg get scheme://repo/root green` and `kpt pkg get scheme://repo/orange blue` the `green` and +`blue` local folder packages are appended to the inheritance tree like this: + +``` +-- root + \-- orange {upstream: root} + | `-- blue {upstream: orange} ** working copy in ./blue ** + \-- green {upstream: root} ** working copy in ./green ** +``` + +By comparison, `kpt pkg pull` does not create a new package node or identity - it only extracts a copy of existing package +contents to a working directory. In this example, if the user additionally ran `kpt pkg pull scheme://repo/root root` and +`kpt pkg pull scheme://repo/orange orange` the overall state would be this: + +``` +-- root ** working copy in ./root ** + \-- orange {upstream: root} ** working copy in ./orange ** + | `-- blue {upstream: orange} ** working copy in ./blue ** + \-- green {upstream: root} ** working copy in ./green ** +``` + +### Alternatives to push/pull + +There are several ways that pull and push could appear as commands. Those two names are very conventional, but +alternatives to consider could be: + +### `kpt pkg copy` + +``` +Usage: kpt pkg copy {SOURCE} {DEST} + SOURCE Package source location: a local DIR, or `oci://` image, or git repo and path + DEST Package destination: a local DIR, or `oci://` image. +``` + +Puts a copy of the SOURCE package at the DEST location. The package contents would be entirely unchanged by this operation (unlike `kpt pkg get`). + +To pull from remote image to local folder: + +``` +kpt pkg copy \ + oci://us-docker.pkg.dev/the-project-id/the-repo-name/the-package:v14 \ + the-package +``` + +To push from local folder to new remote image tag: + +``` +kpt pkg copy \ + the-package \ + oci://us-docker.pkg.dev/the-project-id/the-repo-name/the-package:v15 +``` + +To copy a package image from one OCI repo to another: + +``` +kpt pkg copy \ + oci://us-docker.pkg.dev/the-project-id/dev-blueprints/the-package:v25 \ + oci://us-docker.pkg.dev/the-project-id/prod-blueprints/the-package:v25 +``` + +To copy a package from a git location to an OCI repo: + +``` +kpt pkg copy \ + https://github.com/GoogleCloudPlatform/blueprints.git/catalog/gke@main \ + oci://us-docker.pkg.dev/the-project-id/gcp-catalog/gke:latest +``` + +## User Guide + +### Creating a package respository + +Before kpt packages can be pushed and pulled as OCI images, a suitable repository +must be created. Google Artifact Registry and Google Container Registry are both +excellent choices. + +```shell +# Choose names and locations +LOCATION="us" +PROJECT_ID="kpt-demo-73823" +REPOSITORY_NAME="blueprints" + +# Base name for any images in this repository +REPOSITORY="${LOCATION}-docker.pkg.dev/${PROJECT_ID}/${REPOSITORY_NAME}" + +# Create the repository +gcloud artifacts repositories create --location="${LOCATION}" --repository-format=docker --project="${PROJECT_ID}" "${REPOSITORY_NAME}" +``` + +### Creating and pushing a new package + +Creating a new package is no different. But when ready to publish, the `kpt pkg push` command +is used instead of source control operations. + +```shell +# A package in a new directory +mkdir hello-world +kpt pkg init hello-world --description="A simple blueprint" + +# Store the contents in the repository, tagged as v1 +kpt pkg push hello-world --image=${REPOSITORY}/hello-world:v1 + +# The local files are not needed any more, pushing has stored them all +rm -r hello-world +``` + +### Pulling and updating a package + +Because the package folder was discarded earlier, the `kpt pkg pull` command +is used to place the contents of a particular version at a location. These set of +commands may be run in `Cloud Build` steps as well, if you are automating the +publication of packages as part of a CI/CD process. + +```shell +# Recreate the folder and extract the pulled image +kpt pkg pull hello-world --image=${REPOSITORY}/hello-world:v1 + +# Add a sub-package from a git repo +kpt pkg get https://github.com/GoogleCloudPlatform/blueprints.git/catalog/bucket hello-world/my-bucket + +# Render to be sure contents are hydrated, and push to a new version tag +kpt pkg render hello-world +kpt pkg push hello-world --image=${REPOSITORY}/hello-world:v2 +``` + +Similar to container image tags, the package image tags like `:v1` and `:v2` above may be used any +number of ways based on your preferred workflows. The tag `:latest` is used by default if all +pulls and pushes should read from and overwrite the same location. Semantic tags like `:draft` and +environmental tags like `:dev`, `:qa`, and `:prod` may be also used. + +No matter what tags are used, the image repository and `kpt` cli will treat them as an alphanumeric +label. + +### Using OCI repository as an upstream + +In addition to providing storage for packages, an OCI registry may also +be used as a source of upstream images to clone. The `oci://` prefix on this +command is required to ensure + +```shell +# Clone the hello-world v1 blueprint into a new folder +kpt pkg get oci://${REPOSITORY}/hello-world:v1 greetings-planet + +# Push the results to the repository, using default `latest` tag in this example +kpt pkg push greetings-planet --image=${REPOSITORY}/greetings-planet +``` + +Looking in the `greetings-planet/Kptfile` at this point will show that +the `hello-world:v1` image is the `upstream`, and the `upstreamLock` will show +exactly the digest that this clone is up-to-date with. + +```yaml +apiVersion: kpt.dev/v1 +kind: Kptfile +metadata: + name: greetings-planet +upstream: + type: oci + oci: + image: us-docker.pkg.dev/kpt-demo-73823/blueprints/hello-world:v1 + updateStrategy: resource-merge +upstreamLock: + type: oci + oci: + image: us-docker.pkg.dev/kpt-demo-73823/blueprints/hello-world@sha256:1632e00af3fe858c5e3b3f9e75c16e6327449155 +``` + +### Adding a subpackage from an OCI upstream subfolder + +Often a folder inside a package is meant to be used as a way to create "more of the same". + +To use an OCI image subfolder as the source of a subpackage, the path is added in a +way that's distinct from the image itself. + +```shell +# Clone the hello-world v1 blueprint into a new folder +kpt pkg get oci://${REPOSITORY}/hello-world//my-bucket:v1 greetings-planet/another-bucket +``` + +The `greetings-planet` package will now contain both a `greetings-planet/my-bucket` as well as a +`greetings-planet/another-bucket` folder. The contents in locations will now both receive changes +when the upstream `hello-world/my-bucket` is updated. + +### Updating package with upstream changes + +The value of the upstream image tag is used to `kpt pkg update` to a specific version. +This works no matter if the tag appears to look like a version number or not. + +```shell +# Update the greetings-planet by applying any differences between the upstreamLock digest and the `v2` tag +kpt pkg update greetings-planet@v2 + +# Overwrite the `greetings-planet:latest` image with the folder contents +kpt pkg push greetings-planet --image=${REPOSITORY}/greetings-planet +``` + +The Kptfile will now show that the `upstream` and `upstreamLock` have both been changed. + +```yaml +apiVersion: kpt.dev/v1 +kind: Kptfile +metadata: + name: greetings-planet +upstream: + type: oci + oci: + image: us-docker.pkg.dev/kpt-demo-73823/blueprints/hello-world:v2 + updateStrategy: resource-merge +upstreamLock: + type: oci + oci: + image: us-docker.pkg.dev/kpt-demo-73823/blueprints/hello-world@sha256:a6f1ed69c6ab51e2a148f6d4926bccb24c843887 +``` + +Just like with a git upstream, it is also possible to `kpt pkg update` without providing a different +tag or version value. This is similar to pulling from a remote branch where the name of the branch does +change but the latest commit on that branch is a different hash. + +In that case the Kptfile `upstream` tag will not change, but if that tag has been overwritten with +new contents then the differences between `upstreamLock` and current contents will be applied to the local copy, +and the `upstreamLock` will be changed tot he current digest. + +### Updating package to a specific upstream push + +Much like you can `kpt pkg update` to a specific commit hash in git, you can update to an +exact image digest with OCI. This is an even more precise reference than by a version number because, +like git commits, the image digest is based on the package file contents and cannot be forged or altered. + +```shell +# Update instread to an exact image digest of the upstream location +kpt pkg update greetings-planet@sha256:3b42daa41102fa83bce07bd82a72edcd691868d6 +``` + +The resulting Kptfile in the local folder will look like this. + +```yaml +apiVersion: kpt.dev/v1 +kind: Kptfile +metadata: + name: greetings-planet +upstream: + type: oci + oci: + image: us-docker.pkg.dev/kpt-demo-73823/blueprints/hello-world@sha256:3b42daa41102fa83bce07bd82a72edcd691868d6 + updateStrategy: resource-merge +upstreamLock: + type: oci + oci: + image: us-docker.pkg.dev/kpt-demo-73823/blueprints/hello-world@sha256:3b42daa41102fa83bce07bd82a72edcd691868d6 +``` + +## Open Issues/Questions + +> Please list any open questions here in the following format: +> +> ### \ +> +> Resolution: Please list the resolution if resolved during the design process or +> specify __Not Yet Resolved__ + +### What additional container registries should be supported? + +The protocol and information is the same. It would mainly be a question +of how the credentials for the call are provided. + +### What commands on exising kpt binary will work on Kptfile with `oci` + +It may be possible `kpt` commands that to not process `upstream` structures +may not require update to work correctly. `kpt fn` and `kpt live` commands +should be tested to see how they behave. + +## Alternatives Considered + +If there is an industry precedent or alternative approaches please list them +here as well as citing *why* you decided not to pursue those paths. + +### \ + +Links and description of the approach, the pros and cons identified during the +design. diff --git a/e2e/live/end-to-end-test.sh b/e2e/live/end-to-end-test.sh index c5ed11102c..06f1c93786 100755 --- a/e2e/live/end-to-end-test.sh +++ b/e2e/live/end-to-end-test.sh @@ -437,11 +437,10 @@ ${BIN_DIR}/kpt live apply e2e/live/testdata/rg-test-case-1a > $OUTPUT_DIR/status # The ResourceGroup inventory CRD is automatically installed on the initial apply. assertContains "installing inventory ResourceGroup CRD" assertContains "namespace/rg-test-namespace unchanged" -assertContains "1 resource(s) applied. 0 created, 1 unchanged, 0 configured, 0 failed" assertContains "pod/pod-a created" assertContains "pod/pod-b created" assertContains "pod/pod-c created" -assertContains "3 resource(s) applied. 3 created, 0 unchanged, 0 configured, 0 failed" +assertContains "4 resource(s) applied. 3 created, 1 unchanged, 0 configured, 0 failed" wait 2 # Validate resources in the cluster # ConfigMap inventory with four inventory items. @@ -451,11 +450,10 @@ assertRGInventory "rg-test-namespace" "4" ${BIN_DIR}/kpt live apply e2e/live/testdata/rg-test-case-1a > $OUTPUT_DIR/status 2>&1 assertNotContains "installing inventory ResourceGroup CRD" # Not applied again assertContains "namespace/rg-test-namespace unchanged" -assertContains "1 resource(s) applied. 0 created, 1 unchanged, 0 configured, 0 failed" assertContains "pod/pod-a unchanged" assertContains "pod/pod-b unchanged" assertContains "pod/pod-c unchanged" -assertContains "3 resource(s) applied. 0 created, 3 unchanged, 0 configured, 0 failed" +assertContains "4 resource(s) applied. 0 created, 4 unchanged, 0 configured, 0 failed" wait 2 printResult @@ -529,12 +527,11 @@ printResult echo "[ResourceGroup] Testing initial apply dry-run" echo "kpt live apply --dry-run e2e/live/testdata/rg-test-case-1a" ${BIN_DIR}/kpt live apply --dry-run e2e/live/testdata/rg-test-case-1a > $OUTPUT_DIR/status -assertContains "namespace/rg-test-namespace created (dry-run)" -assertContains "1 resource(s) applied. 1 created, 0 unchanged, 0 configured, 0 failed (dry-run)" -assertContains "pod/pod-a created (dry-run)" -assertContains "pod/pod-b created (dry-run)" -assertContains "pod/pod-c created (dry-run)" -assertContains "3 resource(s) applied. 3 created, 0 unchanged, 0 configured, 0 failed" +assertContains "namespace/rg-test-namespace created" +assertContains "pod/pod-a created" +assertContains "pod/pod-b created" +assertContains "pod/pod-c created" +assertContains "4 resource(s) applied. 4 created, 0 unchanged, 0 configured, 0 failed" printResult # Test: Basic kpt live apply @@ -545,11 +542,10 @@ ${BIN_DIR}/kpt live apply e2e/live/testdata/rg-test-case-1a > $OUTPUT_DIR/status # The ResourceGroup CRD is already installed. assertNotContains "installing inventory ResourceGroup CRD" assertContains "namespace/rg-test-namespace unchanged" -assertContains "1 resource(s) applied. 0 created, 1 unchanged, 0 configured, 0 failed" assertContains "pod/pod-a created" assertContains "pod/pod-b created" assertContains "pod/pod-c created" -assertContains "3 resource(s) applied. 3 created, 0 unchanged, 0 configured, 0 failed" +assertContains "4 resource(s) applied. 3 created, 1 unchanged, 0 configured, 0 failed" wait 2 # Validate resources in the cluster # ConfigMap inventory with four inventory items. @@ -565,11 +561,10 @@ ${BIN_DIR}/kpt live apply link-to-rg-test-case-1a > $OUTPUT_DIR/status # The ResourceGroup CRD is already installed. assertNotContains "installing inventory ResourceGroup CRD" assertContains "namespace/rg-test-namespace unchanged" -assertContains "1 resource(s) applied. 0 created, 1 unchanged, 0 configured, 0 failed" assertContains "pod/pod-a unchanged" assertContains "pod/pod-b unchanged" assertContains "pod/pod-c unchanged" -assertContains "3 resource(s) applied. 0 created, 3 unchanged, 0 configured, 0 failed" +assertContains "4 resource(s) applied. 0 created, 4 unchanged, 0 configured, 0 failed" wait 2 # Validate resources in the cluster # ConfigMap inventory with four inventory items. @@ -602,14 +597,13 @@ echo "[ResourceGroup] Testing basic apply dry-run" echo "kpt live apply --dry-run e2e/live/testdata/rg-test-case-1b" cp -f e2e/live/testdata/rg-test-case-1a/Kptfile e2e/live/testdata/rg-test-case-1b ${BIN_DIR}/kpt live apply --dry-run e2e/live/testdata/rg-test-case-1b > $OUTPUT_DIR/status -assertContains "namespace/rg-test-namespace configured (dry-run)" -assertContains "1 resource(s) applied. 0 created, 0 unchanged, 1 configured, 0 failed (dry-run)" -assertContains "pod/pod-b configured (dry-run)" -assertContains "pod/pod-c configured (dry-run)" -assertContains "pod/pod-d created (dry-run)" -assertContains "3 resource(s) applied. 1 created, 0 unchanged, 2 configured, 0 failed (dry-run)" -assertContains "pod/pod-a pruned (dry-run)" -assertContains "1 resource(s) pruned, 0 skipped, 0 failed (dry-run)" +assertContains "namespace/rg-test-namespace configured" +assertContains "pod/pod-b configured" +assertContains "pod/pod-c configured" +assertContains "pod/pod-d created" +assertContains "4 resource(s) applied. 1 created, 0 unchanged, 3 configured, 0 failed" +assertContains "pod/pod-a pruned" +assertContains "1 resource(s) pruned, 0 skipped, 0 failed" wait 2 # Validate resources in the cluster # ConfigMap inventory with four inventory items. @@ -626,11 +620,10 @@ echo "kpt live apply e2e/live/testdata/rg-test-case-1b" ${BIN_DIR}/kpt live apply e2e/live/testdata/rg-test-case-1b > $OUTPUT_DIR/status assertNotContains "installing inventory ResourceGroup CRD" # CRD already installed assertContains "namespace/rg-test-namespace unchanged" -assertContains "1 resource(s) applied. 0 created, 1 unchanged, 0 configured, 0 failed" assertContains "pod/pod-b unchanged" assertContains "pod/pod-c unchanged" assertContains "pod/pod-d created" -assertContains "3 resource(s) applied. 1 created, 2 unchanged, 0 configured, 0 failed" +assertContains "4 resource(s) applied. 1 created, 3 unchanged, 0 configured, 0 failed" assertContains "pod/pod-a pruned" assertContains "1 resource(s) pruned, 0 skipped, 0 failed" wait 2 @@ -647,12 +640,11 @@ printResult echo "[ResourceGroup] Testing basic destroy dry-run" echo "kpt live destroy --dry-run e2e/live/testdata/rg-test-case-1b" ${BIN_DIR}/kpt live destroy --dry-run e2e/live/testdata/rg-test-case-1b > $OUTPUT_DIR/status -assertContains "pod/pod-d deleted (dry-run)" -assertContains "pod/pod-c deleted (dry-run)" -assertContains "pod/pod-b deleted (dry-run)" -assertContains "3 resource(s) deleted, 0 skipped (dry-run)" -assertContains "namespace/rg-test-namespace deleted (dry-run)" -assertContains "1 resource(s) deleted, 0 skipped (dry-run)" +assertContains "pod/pod-d deleted" +assertContains "pod/pod-c deleted" +assertContains "pod/pod-b deleted" +assertContains "namespace/rg-test-namespace deleted" +assertContains "4 resource(s) deleted, 0 skipped" # Validate resources NOT DESTROYED in the cluster assertPodExists "pod-b" "rg-test-namespace" assertPodExists "pod-c" "rg-test-namespace" @@ -667,9 +659,8 @@ ${BIN_DIR}/kpt live destroy e2e/live/testdata/rg-test-case-1b > $OUTPUT_DIR/stat assertContains "pod/pod-d deleted" assertContains "pod/pod-c deleted" assertContains "pod/pod-b deleted" -assertContains "3 resource(s) deleted, 0 skipped" assertContains "namespace/rg-test-namespace deleted" -assertContains "1 resource(s) deleted, 0 skipped" +assertContains "4 resource(s) deleted, 0 skipped" # Validate resources NOT in the cluster assertPodNotExists "pod-b" "rg-test-namespace" assertPodNotExists "pod-c" "rg-test-namespace" @@ -684,7 +675,7 @@ cat e2e/live/testdata/stdin-test/pods.yaml | ${BIN_DIR}/kpt live apply - > $OUTP assertContains "pod/pod-a created" assertContains "pod/pod-b created" assertContains "pod/pod-c created" -assertContains "3 resource(s) applied. 3 created, 0 unchanged, 0 configured, 0 failed" +assertContains "4 resource(s) applied. 3 created, 1 unchanged, 0 configured, 0 failed" printResult echo "cat e2e/live/testdata/stdin-test/pods.yaml | kpt live status -" cat e2e/live/testdata/stdin-test/pods.yaml | ${BIN_DIR}/kpt live status - > $OUTPUT_DIR/status 2>&1 @@ -698,7 +689,7 @@ cat e2e/live/testdata/stdin-test/pods.yaml | ${BIN_DIR}/kpt live destroy - > $OU assertContains "pod/pod-a deleted" assertContains "pod/pod-b deleted" assertContains "pod/pod-c deleted" -assertContains "3 resource(s) deleted, 0 skipped" +assertContains "4 resource(s) deleted, 0 skipped" printResult # Test: kpt live apply continue-on-error @@ -829,11 +820,10 @@ echo "kpt live apply e2e/live/testdata/migrate-case-1b" cp -f e2e/live/testdata/migrate-case-1a/Kptfile e2e/live/testdata/migrate-case-1b ${BIN_DIR}/kpt live apply e2e/live/testdata/migrate-case-1b > $OUTPUT_DIR/status assertContains "namespace/test-rg-namespace unchanged" -assertContains "1 resource(s) applied. 0 created, 1 unchanged, 0 configured, 0 failed" assertContains "pod/pod-b unchanged" assertContains "pod/pod-c unchanged" assertContains "pod/pod-d created" -assertContains "3 resource(s) applied. 1 created, 2 unchanged, 0 configured, 0 failed" +assertContains "4 resource(s) applied. 1 created, 3 unchanged, 0 configured, 0 failed" assertContains "pod/pod-a pruned" assertContains "1 resource(s) pruned, 0 skipped, 0 failed" wait 2 diff --git a/e2e/testdata/fn-eval/error-in-pipe/.expected/config.yaml b/e2e/testdata/fn-eval/error-in-pipe/.expected/config.yaml index 214804680f..77df92ae74 100644 --- a/e2e/testdata/fn-eval/error-in-pipe/.expected/config.yaml +++ b/e2e/testdata/fn-eval/error-in-pipe/.expected/config.yaml @@ -21,8 +21,11 @@ stdErr: | Stderr: "[error] /// : failed to configure function: input namespace cannot be empty" Exit code: 1 - + [RUNNING] "gcr.io/kpt-fn/dne" [FAIL] "gcr.io/kpt-fn/dne" in 0s - Error: Function image "gcr.io/kpt-fn/dne" doesn't exist \ No newline at end of file + Stderr: + "docker: Error response from daemon: manifest for gcr.io/kpt-fn/dne:latest not found: manifest unknown: Failed to fetch \"latest\" from request \"/v2/kpt-fn/dne/manifests/latest\"." + "See 'docker run --help'." + Exit code: 125 \ No newline at end of file diff --git a/e2e/testdata/fn-eval/exec-function-stderr/.expected/config.yaml b/e2e/testdata/fn-eval/exec-function-stderr/.expected/config.yaml new file mode 100644 index 0000000000..f45537b8a7 --- /dev/null +++ b/e2e/testdata/fn-eval/exec-function-stderr/.expected/config.yaml @@ -0,0 +1,24 @@ +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +testType: eval +stdErr: | + [RUNNING] "./function.sh" + [PASS] "./function.sh" in 0s + Stderr: + "Hello world 0!" + "Hello world 1!" + "Hello world 2!" + "Hello world 3!" + ...(18 line(s) truncated, use '--truncate-output=false' to disable) diff --git a/e2e/testdata/fn-eval/exec-function-stderr/.expected/diff.patch b/e2e/testdata/fn-eval/exec-function-stderr/.expected/diff.patch new file mode 100644 index 0000000000..cc985fa320 --- /dev/null +++ b/e2e/testdata/fn-eval/exec-function-stderr/.expected/diff.patch @@ -0,0 +1,21 @@ +diff --git a/resources.yaml b/resources.yaml +index e8ae6bb..297b99f 100644 +--- a/resources.yaml ++++ b/resources.yaml +@@ -15,7 +15,7 @@ apiVersion: apps/v1 + kind: Deployment + metadata: + name: nginx-deployment +- namespace: foo ++ namespace: bar + spec: + replicas: 3 + --- +@@ -23,6 +23,6 @@ apiVersion: custom.io/v1 + kind: Custom + metadata: + name: custom +- namespace: foo ++ namespace: bar + spec: + image: nginx:1.2.3 diff --git a/e2e/testdata/fn-eval/exec-function-stderr/.expected/exec.sh b/e2e/testdata/fn-eval/exec-function-stderr/.expected/exec.sh new file mode 100644 index 0000000000..4a180de52c --- /dev/null +++ b/e2e/testdata/fn-eval/exec-function-stderr/.expected/exec.sh @@ -0,0 +1,19 @@ +#! /bin/bash +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -eo pipefail + +kpt fn source \ + | kpt fn eval --exec ./function.sh diff --git a/e2e/testdata/fn-eval/exec-function-stderr/.krmignore b/e2e/testdata/fn-eval/exec-function-stderr/.krmignore new file mode 100644 index 0000000000..9d7a4007d6 --- /dev/null +++ b/e2e/testdata/fn-eval/exec-function-stderr/.krmignore @@ -0,0 +1 @@ +.expected diff --git a/e2e/testdata/fn-eval/exec-function-stderr/function.sh b/e2e/testdata/fn-eval/exec-function-stderr/function.sh new file mode 100755 index 0000000000..c9cb9d7bbe --- /dev/null +++ b/e2e/testdata/fn-eval/exec-function-stderr/function.sh @@ -0,0 +1,21 @@ +#! /usr/bin/env bash +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +sed -e 's/foo/bar/' + +for i in {0..20} +do + >&2 echo "Hello world $i!" +done diff --git a/e2e/testdata/live-apply/json-output/resources/fourth.yaml b/e2e/testdata/fn-eval/exec-function-stderr/resources.yaml similarity index 57% rename from e2e/testdata/live-apply/json-output/resources/fourth.yaml rename to e2e/testdata/fn-eval/exec-function-stderr/resources.yaml index 48e7567715..e8ae6bbaf6 100644 --- a/e2e/testdata/live-apply/json-output/resources/fourth.yaml +++ b/e2e/testdata/fn-eval/exec-function-stderr/resources.yaml @@ -11,27 +11,18 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. - apiVersion: apps/v1 kind: Deployment metadata: - name: fourth-nginx - namespace: json-output - annotations: - config.k8s.io/owning-inventory: json-output - config.kubernetes.io/depends-on: apps/namespaces/json-output/Deployment/third-nginx + name: nginx-deployment + namespace: foo spec: - replicas: 1 - selector: - matchLabels: - app: fourth-nginx - template: - metadata: - labels: - app: fourth-nginx - spec: - containers: - - name: nginx - image: nginx:1.14.2 - ports: - - containerPort: 80 \ No newline at end of file + replicas: 3 +--- +apiVersion: custom.io/v1 +kind: Custom +metadata: + name: custom + namespace: foo +spec: + image: nginx:1.2.3 diff --git a/e2e/testdata/fn-eval/fn-success-with-stderr/pkg/.expected/results.yaml b/e2e/testdata/fn-eval/fn-success-with-stderr/pkg/.expected/results.yaml index cd73b3c490..67f0c37a90 100644 --- a/e2e/testdata/fn-eval/fn-success-with-stderr/pkg/.expected/results.yaml +++ b/e2e/testdata/fn-eval/fn-success-with-stderr/pkg/.expected/results.yaml @@ -5,6 +5,5 @@ metadata: exitCode: 0 items: - image: gcr.io/kpt-fn/starlark:v0.2 - stderr: | - function succeeded, reporting it on stderr + stderr: function succeeded, reporting it on stderr exitCode: 0 diff --git a/e2e/testdata/fn-eval/missing-fn-image/.expected/config.yaml b/e2e/testdata/fn-eval/missing-fn-image/.expected/config.yaml index cbef6dd615..fe98cda9a6 100644 --- a/e2e/testdata/fn-eval/missing-fn-image/.expected/config.yaml +++ b/e2e/testdata/fn-eval/missing-fn-image/.expected/config.yaml @@ -17,4 +17,4 @@ exitCode: 1 image: gcr.io/kpt-fn/dne # non-existing image args: namespace: staging -stdErr: 'Function image "gcr.io/kpt-fn/dne" doesn''t exist' +stdErr: 'gcr.io/kpt-fn/dne:latest not found' diff --git a/e2e/testdata/fn-render/exec-function-stderr/.expected/config.yaml b/e2e/testdata/fn-render/exec-function-stderr/.expected/config.yaml new file mode 100644 index 0000000000..9183295c7b --- /dev/null +++ b/e2e/testdata/fn-render/exec-function-stderr/.expected/config.yaml @@ -0,0 +1,24 @@ +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +allowExec: true +stdErr: | + [RUNNING] "./testdata/fn-render/exec-function-stderr/function.sh" + [PASS] "./testdata/fn-render/exec-function-stderr/function.sh" in 0s + Stderr: + "Hello world 0!" + "Hello world 1!" + "Hello world 2!" + "Hello world 3!" + ...(18 line(s) truncated, use '--truncate-output=false' to disable) diff --git a/e2e/testdata/fn-render/exec-function-stderr/.expected/diff.patch b/e2e/testdata/fn-render/exec-function-stderr/.expected/diff.patch new file mode 100644 index 0000000000..cc985fa320 --- /dev/null +++ b/e2e/testdata/fn-render/exec-function-stderr/.expected/diff.patch @@ -0,0 +1,21 @@ +diff --git a/resources.yaml b/resources.yaml +index e8ae6bb..297b99f 100644 +--- a/resources.yaml ++++ b/resources.yaml +@@ -15,7 +15,7 @@ apiVersion: apps/v1 + kind: Deployment + metadata: + name: nginx-deployment +- namespace: foo ++ namespace: bar + spec: + replicas: 3 + --- +@@ -23,6 +23,6 @@ apiVersion: custom.io/v1 + kind: Custom + metadata: + name: custom +- namespace: foo ++ namespace: bar + spec: + image: nginx:1.2.3 diff --git a/e2e/testdata/fn-render/exec-function-stderr/.krmignore b/e2e/testdata/fn-render/exec-function-stderr/.krmignore new file mode 100644 index 0000000000..9d7a4007d6 --- /dev/null +++ b/e2e/testdata/fn-render/exec-function-stderr/.krmignore @@ -0,0 +1 @@ +.expected diff --git a/e2e/testdata/fn-render/exec-function-stderr/Kptfile b/e2e/testdata/fn-render/exec-function-stderr/Kptfile new file mode 100644 index 0000000000..6f2fe1122f --- /dev/null +++ b/e2e/testdata/fn-render/exec-function-stderr/Kptfile @@ -0,0 +1,7 @@ +apiVersion: kpt.dev/v1 +kind: Kptfile +metadata: + name: app +pipeline: + mutators: + - exec: "./testdata/fn-render/exec-function-stderr/function.sh" diff --git a/e2e/testdata/fn-render/exec-function-stderr/function.sh b/e2e/testdata/fn-render/exec-function-stderr/function.sh new file mode 100755 index 0000000000..c9cb9d7bbe --- /dev/null +++ b/e2e/testdata/fn-render/exec-function-stderr/function.sh @@ -0,0 +1,21 @@ +#! /usr/bin/env bash +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +sed -e 's/foo/bar/' + +for i in {0..20} +do + >&2 echo "Hello world $i!" +done diff --git a/e2e/testdata/fn-render/exec-function-stderr/resources.yaml b/e2e/testdata/fn-render/exec-function-stderr/resources.yaml new file mode 100644 index 0000000000..e8ae6bbaf6 --- /dev/null +++ b/e2e/testdata/fn-render/exec-function-stderr/resources.yaml @@ -0,0 +1,28 @@ +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: nginx-deployment + namespace: foo +spec: + replicas: 3 +--- +apiVersion: custom.io/v1 +kind: Custom +metadata: + name: custom + namespace: foo +spec: + image: nginx:1.2.3 diff --git a/e2e/testdata/fn-render/exec-function-with-args/.expected/config.yaml b/e2e/testdata/fn-render/exec-function-with-args/.expected/config.yaml new file mode 100644 index 0000000000..1794ec551a --- /dev/null +++ b/e2e/testdata/fn-render/exec-function-with-args/.expected/config.yaml @@ -0,0 +1,15 @@ +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +allowExec: true diff --git a/e2e/testdata/fn-render/exec-function-with-args/.expected/diff.patch b/e2e/testdata/fn-render/exec-function-with-args/.expected/diff.patch new file mode 100644 index 0000000000..cc985fa320 --- /dev/null +++ b/e2e/testdata/fn-render/exec-function-with-args/.expected/diff.patch @@ -0,0 +1,21 @@ +diff --git a/resources.yaml b/resources.yaml +index e8ae6bb..297b99f 100644 +--- a/resources.yaml ++++ b/resources.yaml +@@ -15,7 +15,7 @@ apiVersion: apps/v1 + kind: Deployment + metadata: + name: nginx-deployment +- namespace: foo ++ namespace: bar + spec: + replicas: 3 + --- +@@ -23,6 +23,6 @@ apiVersion: custom.io/v1 + kind: Custom + metadata: + name: custom +- namespace: foo ++ namespace: bar + spec: + image: nginx:1.2.3 diff --git a/e2e/testdata/fn-render/exec-function-with-args/.krmignore b/e2e/testdata/fn-render/exec-function-with-args/.krmignore new file mode 100644 index 0000000000..9d7a4007d6 --- /dev/null +++ b/e2e/testdata/fn-render/exec-function-with-args/.krmignore @@ -0,0 +1 @@ +.expected diff --git a/e2e/testdata/fn-render/exec-function-with-args/Kptfile b/e2e/testdata/fn-render/exec-function-with-args/Kptfile new file mode 100644 index 0000000000..0d98dbb169 --- /dev/null +++ b/e2e/testdata/fn-render/exec-function-with-args/Kptfile @@ -0,0 +1,7 @@ +apiVersion: kpt.dev/v1 +kind: Kptfile +metadata: + name: app +pipeline: + mutators: + - exec: "sed -e 's/foo/bar/'" diff --git a/e2e/testdata/fn-render/exec-function-with-args/resources.yaml b/e2e/testdata/fn-render/exec-function-with-args/resources.yaml new file mode 100644 index 0000000000..e8ae6bbaf6 --- /dev/null +++ b/e2e/testdata/fn-render/exec-function-with-args/resources.yaml @@ -0,0 +1,28 @@ +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: nginx-deployment + namespace: foo +spec: + replicas: 3 +--- +apiVersion: custom.io/v1 +kind: Custom +metadata: + name: custom + namespace: foo +spec: + image: nginx:1.2.3 diff --git a/e2e/testdata/fn-render/exec-without-permissions/.expected/config.yaml b/e2e/testdata/fn-render/exec-without-permissions/.expected/config.yaml new file mode 100644 index 0000000000..840b9e0a49 --- /dev/null +++ b/e2e/testdata/fn-render/exec-without-permissions/.expected/config.yaml @@ -0,0 +1,16 @@ +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +exitCode: 1 +stdErr: 'Error: must run with `--allow-exec` option to allow running function binaries' \ No newline at end of file diff --git a/e2e/testdata/fn-render/exec-without-permissions/.krmignore b/e2e/testdata/fn-render/exec-without-permissions/.krmignore new file mode 100644 index 0000000000..9d7a4007d6 --- /dev/null +++ b/e2e/testdata/fn-render/exec-without-permissions/.krmignore @@ -0,0 +1 @@ +.expected diff --git a/e2e/testdata/fn-render/exec-without-permissions/Kptfile b/e2e/testdata/fn-render/exec-without-permissions/Kptfile new file mode 100644 index 0000000000..0d98dbb169 --- /dev/null +++ b/e2e/testdata/fn-render/exec-without-permissions/Kptfile @@ -0,0 +1,7 @@ +apiVersion: kpt.dev/v1 +kind: Kptfile +metadata: + name: app +pipeline: + mutators: + - exec: "sed -e 's/foo/bar/'" diff --git a/e2e/testdata/fn-render/exec-without-permissions/resources.yaml b/e2e/testdata/fn-render/exec-without-permissions/resources.yaml new file mode 100644 index 0000000000..e8ae6bbaf6 --- /dev/null +++ b/e2e/testdata/fn-render/exec-without-permissions/resources.yaml @@ -0,0 +1,28 @@ +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: nginx-deployment + namespace: foo +spec: + replicas: 3 +--- +apiVersion: custom.io/v1 +kind: Custom +metadata: + name: custom + namespace: foo +spec: + image: nginx:1.2.3 diff --git a/e2e/testdata/fn-render/fn-success-with-stderr/.expected/results.yaml b/e2e/testdata/fn-render/fn-success-with-stderr/.expected/results.yaml index cd73b3c490..67f0c37a90 100644 --- a/e2e/testdata/fn-render/fn-success-with-stderr/.expected/results.yaml +++ b/e2e/testdata/fn-render/fn-success-with-stderr/.expected/results.yaml @@ -5,6 +5,5 @@ metadata: exitCode: 0 items: - image: gcr.io/kpt-fn/starlark:v0.2 - stderr: | - function succeeded, reporting it on stderr + stderr: function succeeded, reporting it on stderr exitCode: 0 diff --git a/e2e/testdata/fn-render/missing-fn-image/.expected/config.yaml b/e2e/testdata/fn-render/missing-fn-image/.expected/config.yaml index fab1748ae6..7bee7e3546 100644 --- a/e2e/testdata/fn-render/missing-fn-image/.expected/config.yaml +++ b/e2e/testdata/fn-render/missing-fn-image/.expected/config.yaml @@ -13,4 +13,4 @@ # limitations under the License. exitCode: 1 -stdErr: 'Error: Function image "gcr.io/kpt-fn/dne" doesn''t exist' +stdErr: 'gcr.io/kpt-fn/dne:latest not found' diff --git a/e2e/testdata/live-apply/apply-depends-on/config.yaml b/e2e/testdata/live-apply/apply-depends-on/config.yaml index 0a7223d53e..b954988009 100644 --- a/e2e/testdata/live-apply/apply-depends-on/config.yaml +++ b/e2e/testdata/live-apply/apply-depends-on/config.yaml @@ -14,11 +14,18 @@ parallel: true +kptArgs: + - "--reconcile-timeout=2m" + stdOut: | deployment.apps/first-nginx created - 1 resource(s) applied. 1 created, 0 unchanged, 0 configured, 0 failed + deployment.apps/first-nginx reconcile pending + deployment.apps/first-nginx reconciled deployment.apps/second-nginx created - 1 resource(s) applied. 1 created, 0 unchanged, 0 configured, 0 failed + 2 resource(s) applied. 2 created, 0 unchanged, 0 configured, 0 failed + deployment.apps/second-nginx reconcile pending + deployment.apps/second-nginx reconciled + 2 resource(s) reconciled, 0 skipped, 0 failed to reconcile, 0 timed out inventory: - group: apps diff --git a/e2e/testdata/live-apply/crd-and-cr/config.yaml b/e2e/testdata/live-apply/crd-and-cr/config.yaml index 96141a5fda..a0e5ef4a3b 100644 --- a/e2e/testdata/live-apply/crd-and-cr/config.yaml +++ b/e2e/testdata/live-apply/crd-and-cr/config.yaml @@ -13,11 +13,19 @@ # limitations under the License. parallel: true + +kptArgs: + - "--reconcile-timeout=1m" + stdOut: | customresourcedefinition.apiextensions.k8s.io/customs.kpt.dev created - 1 resource(s) applied. 1 created, 0 unchanged, 0 configured, 0 failed + customresourcedefinition.apiextensions.k8s.io/customs.kpt.dev reconcile pending + customresourcedefinition.apiextensions.k8s.io/customs.kpt.dev reconciled custom.kpt.dev/cr created - 1 resource(s) applied. 1 created, 0 unchanged, 0 configured, 0 failed + 2 resource(s) applied. 2 created, 0 unchanged, 0 configured, 0 failed + custom.kpt.dev/cr reconcile pending + custom.kpt.dev/cr reconciled + 2 resource(s) reconciled, 0 skipped, 0 failed to reconcile, 0 timed out inventory: - group: apiextensions.k8s.io kind: CustomResourceDefinition diff --git a/e2e/testdata/live-apply/dry-run-with-install-rg/config.yaml b/e2e/testdata/live-apply/dry-run-with-install-rg/config.yaml index 5f8ab9af10..add5fb5e03 100644 --- a/e2e/testdata/live-apply/dry-run-with-install-rg/config.yaml +++ b/e2e/testdata/live-apply/dry-run-with-install-rg/config.yaml @@ -18,8 +18,8 @@ kptArgs: - "--dry-run" - "--install-resource-group" stdOut: | - deployment.apps/nginx-deployment created (dry-run) - 1 resource(s) applied. 1 created, 0 unchanged, 0 configured, 0 failed (dry-run) + deployment.apps/nginx-deployment created + 1 resource(s) applied. 1 created, 0 unchanged, 0 configured, 0 failed stdErr: | installing inventory ResourceGroup CRD. exitCode: 0 \ No newline at end of file diff --git a/e2e/testdata/live-apply/install-rg-on-apply/config.yaml b/e2e/testdata/live-apply/install-rg-on-apply/config.yaml index 6ea6d5df3c..b0019b96ee 100644 --- a/e2e/testdata/live-apply/install-rg-on-apply/config.yaml +++ b/e2e/testdata/live-apply/install-rg-on-apply/config.yaml @@ -14,6 +14,8 @@ parallel: false noResourceGroup: true +kptArgs: + - "--reconcile-timeout=1m" stdOut: | deployment.apps/nginx-deployment created 1 resource(s) applied. 1 created, 0 unchanged, 0 configured, 0 failed diff --git a/e2e/testdata/live-apply/json-output/config.yaml b/e2e/testdata/live-apply/json-output/config.yaml index 892c7b859a..2a5526d4a8 100644 --- a/e2e/testdata/live-apply/json-output/config.yaml +++ b/e2e/testdata/live-apply/json-output/config.yaml @@ -15,25 +15,29 @@ parallel: true kptArgs: - "--output=json" + - "--reconcile-timeout=2m" stdOut: | {"eventType":"resourceApplied","group":"","kind":"ConfigMap","name":"cm","namespace":"json-output","operation":"Created","timestamp":"","type":"apply"} - {"eventType":"resourceApplied","group":"apps","kind":"Deployment","name":"third-nginx","namespace":"json-output","operation":"Created","timestamp":"","type":"apply"} + {"eventType":"resourceReconciled","group":"","kind":"ConfigMap","name":"cm","namespace":"json-output","operation":"Pending","timestamp":"","type":"wait"} + {"eventType":"resourceReconciled","group":"","kind":"ConfigMap","name":"cm","namespace":"json-output","operation":"Reconciled","timestamp":"","type":"wait"} + {"eventType":"resourceApplied","group":"apps","kind":"Deployment","name":"nginx","namespace":"json-output","operation":"Created","timestamp":"","type":"apply"} {"configuredCount":0,"count":2,"createdCount":2,"eventType":"completed","failedCount":0,"serverSideCount":0,"timestamp":"","type":"apply","unchangedCount":0} - {"eventType":"resourceApplied","group":"apps","kind":"Deployment","name":"fourth-nginx","namespace":"json-output","operation":"Created","timestamp":"","type":"apply"} - {"configuredCount":0,"count":1,"createdCount":1,"eventType":"completed","failedCount":0,"serverSideCount":0,"timestamp":"","type":"apply","unchangedCount":0} + {"eventType":"resourceReconciled","group":"apps","kind":"Deployment","name":"nginx","namespace":"json-output","operation":"Pending","timestamp":"","type":"wait"} + {"eventType":"resourceReconciled","group":"apps","kind":"Deployment","name":"nginx","namespace":"json-output","operation":"Reconciled","timestamp":"","type":"wait"} {"eventType":"resourcePruned","group":"apps","kind":"Deployment","name":"second-nginx","namespace":"json-output","operation":"Pruned","timestamp":"","type":"prune"} - {"eventType":"completed","pruned":1,"skipped":0,"timestamp":"","type":"prune"} + {"eventType":"resourceReconciled","group":"apps","kind":"Deployment","name":"second-nginx","namespace":"json-output","operation":"Pending","timestamp":"","type":"wait"} + {"eventType":"resourceReconciled","group":"apps","kind":"Deployment","name":"second-nginx","namespace":"json-output","operation":"Reconciled","timestamp":"","type":"wait"} {"eventType":"resourcePruned","group":"apps","kind":"Deployment","name":"first-nginx","namespace":"json-output","operation":"Pruned","timestamp":"","type":"prune"} - {"eventType":"completed","pruned":1,"skipped":0,"timestamp":"","type":"prune"} + {"eventType":"completed","failed":0,"pruned":2,"skipped":0,"timestamp":"","type":"prune"} + {"eventType":"resourceReconciled","group":"apps","kind":"Deployment","name":"first-nginx","namespace":"json-output","operation":"Pending","timestamp":"","type":"wait"} + {"eventType":"resourceReconciled","group":"apps","kind":"Deployment","name":"first-nginx","namespace":"json-output","operation":"Reconciled","timestamp":"","type":"wait"} + {"eventType":"completed","failed":0,"reconciled":4,"skipped":0,"timeout":0,"timestamp":"","type":"wait"} + inventory: - kind: ConfigMap name: cm namespace: json-output - group: apps kind: Deployment - name: third-nginx - namespace: json-output - - group: apps - kind: Deployment - name: fourth-nginx + name: nginx namespace: json-output \ No newline at end of file diff --git a/e2e/testdata/live-apply/json-output/resources/third.yaml b/e2e/testdata/live-apply/json-output/resources/dep.yaml similarity index 87% rename from e2e/testdata/live-apply/json-output/resources/third.yaml rename to e2e/testdata/live-apply/json-output/resources/dep.yaml index 123b05bf0f..eb0129b45c 100644 --- a/e2e/testdata/live-apply/json-output/resources/third.yaml +++ b/e2e/testdata/live-apply/json-output/resources/dep.yaml @@ -15,19 +15,19 @@ apiVersion: apps/v1 kind: Deployment metadata: - name: third-nginx + name: nginx namespace: json-output annotations: - config.k8s.io/owning-inventory: json-output + config.kubernetes.io/depends-on: apps/namespaces/json-output/ConfigMap/cm spec: replicas: 1 selector: matchLabels: - app: third-nginx + app: nginx template: metadata: labels: - app: third-nginx + app: nginx spec: containers: - name: nginx diff --git a/e2e/testdata/live-apply/prune-depends-on/config.yaml b/e2e/testdata/live-apply/prune-depends-on/config.yaml index a30d858ff0..2f85051b72 100644 --- a/e2e/testdata/live-apply/prune-depends-on/config.yaml +++ b/e2e/testdata/live-apply/prune-depends-on/config.yaml @@ -13,13 +13,24 @@ # limitations under the License. parallel: true + +kptArgs: + - "--reconcile-timeout=1m" + stdOut: | configmap/cm created 1 resource(s) applied. 1 created, 0 unchanged, 0 configured, 0 failed + configmap/cm reconcile pending + configmap/cm reconciled deployment.apps/second-nginx pruned - 1 resource(s) pruned, 0 skipped, 0 failed + deployment.apps/second-nginx reconcile pending + deployment.apps/second-nginx reconciled deployment.apps/first-nginx pruned - 1 resource(s) pruned, 0 skipped, 0 failed + 2 resource(s) pruned, 0 skipped, 0 failed to prune + deployment.apps/first-nginx reconcile pending + deployment.apps/first-nginx reconciled + 3 resource(s) reconciled, 0 skipped, 0 failed to reconcile, 0 timed out + inventory: - kind: ConfigMap name: cm diff --git a/go.mod b/go.mod index 489136b8c9..42323e643a 100644 --- a/go.mod +++ b/go.mod @@ -11,13 +11,12 @@ require ( github.com/igorsobreira/titlecase v0.0.0-20140109233139-4156b5b858ac github.com/otiai10/copy v1.6.0 github.com/philopon/go-toposort v0.0.0-20170620085441-9be86dbd762f - github.com/posener/complete/v2 v2.0.1-alpha.12 github.com/spf13/cobra v1.2.1 - github.com/spf13/pflag v1.0.5 github.com/stretchr/testify v1.7.0 github.com/xlab/treeprint v0.0.0-20181112141820-a009c3971eca + golang.org/x/mod v0.5.1 gotest.tools v2.2.0+incompatible - k8s.io/api v0.22.3 + k8s.io/api v0.22.3 // indirect k8s.io/apiextensions-apiserver v0.22.2 k8s.io/apimachinery v0.22.3 k8s.io/cli-runtime v0.22.2 @@ -25,8 +24,7 @@ require ( k8s.io/klog/v2 v2.10.0 k8s.io/kube-openapi v0.0.0-20211109043139-026bd182f079 // indirect k8s.io/kubectl v0.22.2 - k8s.io/utils v0.0.0-20210820185131-d34e5cb4466e - sigs.k8s.io/cli-utils v0.26.1-0.20211020064957-d62b5c62002d + sigs.k8s.io/cli-utils v0.27.0 sigs.k8s.io/kustomize/api v0.8.11 sigs.k8s.io/kustomize/kyaml v0.13.1-0.20211203194734-cd2c6a1ad117 ) diff --git a/go.sum b/go.sum index 973bff733f..96ca9d5ca6 100644 --- a/go.sum +++ b/go.sum @@ -486,13 +486,11 @@ github.com/grpc-ecosystem/grpc-gateway v1.16.0/go.mod h1:BDjrQk3hbvj6Nolgz8mAMFb github.com/hashicorp/consul/api v1.1.0/go.mod h1:VmuI/Lkw1nC05EYQWNKwWGbkg+FbDBtguAZLlVdkD9Q= github.com/hashicorp/consul/sdk v0.1.1/go.mod h1:VKf9jXwCTEY1QZP2MOLRhb5i/I/ssyNV1vwHyQBF0x8= github.com/hashicorp/errwrap v0.0.0-20141028054710-7554cd9344ce/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4= -github.com/hashicorp/errwrap v1.0.0 h1:hLrqtEDnRye3+sgx6z4qVLNuviH3MR5aQ0ykNJa/UYA= github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4= github.com/hashicorp/go-cleanhttp v0.5.1/go.mod h1:JpRdi6/HCYpAwUzNwuwqhbovhLtngrth3wmdIIUrZ80= github.com/hashicorp/go-immutable-radix v1.0.0/go.mod h1:0y9vanUI8NX6FsYoO3zeMjhV/C5i9g4Q3DwcSNZ4P60= github.com/hashicorp/go-msgpack v0.5.3/go.mod h1:ahLV/dePpqEmjfWmKiqvPkv/twdG7iPBM1vqhUKIvfM= github.com/hashicorp/go-multierror v0.0.0-20161216184304-ed905158d874/go.mod h1:JMRHfdO9jKNzS/+BTlxCjKNQHg/jZAft8U7LloJvN7I= -github.com/hashicorp/go-multierror v1.0.0 h1:iVjPR7a6H0tWELX5NxNe7bYopibicUzc7uPribsnS6o= github.com/hashicorp/go-multierror v1.0.0/go.mod h1:dHtQlpGsu+cZNNAkkCN/P3hoUDHhCYQXV3UM06sGGrk= github.com/hashicorp/go-rootcerts v1.0.0/go.mod h1:K6zTfqpRlCUIjkwsN4Z+hiSfzSTQa6eBIzfwKfwNnHU= github.com/hashicorp/go-sockaddr v1.0.0/go.mod h1:7Xibr9yA9JjQq1JpNB2Vw7kxv8xerXegt+ozgdvDeDU= @@ -697,12 +695,7 @@ github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINE github.com/pkg/sftp v1.10.1/go.mod h1:lYOWFsE0bwd1+KfKJaKeuokY15vzFx25BLbzYYoAxZI= github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= -github.com/posener/complete v1.1.1 h1:ccV59UEOTzVDnDUEFdT95ZzHVZ+5+158q8+SJb2QV5w= github.com/posener/complete v1.1.1/go.mod h1:em0nMJCgc9GFtwrmVmEMR/ZL6WyhyjMBndrE9hABlRI= -github.com/posener/complete/v2 v2.0.1-alpha.12 h1:0wvkuDfHb5vSZlNBYgpEH4XQHpF46MjLPHav8XC77Nc= -github.com/posener/complete/v2 v2.0.1-alpha.12/go.mod h1://JlL91cS2JV7rOl6LVHrRqBXoBUecJu3ILQPgbJiMQ= -github.com/posener/script v1.0.4 h1:nSuXW5ZdmFnQIueLB2s0qvs4oNsUloM1Zydzh75v42w= -github.com/posener/script v1.0.4/go.mod h1:Rg3ijooqulo05aGLyGsHoLmIOUzHUVK19WVgrYBPU/E= github.com/pquerna/cachecontrol v0.0.0-20171018203845-0dec1b30a021/go.mod h1:prYjPmNq4d1NPVmpShWobRqXY3q7Vp+80DqgxxUrUIA= github.com/prometheus/client_golang v0.0.0-20180209125602-c332b6f63c06/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw= github.com/prometheus/client_golang v0.9.1/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw= @@ -943,6 +936,8 @@ golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.4.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.4.1/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= +golang.org/x/mod v0.5.1 h1:OJxoQ/rynoF0dcCdI7cLPktw/hR2cueqYfjm43oqK38= +golang.org/x/mod v0.5.1/go.mod h1:5OXOZSfqPIIbmVBIIKWRFfZjPR0E5r58TLhUjH0a2Ro= golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= @@ -1434,8 +1429,8 @@ rsc.io/sampler v1.3.0/go.mod h1:T1hPZKmBbMNahiBKFy5HrXp6adAjACjK9JXDnKaTXpA= sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.0.14/go.mod h1:LEScyzhFmoF5pso/YSeBstl57mOzx9xlU9n85RGrDQg= sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.0.15/go.mod h1:LEScyzhFmoF5pso/YSeBstl57mOzx9xlU9n85RGrDQg= sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.0.22/go.mod h1:LEScyzhFmoF5pso/YSeBstl57mOzx9xlU9n85RGrDQg= -sigs.k8s.io/cli-utils v0.26.1-0.20211020064957-d62b5c62002d h1:yxJZ6HujyxXTLuHuZ8/HkzWy6g+eTpslhUzAzPpA9dE= -sigs.k8s.io/cli-utils v0.26.1-0.20211020064957-d62b5c62002d/go.mod h1:8ll2fyx+bzjbwmwUnKBQU+2LDbMDsxy44DiDZ+drALg= +sigs.k8s.io/cli-utils v0.27.0 h1:BxI7lPNn0fBZa5g4UwR+ShJyL4CCxELA6tLHbr2YrpU= +sigs.k8s.io/cli-utils v0.27.0/go.mod h1:8ll2fyx+bzjbwmwUnKBQU+2LDbMDsxy44DiDZ+drALg= sigs.k8s.io/controller-runtime v0.10.1 h1:+eLHgY/VrJWnfg6iXUqhCUqNXgPH1NZeP9drNAAgWlg= sigs.k8s.io/controller-runtime v0.10.1/go.mod h1:CQp8eyUQZ/Q7PJvnIrB6/hgfTC1kBkGylwsLgOQi1WY= sigs.k8s.io/kustomize/api v0.8.11 h1:LzQzlq6Z023b+mBtc6v72N2mSHYmN8x7ssgbf/hv0H8= diff --git a/internal/cmdapply/cmdapply.go b/internal/cmdapply/cmdapply.go index ffc4ff9d26..c7611b0a58 100644 --- a/internal/cmdapply/cmdapply.go +++ b/internal/cmdapply/cmdapply.go @@ -25,17 +25,16 @@ import ( "github.com/GoogleContainerTools/kpt/internal/util/argutil" "github.com/GoogleContainerTools/kpt/internal/util/strings" "github.com/GoogleContainerTools/kpt/pkg/live" - "github.com/GoogleContainerTools/kpt/thirdparty/cli-utils/flagutils" - "github.com/GoogleContainerTools/kpt/thirdparty/cli-utils/printers" "github.com/spf13/cobra" v1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" "k8s.io/cli-runtime/pkg/genericclioptions" "k8s.io/kubectl/pkg/cmd/util" + "sigs.k8s.io/cli-utils/cmd/flagutils" "sigs.k8s.io/cli-utils/pkg/apply" "sigs.k8s.io/cli-utils/pkg/common" "sigs.k8s.io/cli-utils/pkg/inventory" - status "sigs.k8s.io/cli-utils/pkg/util/factory" + "sigs.k8s.io/cli-utils/pkg/printers" ) // NewRunner returns a command runner @@ -81,6 +80,8 @@ func NewRunner(ctx context.Context, factory util.Factory, "If true, install the inventory ResourceGroup CRD before applying.") c.Flags().BoolVar(&r.dryRun, "dry-run", false, "dry-run apply for the resources in the package.") + c.Flags().BoolVar(&r.printStatusEvents, "status-events", false, + "Print status events (always enabled for table output)") return r } @@ -106,6 +107,7 @@ type Runner struct { pruneTimeout time.Duration inventoryPolicyString string dryRun bool + printStatusEvents bool inventoryPolicy inventory.InventoryPolicy prunePropPolicy v1.DeletionPropagation @@ -208,33 +210,36 @@ func runApply(r *Runner, invInfo inventory.InventoryInfo, objs []*unstructured.U // Run the applier. It will return a channel where we can receive updates // to keep track of progress and any issues. - poller, err := status.NewStatusPoller(r.factory) - if err != nil { - return err - } invClient, err := inventory.NewInventoryClient(r.factory, live.WrapInventoryObj, live.InvToUnstructuredFunc) if err != nil { return err } - applier, err := apply.NewApplier(r.factory, invClient, poller) + applier, err := apply.NewApplier(r.factory, invClient) if err != nil { return err } ch := applier.Run(r.ctx, invInfo, objs, apply.Options{ - ServerSideOptions: r.serverSideOptions, - PollInterval: r.period, - ReconcileTimeout: r.reconcileTimeout, - // If we are not waiting for status, tell the applier to not - // emit the events. - EmitStatusEvents: r.reconcileTimeout != time.Duration(0) || r.pruneTimeout != time.Duration(0), + ServerSideOptions: r.serverSideOptions, + PollInterval: r.period, + ReconcileTimeout: r.reconcileTimeout, + EmitStatusEvents: true, // We are always waiting for reconcile. DryRunStrategy: dryRunStrategy, PrunePropagationPolicy: r.prunePropPolicy, PruneTimeout: r.pruneTimeout, InventoryPolicy: r.inventoryPolicy, }) + // Print the preview strategy unless the output format is json. + if dryRunStrategy.ClientOrServerDryRun() && r.output != printers.JSONPrinter { + if dryRunStrategy.ServerDryRun() { + fmt.Println("Dry-run strategy: server") + } else { + fmt.Println("Dry-run strategy: client") + } + } + // The printer will print updates from the channel. It will block // until the channel is closed. printer := printers.GetPrinter(r.output, r.ioStreams) - return printer.Print(ch, dryRunStrategy) + return printer.Print(ch, dryRunStrategy, r.printStatusEvents) } diff --git a/internal/cmdcomplete/complete.go b/internal/cmdcomplete/complete.go deleted file mode 100644 index d77a3d52dd..0000000000 --- a/internal/cmdcomplete/complete.go +++ /dev/null @@ -1,81 +0,0 @@ -// Copyright 2019 Google LLC -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -// Package cmdcomplete contains the completion command -package cmdcomplete - -import ( - "strings" - - "github.com/GoogleContainerTools/kpt/internal/util/cmdutil" - kptfilev1 "github.com/GoogleContainerTools/kpt/pkg/api/kptfile/v1" - "github.com/posener/complete/v2" - "github.com/posener/complete/v2/predict" - "github.com/spf13/cobra" - "github.com/spf13/pflag" -) - -type VisitFlags func(cmd *cobra.Command, flag *pflag.Flag, cc *complete.Command) - -// Complete returns a completion command for a cobra command -func Complete(cmd *cobra.Command, skipHelp bool, visitFlags VisitFlags) *complete.Command { - cc := &complete.Command{ - Flags: map[string]complete.Predictor{}, - Sub: map[string]*complete.Command{}, - } - if strings.Contains(cmd.Use, "DIR") { - // if usage contains directory, then use a file predictor - cc.Args = predict.Dirs("*") - } - - // add each command - if !skipHelp { - cc.Sub["help"] = &complete.Command{Sub: map[string]*complete.Command{}} - } - for i := range cmd.Commands() { - c := cmd.Commands()[i] - if c.Hidden || c.Deprecated != "" { - continue - } - name := strings.Split(c.Use, " ")[0] - cc.Sub[name] = Complete(c, true, visitFlags) - if !skipHelp { - cc.Sub["help"].Sub[name] = cc.Sub[name] - } - } - - cmd.Flags().VisitAll(func(flag *pflag.Flag) { - if visitFlags != nil { - // extension support for other commands that embed this one - visitFlags(cmd, flag, cc) - } - if flag.Name == "strategy" { - cc.Flags[flag.Name] = predict.Options(predict.OptValues(kptfilev1.UpdateStrategiesAsStrings()...)) - return - } - if flag.Name == "pattern" { - cc.Flags[flag.Name] = predict.Options(predict.OptValues("%k_%n.yaml")) - return - } - if flag.Name == "image" || flag.Shorthand == "i" { - fnImages := cmdutil.FetchFunctionImages() - cc.Flags[flag.Name] = predict.Options(predict.OptValues(fnImages...)) - cc.Flags[flag.Shorthand] = predict.Options(predict.OptValues(fnImages...)) - return - } - cc.Flags[flag.Name] = predict.Nothing - }) - - return cc -} diff --git a/internal/cmddestroy/cmddestroy.go b/internal/cmddestroy/cmddestroy.go index 5ae7930e17..928fa082af 100644 --- a/internal/cmddestroy/cmddestroy.go +++ b/internal/cmddestroy/cmddestroy.go @@ -23,15 +23,14 @@ import ( "github.com/GoogleContainerTools/kpt/internal/util/argutil" "github.com/GoogleContainerTools/kpt/internal/util/strings" "github.com/GoogleContainerTools/kpt/pkg/live" - "github.com/GoogleContainerTools/kpt/thirdparty/cli-utils/flagutils" - "github.com/GoogleContainerTools/kpt/thirdparty/cli-utils/printers" "github.com/spf13/cobra" "k8s.io/cli-runtime/pkg/genericclioptions" "k8s.io/kubectl/pkg/cmd/util" + "sigs.k8s.io/cli-utils/cmd/flagutils" "sigs.k8s.io/cli-utils/pkg/apply" "sigs.k8s.io/cli-utils/pkg/common" "sigs.k8s.io/cli-utils/pkg/inventory" - status "sigs.k8s.io/cli-utils/pkg/util/factory" + "sigs.k8s.io/cli-utils/pkg/printers" ) func NewRunner(ctx context.Context, factory util.Factory, @@ -60,6 +59,8 @@ func NewRunner(ctx context.Context, factory util.Factory, fmt.Sprintf("%q and %q.", flagutils.InventoryPolicyStrict, flagutils.InventoryPolicyAdopt)) c.Flags().BoolVar(&r.dryRun, "dry-run", false, "dry-run apply for the resources in the package.") + c.Flags().BoolVar(&r.printStatusEvents, "status-events", false, + "Print status events (always enabled for table output)") return r } @@ -81,6 +82,7 @@ type Runner struct { output string inventoryPolicyString string dryRun bool + printStatusEvents bool inventoryPolicy inventory.InventoryPolicy @@ -154,26 +156,31 @@ func (r *Runner) runE(c *cobra.Command, args []string) error { func runDestroy(r *Runner, inv inventory.InventoryInfo, dryRunStrategy common.DryRunStrategy) error { // Run the destroyer. It will return a channel where we can receive updates // to keep track of progress and any issues. - poller, err := status.NewStatusPoller(r.factory) - if err != nil { - return err - } invClient, err := inventory.NewInventoryClient(r.factory, live.WrapInventoryObj, live.InvToUnstructuredFunc) if err != nil { return err } - destroyer, err := apply.NewDestroyer(r.factory, invClient, poller) + destroyer, err := apply.NewDestroyer(r.factory, invClient) if err != nil { return err } options := apply.DestroyerOptions{ - InventoryPolicy: r.inventoryPolicy, - DryRunStrategy: dryRunStrategy, + InventoryPolicy: r.inventoryPolicy, + DryRunStrategy: dryRunStrategy, + EmitStatusEvents: true, } ch := destroyer.Run(context.Background(), inv, options) + // Print the preview strategy unless the output format is json. + if dryRunStrategy.ClientOrServerDryRun() && r.output != printers.JSONPrinter { + if dryRunStrategy.ServerDryRun() { + fmt.Println("Dry-run strategy: server") + } else { + fmt.Println("Dry-run strategy: client") + } + } // The printer will print updates from the channel. It will block // until the channel is closed. printer := printers.GetPrinter(r.output, r.ioStreams) - return printer.Print(ch, dryRunStrategy) + return printer.Print(ch, dryRunStrategy, r.printStatusEvents) } diff --git a/internal/cmdfndoc/cmdfndoc.go b/internal/cmdfndoc/cmdfndoc.go index ae3c11516b..11e9324581 100644 --- a/internal/cmdfndoc/cmdfndoc.go +++ b/internal/cmdfndoc/cmdfndoc.go @@ -43,6 +43,9 @@ func NewRunner(ctx context.Context, parent string) *Runner { } r.Command = c c.Flags().StringVarP(&r.Image, "image", "i", "", "kpt function image name") + _ = r.Command.RegisterFlagCompletionFunc("image", func(cmd *cobra.Command, args []string, toComplete string) ([]string, cobra.ShellCompDirective) { + return cmdutil.FetchFunctionImages(), cobra.ShellCompDirectiveDefault + }) cmdutil.FixDocs("kpt", parent, c) return r } diff --git a/internal/cmdget/cmdget.go b/internal/cmdget/cmdget.go index 43aee5340d..9cb9e14aba 100644 --- a/internal/cmdget/cmdget.go +++ b/internal/cmdget/cmdget.go @@ -53,6 +53,9 @@ func NewRunner(ctx context.Context, parent string) *Runner { c.Flags().StringVar(&r.strategy, "strategy", string(kptfilev1.ResourceMerge), "update strategy that should be used when updating this package -- must be one of: "+ strings.Join(kptfilev1.UpdateStrategiesAsStrings(), ",")) + _ = c.RegisterFlagCompletionFunc("strategy", func(cmd *cobra.Command, args []string, toComplete string) ([]string, cobra.ShellCompDirective) { + return kptfilev1.UpdateStrategiesAsStrings(), cobra.ShellCompDirectiveDefault + }) return r } diff --git a/internal/cmdmigrate/migratecmd.go b/internal/cmdmigrate/migratecmd.go index ff06693d70..51981dbf9f 100644 --- a/internal/cmdmigrate/migratecmd.go +++ b/internal/cmdmigrate/migratecmd.go @@ -231,16 +231,18 @@ func (mr *MigrateRunner) retrieveConfigMapInv(reader io.Reader, args []string) ( if err != nil { return nil, err } - cmInv, _, err := mr.cmLoader.InventoryInfo(objs) + cmInvObj, _, err := inventory.SplitUnstructureds(objs) if err != nil { + return nil, err + } + if cmInvObj == nil { // No ConfigMap inventory means the migration has already run before. - if _, ok := err.(inventory.NoInventoryObjError); ok { //nolint - fmt.Fprintln(mr.ioStreams.Out, "no ConfigMap inventory...completed") - } - } else { - fmt.Fprintf(mr.ioStreams.Out, "success (inventory-id: %s)\n", cmInv.ID()) + fmt.Fprintln(mr.ioStreams.Out, "no ConfigMap inventory...completed") + return nil, inventory.NoInventoryObjError{} } - return cmInv, err + cmInv := inventory.WrapInventoryInfoObj(cmInvObj) + fmt.Fprintf(mr.ioStreams.Out, "success (inventory-id: %s)\n", cmInv.ID()) + return cmInv, nil } // retrieveInvObjs returns the object references from the passed diff --git a/internal/cmdmigrate/migratecmd_test.go b/internal/cmdmigrate/migratecmd_test.go index e83eabf2d5..76c999e1a3 100644 --- a/internal/cmdmigrate/migratecmd_test.go +++ b/internal/cmdmigrate/migratecmd_test.go @@ -286,14 +286,14 @@ func TestKptMigrate_migrateObjs(t *testing.T) { }, "One migrate object is valid": { invObj: kptfileStr, - objs: []object.ObjMetadata{object.UnstructuredToObjMetaOrDie(pod1)}, + objs: []object.ObjMetadata{object.UnstructuredToObjMetadata(pod1)}, isError: false, }, "Multiple migrate objects are valid": { invObj: kptfileStr, objs: []object.ObjMetadata{ - object.UnstructuredToObjMetaOrDie(pod1), - object.UnstructuredToObjMetaOrDie(pod2), + object.UnstructuredToObjMetadata(pod1), + object.UnstructuredToObjMetadata(pod2), }, isError: false, }, diff --git a/internal/cmdpush/cmdpush.go b/internal/cmdpush/cmdpush.go index 33e9c8a49a..20654e9e05 100644 --- a/internal/cmdpush/cmdpush.go +++ b/internal/cmdpush/cmdpush.go @@ -121,7 +121,7 @@ func (r *Runner) preRunE(_ *cobra.Command, args []string) error { } if r.Origin != "" { - _, err := parse.ParseArgs(r.ctx, args, parse.Options{ + _, err := parse.ParseArgs(r.ctx, []string{r.Origin, path}, parse.Options{ SetOci: func(oci *kptfilev1.Oci) error { r.Push.Origin = remote.NewOciOrigin(oci) return nil diff --git a/internal/cmdrender/cmdrender.go b/internal/cmdrender/cmdrender.go index ee5bca1de4..2faf44be65 100644 --- a/internal/cmdrender/cmdrender.go +++ b/internal/cmdrender/cmdrender.go @@ -45,8 +45,10 @@ func NewRunner(ctx context.Context, parent string) *Runner { "path to a directory to save function results") c.Flags().StringVarP(&r.dest, "output", "o", "", fmt.Sprintf("output resources are written to provided location. Allowed values: %s|%s|", cmdutil.Stdout, cmdutil.Unwrap)) - c.Flags().StringVar(&r.imagePullPolicy, "image-pull-policy", string(fnruntime.AlwaysPull), + c.Flags().StringVar(&r.imagePullPolicy, "image-pull-policy", string(fnruntime.IfNotPresentPull), fmt.Sprintf("pull image before running the container. It must be one of %s, %s and %s.", fnruntime.AlwaysPull, fnruntime.IfNotPresentPull, fnruntime.NeverPull)) + c.Flags().BoolVar(&r.allowExec, "allow-exec", false, + "allow binary executable to be run during pipeline execution.") cmdutil.FixDocs("kpt", parent, c) r.Command = c return r @@ -61,6 +63,7 @@ type Runner struct { pkgPath string resultsDirPath string imagePullPolicy string + allowExec bool dest string Command *cobra.Command ctx context.Context @@ -98,10 +101,6 @@ func (r *Runner) preRunE(c *cobra.Command, args []string) error { } func (r *Runner) runE(c *cobra.Command, _ []string) error { - err := cmdutil.DockerCmdAvailable() - if err != nil { - return err - } var output io.Writer outContent := bytes.Buffer{} if r.dest != "" { @@ -114,9 +113,9 @@ func (r *Runner) runE(c *cobra.Command, _ []string) error { ResultsDirPath: r.resultsDirPath, Output: output, ImagePullPolicy: cmdutil.StringToImagePullPolicy(r.imagePullPolicy), + AllowExec: r.allowExec, } - err = executor.Execute(r.ctx) - if err != nil { + if err := executor.Execute(r.ctx); err != nil { return err } diff --git a/internal/cmdrender/executor.go b/internal/cmdrender/executor.go index b321a90ea3..8489e73db5 100644 --- a/internal/cmdrender/executor.go +++ b/internal/cmdrender/executor.go @@ -28,6 +28,7 @@ import ( "github.com/GoogleContainerTools/kpt/internal/printer" "github.com/GoogleContainerTools/kpt/internal/types" "github.com/GoogleContainerTools/kpt/internal/util/attribution" + "github.com/GoogleContainerTools/kpt/internal/util/cmdutil" "github.com/GoogleContainerTools/kpt/internal/util/printerutil" fnresult "github.com/GoogleContainerTools/kpt/pkg/api/fnresult/v1" kptfilev1 "github.com/GoogleContainerTools/kpt/pkg/api/kptfile/v1" @@ -37,12 +38,15 @@ import ( "sigs.k8s.io/kustomize/kyaml/yaml" ) +var errAllowedExecNotSpecified error = fmt.Errorf("must run with `--allow-exec` option to allow running function binaries") + // Executor hydrates a given pkg. type Executor struct { PkgPath string ResultsDirPath string Output io.Writer ImagePullPolicy fnruntime.ImagePullPolicy + AllowExec bool } // Execute runs a pipeline. @@ -62,6 +66,7 @@ func (e *Executor) Execute(ctx context.Context) error { pkgs: map[types.UniquePath]*pkgNode{}, fnResults: fnresult.NewResultList(), imagePullPolicy: e.ImagePullPolicy, + allowExec: e.AllowExec, } if _, err = hydrate(ctx, root, hctx); err != nil { @@ -162,6 +167,15 @@ type hydrationContext struct { // imagePullPolicy controls the image pulling behavior. imagePullPolicy fnruntime.ImagePullPolicy + + // allowExec determines if function binary executable are allowed + // to be run during pipeline execution. Running function binaries is a + // privileged operation, so explicit permission is required. + allowExec bool + + // bookkeeping to ensure docker command availability check is done once + // during rendering + dockerCheckDone bool } // @@ -426,11 +440,22 @@ func (pn *pkgNode) runValidators(ctx context.Context, hctx *hydrationContext, in if err != nil { return err } + var validator kio.Filter displayResourceCount := false if len(fn.Selectors) > 0 { displayResourceCount = true } - validator, err := fnruntime.NewContainerRunner(ctx, &fn, pn.pkg.UniquePath, hctx.fnResults, hctx.imagePullPolicy, displayResourceCount) + if fn.Exec != "" && !hctx.allowExec { + return errAllowedExecNotSpecified + } + if fn.Image != "" && !hctx.dockerCheckDone { + err := cmdutil.DockerCmdAvailable() + if err != nil { + return err + } + hctx.dockerCheckDone = true + } + validator, err = fnruntime.NewRunner(ctx, &fn, pn.pkg.UniquePath, hctx.fnResults, hctx.imagePullPolicy, displayResourceCount) if err != nil { return err } @@ -527,17 +552,28 @@ func pathRelToRoot(rootPkgPath, subPkgPath, resourcePath string) (relativePath s func fnChain(ctx context.Context, hctx *hydrationContext, pkgPath types.UniquePath, fns []kptfilev1.Function) ([]kio.Filter, error) { var runners []kio.Filter for i := range fns { + var err error + var runner kio.Filter fn := fns[i] - fn.Image = fnruntime.AddDefaultImagePathPrefix(fn.Image) displayResourceCount := false if len(fn.Selectors) > 0 { displayResourceCount = true } - r, err := fnruntime.NewContainerRunner(ctx, &fn, pkgPath, hctx.fnResults, hctx.imagePullPolicy, displayResourceCount) + if fn.Exec != "" && !hctx.allowExec { + return nil, errAllowedExecNotSpecified + } + if fn.Image != "" && !hctx.dockerCheckDone { + err := cmdutil.DockerCmdAvailable() + if err != nil { + return nil, err + } + hctx.dockerCheckDone = true + } + runner, err = fnruntime.NewRunner(ctx, &fn, pkgPath, hctx.fnResults, hctx.imagePullPolicy, displayResourceCount) if err != nil { return nil, err } - runners = append(runners, r) + runners = append(runners, runner) } return runners, nil } diff --git a/internal/cmdupdate/cmdupdate.go b/internal/cmdupdate/cmdupdate.go index 722cda2b3c..546d945c1a 100644 --- a/internal/cmdupdate/cmdupdate.go +++ b/internal/cmdupdate/cmdupdate.go @@ -53,6 +53,9 @@ func NewRunner(ctx context.Context, parent string) *Runner { "the update strategy that will be used when updating the package. This will change "+ "the default strategy for the package -- must be one of: "+ strings.Join(kptfilev1.UpdateStrategiesAsStrings(), ",")) + _ = c.RegisterFlagCompletionFunc("strategy", func(cmd *cobra.Command, args []string, toComplete string) ([]string, cobra.ShellCompDirective) { + return kptfilev1.UpdateStrategiesAsStrings(), cobra.ShellCompDirectiveDefault + }) cmdutil.FixDocs("kpt", parent, c) r.Command = c return r diff --git a/internal/docs/generated/fndocs/docs.go b/internal/docs/generated/fndocs/docs.go index f37a533c7f..e8d677b96a 100644 --- a/internal/docs/generated/fndocs/docs.go +++ b/internal/docs/generated/fndocs/docs.go @@ -239,6 +239,11 @@ Args: Flags: + --allow-exec: + Allow executable binaries to run as function. Note that executable binaries + can perform privileged operations on your system, so ensure that binaries + referred in the pipeline are trusted and safe to execute. + --image-pull-policy: If the image should be pulled before rendering the package(s). It can be set to one of always, ifNotPresent, never. If unspecified, always will be the diff --git a/internal/errors/resolver/live.go b/internal/errors/resolver/live.go index 00f42ff892..d01c6dbddd 100644 --- a/internal/errors/resolver/live.go +++ b/internal/errors/resolver/live.go @@ -19,9 +19,9 @@ import ( "github.com/GoogleContainerTools/kpt/internal/cmdutil" "github.com/GoogleContainerTools/kpt/internal/errors" "github.com/GoogleContainerTools/kpt/pkg/live" - "sigs.k8s.io/cli-utils/pkg/apply/taskrunner" "sigs.k8s.io/cli-utils/pkg/inventory" "sigs.k8s.io/cli-utils/pkg/manifestreader" + "sigs.k8s.io/cli-utils/pkg/print/common" ) //nolint:gochecknoinits @@ -42,14 +42,6 @@ package or automatically deleting omitted resources (pruning). Error: Package has multiple inventory object templates. The package should have one and only one inventory object template. -` - //nolint:lll - timeoutErrorMsg = ` -Error: Timeout after {{printf "%.0f" .err.Timeout.Seconds}} seconds waiting for {{printf "%d" (len .err.TimedOutResources)}} out of {{printf "%d" (len .err.Identifiers)}} resources to reach condition {{ .err.Condition}}:{{ printf "\n" }} - -{{- range .err.TimedOutResources}} -{{printf "%s/%s %s %s" .Identifier.GroupKind.Kind .Identifier.Name .Status .Message }} -{{- end}} ` resourceGroupCRDInstallErrorMsg = ` @@ -85,15 +77,13 @@ Details: ` unknownTypesMsg = ` -Error: {{ printf "%d" (len .err.GroupKinds) }} resource types could not be found in the cluster or as CRDs among the applied resources. +Error: {{ printf "%d" (len .err.GroupVersionKinds) }} resource types could not be found in the cluster or as CRDs among the applied resources. Resource types: -{{- range .err.GroupKinds}} +{{- range .err.GroupVersionKinds}} {{ printf "%s" .String }} {{- end}} ` - - TimeoutErrorExitCode = 3 ) // liveErrorResolver is an implementation of the ErrorResolver interface @@ -119,16 +109,6 @@ func (*liveErrorResolver) Resolve(err error) (ResolvedResult, bool) { }, true } - var timeoutError *taskrunner.TimeoutError - if errors.As(err, &timeoutError) { - return ResolvedResult{ - Message: ExecuteTemplate(timeoutErrorMsg, map[string]interface{}{ - "err": *timeoutError, - }), - ExitCode: TimeoutErrorExitCode, - }, true - } - var resourceGroupCRDInstallError *cmdutil.ResourceGroupCRDInstallError if errors.As(err, &resourceGroupCRDInstallError) { return ResolvedResult{ @@ -182,5 +162,14 @@ func (*liveErrorResolver) Resolve(err error) (ResolvedResult, bool) { }), }, true } + + var resultError *common.ResultError + if errors.As(err, &resultError) { + return ResolvedResult{ + Message: resultError.Error(), + ExitCode: 3, + }, true + } + return ResolvedResult{}, false } diff --git a/internal/errors/resolver/live_test.go b/internal/errors/resolver/live_test.go index 08164610df..b19f563055 100644 --- a/internal/errors/resolver/live_test.go +++ b/internal/errors/resolver/live_test.go @@ -17,14 +17,11 @@ package resolver import ( "strings" "testing" - "time" "github.com/GoogleContainerTools/kpt/internal/errors" "github.com/stretchr/testify/assert" "k8s.io/apimachinery/pkg/runtime/schema" - "sigs.k8s.io/cli-utils/pkg/apply/taskrunner" - "sigs.k8s.io/cli-utils/pkg/kstatus/status" - "sigs.k8s.io/cli-utils/pkg/object" + "sigs.k8s.io/cli-utils/pkg/manifestreader" ) func TestLiveErrorResolver(t *testing.T) { @@ -34,39 +31,21 @@ func TestLiveErrorResolver(t *testing.T) { }{ "nested timeoutError": { err: &errors.Error{ - Err: &taskrunner.TimeoutError{ - Identifiers: []object.ObjMetadata{ + Err: &manifestreader.UnknownTypesError{ + GroupVersionKinds: []schema.GroupVersionKind{ { - GroupKind: schema.GroupKind{ - Group: "apps", - Kind: "Deployment", - }, - Name: "test", - Namespace: "test-ns", - }, - }, - Condition: taskrunner.AllCurrent, - Timeout: 3 * time.Second, - TimedOutResources: []taskrunner.TimedOutResource{ - { - Identifier: object.ObjMetadata{ - GroupKind: schema.GroupKind{ - Group: "apps", - Kind: "Deployment", - }, - Name: "test", - Namespace: "test-ns", - }, - Status: status.InProgressStatus, - Message: "this is a test", + Group: "apps", + Version: "v1", + Kind: "Deployment", }, }, }, }, expected: ` -Error: Timeout after 3 seconds waiting for 1 out of 1 resources to reach condition AllCurrent: +Error: 1 resource types could not be found in the cluster or as CRDs among the applied resources. -Deployment/test InProgress this is a test +Resource types: +apps/v1, Kind=Deployment `, }, } diff --git a/internal/fnruntime/container.go b/internal/fnruntime/container.go index 4eca4358f7..092a97f9ea 100644 --- a/internal/fnruntime/container.go +++ b/internal/fnruntime/container.go @@ -15,6 +15,7 @@ package fnruntime import ( + "bufio" "bytes" "context" goerrors "errors" @@ -34,11 +35,10 @@ import ( type containerNetworkName string const ( - networkNameNone containerNetworkName = "none" - networkNameHost containerNetworkName = "host" - defaultLongTimeout time.Duration = 5 * time.Minute - defaultShortTimeout time.Duration = 5 * time.Second - dockerBin string = "docker" + networkNameNone containerNetworkName = "none" + networkNameHost containerNetworkName = "host" + defaultLongTimeout time.Duration = 5 * time.Minute + dockerBin string = "docker" AlwaysPull ImagePullPolicy = "Always" IfNotPresentPull ImagePullPolicy = "IfNotPresent" @@ -85,13 +85,6 @@ type ContainerFn struct { // It reads the input from the given reader and writes the output // to the provided writer. func (f *ContainerFn) Run(reader io.Reader, writer io.Writer) error { - // check and pull image before running to avoid polluting CLI - // output - err := f.prepareImage() - if err != nil { - return err - } - errSink := bytes.Buffer{} cmd, cancel := f.getDockerCmd() defer cancel() @@ -105,7 +98,7 @@ func (f *ContainerFn) Run(reader io.Reader, writer io.Writer) error { return &ExecError{ OriginalErr: exitErr, ExitCode: exitErr.ExitCode(), - Stderr: errSink.String(), + Stderr: filterDockerCLIOutput(&errSink), TruncateOutput: printer.TruncateOutput, } } @@ -113,7 +106,7 @@ func (f *ContainerFn) Run(reader io.Reader, writer io.Writer) error { } if errSink.Len() > 0 { - f.FnResult.Stderr = errSink.String() + f.FnResult.Stderr = filterDockerCLIOutput(&errSink) } return nil } @@ -135,8 +128,16 @@ func (f *ContainerFn) getDockerCmd() (*exec.Cmd, context.CancelFunc) { "--user", uidgid, "--security-opt=no-new-privileges", } - if f.ImagePullPolicy == NeverPull { + + switch f.ImagePullPolicy { + case NeverPull: args = append(args, "--pull", "never") + case AlwaysPull: + args = append(args, "--pull", "pull") + case IfNotPresentPull: + args = append(args, "--pull", "missing") + default: + args = append(args, "--pull", "missing") } for _, storageMount := range f.StorageMounts { args = append(args, "--mount", storageMount.String()) @@ -173,62 +174,6 @@ func NewContainerEnvFromStringSlice(envStr []string) *runtimeutil.ContainerEnv { return ce } -// prepareImage will check local images and pull it if it doesn't -// exist. -func (f *ContainerFn) prepareImage() error { - // If ImagePullPolicy is set to "never", we don't need to do anything here. - if f.ImagePullPolicy == NeverPull { - return nil - } - - // If ImagePullPolicy is set to "ifNotPresent", we scan the local images - // first. If there is a match, we just return. This can be useful for local - // development to prevent the remote image to accidentally override the - // local image when they use the same name and tag. - if f.ImagePullPolicy == IfNotPresentPull { - if foundInLocalCache := f.checkImageExistence(); foundInLocalCache { - return nil - } - } - - // If ImagePullPolicy is set to always (which is the default), we will try - // to pull the image regardless if the tag has been seen in the local cache. - // This can help to ensure we have the latest release for "moving tags" like - // v1 and v1.2. The performance cost is very minimal, since `docker pull` - // checks the SHA first and only pull the missing docker layer(s). - args := []string{"image", "pull", f.Image} - // setup timeout - timeout := defaultLongTimeout - if f.Timeout != 0 { - timeout = f.Timeout - } - ctx, cancel := context.WithTimeout(context.Background(), timeout) - defer cancel() - cmd := exec.CommandContext(ctx, dockerBin, args...) - output, err := cmd.CombinedOutput() - if err != nil { - return &ContainerImageError{ - Image: f.Image, - Output: string(output), - } - } - return nil -} - -// checkImageExistence returns true if the image does exist in -// local cache -func (f *ContainerFn) checkImageExistence() bool { - args := []string{"image", "inspect", f.Image} - ctx, cancel := context.WithTimeout(context.Background(), defaultShortTimeout) - defer cancel() - cmd := exec.CommandContext(ctx, dockerBin, args...) - if _, err := cmd.CombinedOutput(); err == nil { - // image exists locally - return true - } - return false -} - // AddDefaultImagePathPrefix adds default gcr.io/kpt-fn/ path prefix to image if only image name is specified func AddDefaultImagePathPrefix(image string) string { if !strings.Contains(image, "/") { @@ -250,3 +195,48 @@ func (e *ContainerImageError) Error() string { "Error: Function image %q doesn't exist remotely. If you are developing new functions locally, you can choose to set the image pull policy to ifNotPresent or never.\n%v", e.Image, e.Output) } + +// filterDockerCLIOutput filters out docker CLI messages +// from the given buffer. +func filterDockerCLIOutput(in io.Reader) string { + s := bufio.NewScanner(in) + var lines []string + + for s.Scan() { + txt := s.Text() + if !isdockerCLIoutput(txt) { + lines = append(lines, txt) + } + } + return strings.Join(lines, "\n") +} + +// isdockerCLIoutput is helper method to determine if +// the given string is a docker CLI output message. +// Example docker output: +// "Unable to find image 'gcr.io/kpt-fn/starlark:v0.3' locally" +// "v0.3: Pulling from kpt-fn/starlark" +// "4e9f2cdf4387: Already exists" +// "aafbf7df3ddf: Pulling fs layer" +// "aafbf7df3ddf: Verifying Checksum" +// "aafbf7df3ddf: Download complete" +// "6b759ab96cb2: Waiting" +// "aafbf7df3ddf: Pull complete" +// "Digest: sha256:c347e28606fa1a608e8e02e03541a5a46e4a0152005df4a11e44f6c4ab1edd9a" +// "Status: Downloaded newer image for gcr.io/kpt-fn/starlark:v0.3" +// +func isdockerCLIoutput(s string) bool { + if strings.Contains(s, ": Already exists") || + strings.Contains(s, ": Pulling fs layer") || + strings.Contains(s, ": Verifying Checksum") || + strings.Contains(s, ": Download complete") || + strings.Contains(s, ": Pulling from") || + strings.Contains(s, ": Waiting") || + strings.Contains(s, ": Pull complete") || + strings.Contains(s, "Digest: sha256") || + strings.Contains(s, "Status: Downloaded newer image") || + strings.Contains(s, "Unable to find image") { + return true + } + return false +} diff --git a/internal/fnruntime/container_test.go b/internal/fnruntime/container_test.go index 11161f1cd5..f6ac50ac7b 100644 --- a/internal/fnruntime/container_test.go +++ b/internal/fnruntime/container_test.go @@ -24,6 +24,7 @@ import ( "github.com/GoogleContainerTools/kpt/internal/fnruntime" "github.com/GoogleContainerTools/kpt/internal/printer" + fnresult "github.com/GoogleContainerTools/kpt/pkg/api/fnresult/v1" "github.com/stretchr/testify/assert" ) @@ -54,6 +55,9 @@ func TestContainerFn(t *testing.T) { instance := fnruntime.ContainerFn{ Ctx: printer.WithContext(ctx, printer.New(nil, errBuff)), Image: tt.image, + FnResult: &fnresult.Result{ + Image: tt.image, + }, } input := bytes.NewBufferString(tt.input) output := &bytes.Buffer{} diff --git a/internal/fnruntime/fnerrors_test.go b/internal/fnruntime/fnerrors_test.go index b022723b3b..b308d72120 100644 --- a/internal/fnruntime/fnerrors_test.go +++ b/internal/fnruntime/fnerrors_test.go @@ -15,6 +15,7 @@ package fnruntime import ( + "bytes" "testing" "github.com/stretchr/testify/assert" @@ -138,3 +139,76 @@ error message`, }) } } + +func TestDockerCLIOutputFilter(t *testing.T) { + + testcases := []struct { + name string + input string + expected string + }{ + { + name: "should filter docker CLI output successfully", + input: `Unable to find image 'gcr.io/kpt-fn/starlark:v0.3' locally +v0.3: Pulling from kpt-fn/starlark +4e9f2cdf4387: Already exists +aafbf7df3ddf: Pulling fs layer +aafbf7df3ddf: Verifying Checksum +aafbf7df3ddf: Download complete +aafbf7df3ddf: Pull complete +6b759ab96cb2: Waiting +Digest: sha256:c347e28606fa1a608e8e02e03541a5a46e4a0152005df4a11e44f6c4ab1edd9a +Status: Downloaded newer image for gcr.io/kpt-fn/starlark:v0.3 +`, + expected: "", + }, + { + name: "should filter docker messages and shouldn't truncate trailing lines", + input: `Unable to find image 'gcr.io/kpt-fn/starlark:v0.3' locally +v0.3: Pulling from kpt-fn/starlark +4e9f2cdf4387: Already exists +aafbf7df3ddf: Pulling fs layer +aafbf7df3ddf: Verifying Checksum +aafbf7df3ddf: Download complete +aafbf7df3ddf: Pull complete +6b759ab96cb2: Waiting +Digest: sha256:c347e28606fa1a608e8e02e03541a5a46e4a0152005df4a11e44f6c4ab1edd9a +Status: Downloaded newer image for gcr.io/kpt-fn/starlark:v0.3 +line before last line +lastline + +`, + expected: `line before last line +lastline +`, + }, + { + name: "should filter interleaved docker messages", + input: `firstline +Unable to find image 'gcr.io/kpt-fn/starlark:v0.3' locally +v0.3: Pulling from kpt-fn/starlark +4e9f2cdf4387: Already exists +aafbf7df3ddf: Pulling fs layer +aafbf7df3ddf: Verifying Checksum +line in the middle +aafbf7df3ddf: Download complete +aafbf7df3ddf: Pull complete +6b759ab96cb2: Waiting +Digest: sha256:c347e28606fa1a608e8e02e03541a5a46e4a0152005df4a11e44f6c4ab1edd9a +Status: Downloaded newer image for gcr.io/kpt-fn/starlark:v0.3 +lastline +`, + expected: `firstline +line in the middle +lastline`, + }, + } + + for _, tc := range testcases { + tc := tc + t.Run(tc.name, func(t *testing.T) { + s := filterDockerCLIOutput(bytes.NewBufferString(tc.input)) + assert.Equal(t, tc.expected, s) + }) + } +} diff --git a/internal/fnruntime/runner.go b/internal/fnruntime/runner.go index bdbaa1a3ed..051a533032 100644 --- a/internal/fnruntime/runner.go +++ b/internal/fnruntime/runner.go @@ -31,6 +31,7 @@ import ( "github.com/GoogleContainerTools/kpt/internal/types" fnresult "github.com/GoogleContainerTools/kpt/pkg/api/fnresult/v1" kptfilev1 "github.com/GoogleContainerTools/kpt/pkg/api/kptfile/v1" + "github.com/google/shlex" "sigs.k8s.io/kustomize/kyaml/fn/framework" "sigs.k8s.io/kustomize/kyaml/fn/runtime/runtimeutil" "sigs.k8s.io/kustomize/kyaml/kio" @@ -38,33 +39,62 @@ import ( "sigs.k8s.io/kustomize/kyaml/yaml" ) -// NewContainerRunner returns a kio.Filter given a specification of a container function +// NewRunner returns a kio.Filter given a specification of a function // and it's config. -func NewContainerRunner( +func NewRunner( ctx context.Context, f *kptfilev1.Function, pkgPath types.UniquePath, fnResults *fnresult.ResultList, imagePullPolicy ImagePullPolicy, displayResourceCount bool) (kio.Filter, error) { + config, err := newFnConfig(f, pkgPath) if err != nil { return nil, err } + if f.Image != "" { + f.Image = AddDefaultImagePathPrefix(f.Image) + } fnResult := &fnresult.Result{ - Image: f.Image, + Image: f.Image, + ExecPath: f.Exec, // TODO(droot): This is required for making structured results subpackage aware. // Enable this once test harness supports filepath based assertions. // Pkg: string(pkgPath), } - cfn := &ContainerFn{ - Path: pkgPath, - Image: f.Image, - ImagePullPolicy: imagePullPolicy, - Ctx: ctx, - FnResult: fnResult, - } - fltr := &runtimeutil.FunctionFilter{ - Run: cfn.Run, - FunctionConfig: config, + + fltr := &runtimeutil.FunctionFilter{FunctionConfig: config} + switch { + case f.Image != "": + cfn := &ContainerFn{ + Path: pkgPath, + Image: f.Image, + ImagePullPolicy: imagePullPolicy, + Ctx: ctx, + FnResult: fnResult, + } + fltr.Run = cfn.Run + case f.Exec != "": + var execArgs []string + // assuming exec here + s, err := shlex.Split(f.Exec) + if err != nil { + return nil, fmt.Errorf("exec command %q must be valid: %w", f.Exec, err) + } + execPath := f.Exec + if len(s) > 0 { + execPath = s[0] + } + if len(s) > 1 { + execArgs = s[1:] + } + eFn := &ExecFn{ + Path: execPath, + Args: execArgs, + FnResult: fnResult, + } + fltr.Run = eFn.Run + default: + return nil, fmt.Errorf("must specify `exec` or `image` to execute a function") } return NewFunctionRunner(ctx, fltr, pkgPath, fnResult, fnResults, true, displayResourceCount) } @@ -135,6 +165,7 @@ func (fr *FunctionRunner) Filter(input []*yaml.RNode) (output []*yaml.RNode, err if !fr.disableCLIOutput { pr.Printf("[PASS] %q in %v\n", fr.name, time.Since(t0).Truncate(time.Millisecond*100)) printFnResult(fr.ctx, fr.fnResult, printer.NewOpt()) + printFnStderr(fr.ctx, fr.fnResult.Stderr) } return output, err } @@ -315,16 +346,22 @@ func printFnResult(ctx context.Context, fnResult *fnresult.Result, opt *printer. // on kpt CLI. func printFnExecErr(ctx context.Context, fnErr *ExecError) { pr := printer.FromContextOrDie(ctx) - if len(fnErr.Stderr) > 0 { + printFnStderr(ctx, fnErr.Stderr) + pr.Printf(" Exit code: %d\n\n", fnErr.ExitCode) +} + +// printFnStderr prints given stdErr in a user friendly format on kpt CLI. +func printFnStderr(ctx context.Context, stdErr string) { + pr := printer.FromContextOrDie(ctx) + if len(stdErr) > 0 { errLines := &multiLineFormatter{ Title: "Stderr", - Lines: strings.Split(fnErr.Stderr, "\n"), + Lines: strings.Split(stdErr, "\n"), UseQuote: true, TruncateOutput: printer.TruncateOutput, } pr.Printf("%s", errLines.String()) } - pr.Printf(" Exit code: %d\n\n", fnErr.ExitCode) } // path (location) of a KRM resources is tracked in a special key in diff --git a/internal/fnruntime/runner_test.go b/internal/fnruntime/runner_test.go index 8c3e74942e..97fd2b1fec 100644 --- a/internal/fnruntime/runner_test.go +++ b/internal/fnruntime/runner_test.go @@ -18,12 +18,14 @@ package fnruntime import ( "bytes" + "context" "io/ioutil" "os" "path" "strings" "testing" + "github.com/GoogleContainerTools/kpt/internal/printer" "github.com/GoogleContainerTools/kpt/internal/types" kptfilev1 "github.com/GoogleContainerTools/kpt/pkg/api/kptfile/v1" "github.com/stretchr/testify/assert" @@ -577,3 +579,70 @@ file: assert.Equal(t, tc.expected, string(out)) } } + +func TestPrintFnStderr(t *testing.T) { + tests := map[string]struct { + input string // input + truncateOutput bool // whether to truncate output + expected string // expected result + }{ + "no output": { + input: ``, + truncateOutput: true, + expected: ``, + }, + "truncated output": { + input: `0 +1 +2 +3 +4 +5`, + truncateOutput: true, + expected: ` Stderr: + "0" + "1" + "2" + "3" + ...(2 line(s) truncated, use '--truncate-output=false' to disable) +`, + }, + "non-truncated output": { + input: `0 +1 +2 +3 +4 +5`, + truncateOutput: false, + expected: ` Stderr: + "0" + "1" + "2" + "3" + "4" + "5" +`, + }, + } + cleanupFunc := func() func() { + origTruncateOutput := printer.TruncateOutput + return func() { + printer.TruncateOutput = origTruncateOutput + } + }() + defer cleanupFunc() + for testName, tc := range tests { + t.Run(testName, func(t *testing.T) { + printer.TruncateOutput = tc.truncateOutput + out := &bytes.Buffer{} + err := &bytes.Buffer{} + ctx := printer.WithContext(context.Background(), printer.New(out, err)) + + printFnStderr(ctx, tc.input) + + assert.Equal(t, tc.expected, err.String()) + assert.Equal(t, "", out.String()) + }) + } +} diff --git a/internal/util/cmdutil/cmdutil.go b/internal/util/cmdutil/cmdutil.go index 8fda513e9d..fa5ffd752e 100644 --- a/internal/util/cmdutil/cmdutil.go +++ b/internal/util/cmdutil/cmdutil.go @@ -29,6 +29,7 @@ import ( "github.com/GoogleContainerTools/kpt/internal/fnruntime" "github.com/GoogleContainerTools/kpt/internal/util/httputil" "github.com/spf13/cobra" + "golang.org/x/mod/semver" "sigs.k8s.io/kustomize/kyaml/kio" "sigs.k8s.io/kustomize/kyaml/kio/kioutil" ) @@ -39,7 +40,7 @@ const ( Stdout = "stdout" Unwrap = "unwrap" dockerVersionTimeout time.Duration = 5 * time.Second - FunctionsCatalogURL = "https://catalog.kpt.dev/catalog.json" + FunctionsCatalogURL = "https://catalog.kpt.dev/catalog-v2.json" ) // FixDocs replaces instances of old with new in the docs for c @@ -196,18 +197,25 @@ func FetchFunctionImages() []string { return listImages(content) } +// fnName -> v. -> catalogEntry +type catalogV2 map[string]map[string]struct { + LatestPatchVersion string + Examples interface{} +} + // listImages returns the list of latest images from the input catalog content func listImages(content string) []string { var result []string - jsonData := map[string]map[string]interface{}{} + var jsonData catalogV2 err := json.Unmarshal([]byte(content), &jsonData) if err != nil { return result } for fnName, fnInfo := range jsonData { var latestVersion string - for version := range fnInfo { - if latestVersion < version { + for _, catalogEntry := range fnInfo { + version := catalogEntry.LatestPatchVersion + if semver.Compare(version, latestVersion) == 1 { latestVersion = version } } diff --git a/internal/util/cmdutil/cmdutil_test.go b/internal/util/cmdutil/cmdutil_test.go index 8fe1a91dcd..c73792b56c 100644 --- a/internal/util/cmdutil/cmdutil_test.go +++ b/internal/util/cmdutil/cmdutil_test.go @@ -316,30 +316,39 @@ func TestListImages(t *testing.T) { result := listImages(`{ "apply-setters": { "v0.1": { - "apply-setters-simple": { - "LocalExamplePath": "/apply-setters/v0.1/apply-setters-simple", - "RemoteExamplePath": "https://github.com/GoogleContainerTools/kpt-functions-catalog/tree/apply-setters/v0.1/examples/apply-setters-simple", - "RemoteSourcePath": "https://github.com/GoogleContainerTools/kpt-functions-catalog/tree/apply-setters/v0.1/functions/go/apply-setters" + "LatestPatchVersion": "v0.1.1", + "Examples": { + "apply-setters-simple": { + "LocalExamplePath": "/apply-setters/v0.1/apply-setters-simple", + "RemoteExamplePath": "https://github.com/GoogleContainerTools/kpt-functions-catalog/tree/apply-setters/v0.1/examples/apply-setters-simple", + "RemoteSourcePath": "https://github.com/GoogleContainerTools/kpt-functions-catalog/tree/apply-setters/v0.1/functions/go/apply-setters" + } } } }, "gatekeeper": { "v0.1": { - "gatekeeper-warning-only": { - "LocalExamplePath": "/gatekeeper/v0.1/gatekeeper-warning-only", - "RemoteExamplePath": "https://github.com/GoogleContainerTools/kpt-functions-catalog/tree/gatekeeper/v0.1/examples/gatekeeper-warning-only", - "RemoteSourcePath": "https://github.com/GoogleContainerTools/kpt-functions-catalog/tree/gatekeeper/v0.1/functions/go/gatekeeper" + "LatestPatchVersion": "v0.1.3", + "Examples": { + "gatekeeper-warning-only": { + "LocalExamplePath": "/gatekeeper/v0.1/gatekeeper-warning-only", + "RemoteExamplePath": "https://github.com/GoogleContainerTools/kpt-functions-catalog/tree/gatekeeper/v0.1/examples/gatekeeper-warning-only", + "RemoteSourcePath": "https://github.com/GoogleContainerTools/kpt-functions-catalog/tree/gatekeeper/v0.1/functions/go/gatekeeper" + } } }, "v0.2": { - "gatekeeper-warning-only": { - "LocalExamplePath": "/gatekeeper/v0.2/gatekeeper-warning-only", - "RemoteExamplePath": "https://github.com/GoogleContainerTools/kpt-functions-catalog/tree/gatekeeper/v0.2/examples/gatekeeper-warning-only", - "RemoteSourcePath": "https://github.com/GoogleContainerTools/kpt-functions-catalog/tree/gatekeeper/v0.2/functions/go/gatekeeper" + "LatestPatchVersion": "v0.2.1", + "Examples": { + "gatekeeper-warning-only": { + "LocalExamplePath": "/gatekeeper/v0.2/gatekeeper-warning-only", + "RemoteExamplePath": "https://github.com/GoogleContainerTools/kpt-functions-catalog/tree/gatekeeper/v0.2/examples/gatekeeper-warning-only", + "RemoteSourcePath": "https://github.com/GoogleContainerTools/kpt-functions-catalog/tree/gatekeeper/v0.2/functions/go/gatekeeper" + } } } } }`) sort.Strings(result) - assert.Equal(t, []string{"apply-setters:v0.1", "gatekeeper:v0.2"}, result) + assert.Equal(t, []string{"apply-setters:v0.1.1", "gatekeeper:v0.2.1"}, result) } diff --git a/main.go b/main.go index 4e1053af11..311b9ad510 100644 --- a/main.go +++ b/main.go @@ -60,10 +60,10 @@ func runMain() int { // Note(droot): There are too many flags exposed that makes the command // usage verbose but couldn't find a way to make it less verbose. klog.InitFlags(&logFlags) - cmd.Flags().AddGoFlagSet(&logFlags) // By default klog v1 logs to stderr, switch that off - _ = cmd.Flags().Set("logtostderr", "false") - _ = cmd.Flags().Set("alsologtostderr", "false") + _ = logFlags.Set("logtostderr", "false") + _ = logFlags.Set("alsologtostderr", "true") + cmd.Flags().AddGoFlagSet(&logFlags) err = cmd.Execute() if err != nil { diff --git a/pkg/api/kptfile/v1/types.go b/pkg/api/kptfile/v1/types.go index 8f668ab788..ecabd4d9be 100644 --- a/pkg/api/kptfile/v1/types.go +++ b/pkg/api/kptfile/v1/types.go @@ -315,6 +315,13 @@ type Function struct { // image: set-labels Image string `yaml:"image,omitempty" json:"image,omitempty"` + // Exec specifies the function binary executable. + // The executable can be fully qualified or it must exists in the $PATH e.g: + // + // exec: set-namespace + // exec: /usr/local/bin/my-custom-fn + Exec string `yaml:"exec,omitempty" json:"exec,omitempty"` + // `ConfigPath` specifies a slash-delimited relative path to a file in the current directory // containing a KRM resource used as the function config. This resource is // excluded when resolving 'sources', and as a result cannot be operated on diff --git a/pkg/api/kptfile/v1/validation.go b/pkg/api/kptfile/v1/validation.go index 41eb9e8488..6489afd7f3 100644 --- a/pkg/api/kptfile/v1/validation.go +++ b/pkg/api/kptfile/v1/validation.go @@ -67,14 +67,29 @@ func (p *Pipeline) validate(pkgPath types.UniquePath) error { } func (f *Function) validate(fnType string, idx int, pkgPath types.UniquePath) error { - err := ValidateFunctionImageURL(f.Image) - if err != nil { + if f.Image == "" && f.Exec == "" { + return &ValidateError{ + Field: fmt.Sprintf("pipeline.%s[%d]", fnType, idx), + Reason: "must specify a functon (`image` or `exec`) to execute", + } + } + if f.Image != "" && f.Exec != "" { return &ValidateError{ - Field: fmt.Sprintf("pipeline.%s[%d].image", fnType, idx), - Value: f.Image, - Reason: err.Error(), + Field: fmt.Sprintf("pipeline.%s[%d]", fnType, idx), + Reason: "must not specify both `image` and `exec` at the same time", + } + } + if f.Image != "" { + err := ValidateFunctionImageURL(f.Image) + if err != nil { + return &ValidateError{ + Field: fmt.Sprintf("pipeline.%s[%d].image", fnType, idx), + Value: f.Image, + Reason: err.Error(), + } } } + // TODO(droot): validate the exec if len(f.ConfigMap) != 0 && f.ConfigPath != "" { return &ValidateError{ diff --git a/pkg/live/apply-crd-task.go b/pkg/live/apply-crd-task.go index 2a1399ea74..2fe40fe258 100644 --- a/pkg/live/apply-crd-task.go +++ b/pkg/live/apply-crd-task.go @@ -41,7 +41,7 @@ func (a *ApplyCRDTask) Action() event.ResourceAction { } func (a *ApplyCRDTask) Identifiers() object.ObjMetadataSet { - return object.UnstructuredsToObjMetasOrDie([]*unstructured.Unstructured{a.crd}) + return object.UnstructuredSetToObjMetadataSet([]*unstructured.Unstructured{a.crd}) } // NewApplyCRDTask returns a pointer to an ApplyCRDTask struct, @@ -89,4 +89,6 @@ func (a *ApplyCRDTask) Start(taskContext *taskrunner.TaskContext) { }() } -func (a *ApplyCRDTask) ClearTimeout() {} +func (a *ApplyCRDTask) Cancel(_ *taskrunner.TaskContext) {} + +func (a *ApplyCRDTask) StatusUpdate(_ *taskrunner.TaskContext, _ object.ObjMetadata) {} diff --git a/pkg/live/inventoryrg.go b/pkg/live/inventoryrg.go index ae9a708845..7d10f269b0 100644 --- a/pkg/live/inventoryrg.go +++ b/pkg/live/inventoryrg.go @@ -31,8 +31,9 @@ import ( "sigs.k8s.io/cli-utils/pkg/apply/taskrunner" "sigs.k8s.io/cli-utils/pkg/common" "sigs.k8s.io/cli-utils/pkg/inventory" + "sigs.k8s.io/cli-utils/pkg/kstatus/polling" + "sigs.k8s.io/cli-utils/pkg/kstatus/polling/engine" "sigs.k8s.io/cli-utils/pkg/object" - utilfactory "sigs.k8s.io/cli-utils/pkg/util/factory" "sigs.k8s.io/kustomize/kyaml/yaml" ) @@ -149,9 +150,10 @@ func (icm *InventoryResourceGroup) Load() (object.ObjMetadataSet, error) { Kind: strings.TrimSpace(kind), } klog.V(4).Infof("creating obj metadata: %s/%s/%s", namespace, name, groupKind) - objMeta, err := object.CreateObjMetadata(namespace, name, groupKind) - if err != nil { - return []object.ObjMetadata{}, err + objMeta := object.ObjMetadata{ + GroupKind: groupKind, + Name: name, + Namespace: namespace, } objs = append(objs, objMeta) } @@ -266,7 +268,7 @@ func InstallResourceGroupCRD(factory cmdutil.Factory) error { } // Create the task to apply the ResourceGroup CRD. applyRGTask := NewApplyCRDTask(factory, crd) - objs := object.UnstructuredsToObjMetasOrDie([]*unstructured.Unstructured{crd}) + objs := object.UnstructuredSetToObjMetadataSet([]*unstructured.Unstructured{crd}) // Create the tasks to apply the ResourceGroup CRD. tasks := []taskrunner.Task{ applyRGTask, @@ -278,7 +280,7 @@ func InstallResourceGroupCRD(factory cmdutil.Factory) error { for _, t := range tasks { taskQueue <- t } - statusPoller, err := utilfactory.NewStatusPoller(factory) + statusPoller, err := polling.NewStatusPollerFromFactory(factory, []engine.StatusReader{}) if err != nil { handleError(eventChannel, err) return diff --git a/pkg/live/load_test.go b/pkg/live/load_test.go index 919eaca0d5..4562d10a08 100644 --- a/pkg/live/load_test.go +++ b/pkg/live/load_test.go @@ -34,7 +34,7 @@ func TestLoad_LocalDisk(t *testing.T) { testCases := map[string]struct { pkg *pkgbuilder.RootPkg namespace string - expectedObjs []object.ObjMetadata + expectedObjs object.ObjMetadataSet expectedInv kptfile.Inventory expectedErrMsg string }{ @@ -184,7 +184,7 @@ func TestLoad_LocalDisk(t *testing.T) { } assert.NoError(t, err) - objMetas := object.UnstructuredsToObjMetasOrDie(objs) + objMetas := object.UnstructuredSetToObjMetadataSet(objs) sort.Slice(objMetas, func(i, j int) bool { return objMetas[i].String() < objMetas[j].String() }) @@ -199,7 +199,7 @@ func TestLoad_StdIn(t *testing.T) { testCases := map[string]struct { pkg *pkgbuilder.RootPkg namespace string - expectedObjs []object.ObjMetadata + expectedObjs object.ObjMetadataSet expectedInv kptfile.Inventory expectedErrMsg string }{ @@ -343,7 +343,7 @@ func TestLoad_StdIn(t *testing.T) { } assert.NoError(t, err) - objMetas := object.UnstructuredsToObjMetasOrDie(objs) + objMetas := object.UnstructuredSetToObjMetadataSet(objs) sort.Slice(objMetas, func(i, j int) bool { return objMetas[i].String() < objMetas[j].String() }) diff --git a/pkg/live/rgpath_test.go b/pkg/live/rgpath_test.go index 2e529e00cc..4130671a9f 100644 --- a/pkg/live/rgpath_test.go +++ b/pkg/live/rgpath_test.go @@ -23,7 +23,7 @@ func TestPathManifestReader_Read(t *testing.T) { testCases := map[string]struct { manifests map[string]string namespace string - expectedObjs []object.ObjMetadata + expectedObjs object.ObjMetadataSet expectedErrMsg string }{ "Empty package is ok": { @@ -151,7 +151,7 @@ func TestPathManifestReader_Read(t *testing.T) { "cr.yaml": cr, }, namespace: "test-namespace", - expectedErrMsg: "unknown resource types: Custom.custom.io", + expectedErrMsg: "unknown resource types: custom.io/v1/Custom", }, "local-config is filtered out": { manifests: map[string]string{ @@ -211,7 +211,7 @@ func TestPathManifestReader_Read(t *testing.T) { } assert.NoError(t, err) - readObjMetas := object.UnstructuredsToObjMetasOrDie(readObjs) + readObjMetas := object.UnstructuredSetToObjMetadataSet(readObjs) sort.Slice(readObjMetas, func(i, j int) bool { return readObjMetas[i].String() < readObjMetas[j].String() diff --git a/pkg/live/rgstream_test.go b/pkg/live/rgstream_test.go index 15ddf9ad6c..18d59923fc 100644 --- a/pkg/live/rgstream_test.go +++ b/pkg/live/rgstream_test.go @@ -22,7 +22,7 @@ func TestResourceStreamManifestReader_Read(t *testing.T) { testCases := map[string]struct { manifests map[string]string namespace string - expectedObjs []object.ObjMetadata + expectedObjs object.ObjMetadataSet expectedErrMsg string }{ "Kptfile is excluded": { @@ -99,7 +99,7 @@ func TestResourceStreamManifestReader_Read(t *testing.T) { "cr.yaml": cr, }, namespace: "test-namespace", - expectedErrMsg: "unknown resource types: Custom.custom.io", + expectedErrMsg: "unknown resource types: custom.io/v1/Custom", }, } @@ -137,7 +137,7 @@ func TestResourceStreamManifestReader_Read(t *testing.T) { } assert.NoError(t, err) - readObjMetas := object.UnstructuredsToObjMetasOrDie(readObjs) + readObjMetas := object.UnstructuredSetToObjMetadataSet(readObjs) sort.Slice(readObjMetas, func(i, j int) bool { return readObjMetas[i].String() < readObjMetas[j].String() diff --git a/pkg/test/live/runner.go b/pkg/test/live/runner.go index 7ee8bd11b7..304644ab01 100644 --- a/pkg/test/live/runner.go +++ b/pkg/test/live/runner.go @@ -15,6 +15,7 @@ package live import ( + "bufio" "bytes" "errors" "os" @@ -26,6 +27,7 @@ import ( "testing" "github.com/stretchr/testify/assert" + "sigs.k8s.io/cli-utils/pkg/kstatus/status" "sigs.k8s.io/kustomize/kyaml/yaml" ) @@ -96,11 +98,15 @@ func (r *Runner) VerifyExitCode(t *testing.T, err error) { } func (r *Runner) VerifyStdout(t *testing.T, stdout string) { - assert.Equal(t, strings.TrimSpace(r.Config.StdOut), strings.TrimSpace(substituteTimestamps(stdout))) + assert.Equal(t, strings.TrimSpace(r.Config.StdOut), prepOutput(t, stdout)) } func (r *Runner) VerifyStderr(t *testing.T, stderr string) { - assert.Equal(t, strings.TrimSpace(r.Config.StdErr), strings.TrimSpace(substituteTimestamps(stderr))) + assert.Equal(t, strings.TrimSpace(r.Config.StdErr), prepOutput(t, stderr)) +} + +func prepOutput(t *testing.T, s string) string { + return strings.TrimSpace(substituteTimestamps(removeStatusEvents(t, s))) } func (r *Runner) VerifyInventory(t *testing.T, name, namespace string) { @@ -173,3 +179,32 @@ var timestampRegexp = regexp.MustCompile(`\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z`) func substituteTimestamps(text string) string { return timestampRegexp.ReplaceAllString(text, "") } + +var statuses = []status.Status{ + status.InProgressStatus, + status.CurrentStatus, + status.FailedStatus, + status.TerminatingStatus, + status.UnknownStatus, + status.NotFoundStatus, +} + +func removeStatusEvents(t *testing.T, text string) string { + scanner := bufio.NewScanner(strings.NewReader(text)) + var lines []string + +scan: + for scanner.Scan() { + line := scanner.Text() + for _, s := range statuses { + if strings.Contains(line, s.String()) { + continue scan + } + } + lines = append(lines, line) + } + if err := scanner.Err(); err != nil { + t.Fatalf("error scanning output: %v", err) + } + return strings.Join(lines, "\n") +} diff --git a/pkg/test/runner/config.go b/pkg/test/runner/config.go index 44fc75d985..57b3c3035c 100644 --- a/pkg/test/runner/config.go +++ b/pkg/test/runner/config.go @@ -71,6 +71,9 @@ type TestCaseConfig struct { // be the same as the CLI flag. ImagePullPolicy string `json:"imagePullPolicy,omitempty" yaml:"imagePullPolicy,omitempty"` + // AllowExec determines if `fn render` needs to be invoked with `--allow-exec` flag + AllowExec bool `json:"allowExec,omitempty" yaml:"allowExec,omitempty"` + // Skip means should this test case be skipped. Default: false Skip bool `json:"skip,omitempty" yaml:"skip,omitempty"` diff --git a/pkg/test/runner/runner.go b/pkg/test/runner/runner.go index 4309cab839..1754a77672 100644 --- a/pkg/test/runner/runner.go +++ b/pkg/test/runner/runner.go @@ -349,6 +349,10 @@ func (r *Runner) runFnRender() error { kptArgs = append(kptArgs, "--image-pull-policy", r.testCase.Config.ImagePullPolicy) } + if r.testCase.Config.AllowExec { + kptArgs = append(kptArgs, "--allow-exec") + } + if r.testCase.Config.DisableOutputTruncate { kptArgs = append(kptArgs, "--truncate-output=false") } diff --git a/release/images/Dockerfile b/release/images/Dockerfile index 6ebcf4a7cf..cc34596e1a 100644 --- a/release/images/Dockerfile +++ b/release/images/Dockerfile @@ -12,9 +12,9 @@ # See the License for the specific language governing permissions and # limitations under the License. -FROM alpine:3.11 +FROM alpine:3.13 RUN apk update && apk upgrade && \ - apk add --no-cache git less man diffutils bash openssh docker-cli && \ + apk add --no-cache git less mandoc diffutils bash openssh docker-cli && \ rm -rf /var/lib/apt/lists/* && \ rm /var/cache/apk/* # This is set up for the Dockerfile to be used by goreleaser: https://goreleaser.com/customization/docker/ diff --git a/release/images/Dockerfile-gcloud b/release/images/Dockerfile-gcloud index 4cbb114703..5f3c16c76d 100644 --- a/release/images/Dockerfile-gcloud +++ b/release/images/Dockerfile-gcloud @@ -12,7 +12,7 @@ # See the License for the specific language governing permissions and # limitations under the License. -FROM gcr.io/google.com/cloudsdktool/cloud-sdk:355.0.0-alpine +FROM gcr.io/google.com/cloudsdktool/cloud-sdk:367.0.0-alpine RUN apk update && apk upgrade && \ apk add --no-cache git less diffutils bash openssh docker-cli && \ rm -rf /var/lib/apt/lists/* && \ diff --git a/run/run.go b/run/run.go index 176a898220..36ab589647 100644 --- a/run/run.go +++ b/run/run.go @@ -24,7 +24,6 @@ import ( "strings" kptcommands "github.com/GoogleContainerTools/kpt/commands" - "github.com/GoogleContainerTools/kpt/internal/cmdcomplete" "github.com/GoogleContainerTools/kpt/internal/docs/generated/overview" "github.com/GoogleContainerTools/kpt/internal/printer" "github.com/GoogleContainerTools/kpt/internal/util/cmdutil" @@ -36,7 +35,6 @@ var pgr []string func GetMain(ctx context.Context) *cobra.Command { os.Setenv(commandutil.EnableAlphaCommmandsEnvName, "true") - installComp := false cmd := &cobra.Command{ Use: "kpt", Short: overview.CliShort, @@ -46,18 +44,6 @@ func GetMain(ctx context.Context) *cobra.Command { // adjust the error message coming from libraries SilenceErrors: true, RunE: func(cmd *cobra.Command, args []string) error { - if installComp { - os.Setenv("COMP_INSTALL", "1") - os.Setenv("COMP_YES", "1") - fmt.Fprint(cmd.OutOrStdout(), "Installing shell completion...\n") - fmt.Fprint(cmd.OutOrStdout(), - "This will add 'complete -C /Users/$USER/go/bin/kpt kpt' to "+ - ".bashrc, .bash_profile, etc\n") - fmt.Fprint(cmd.OutOrStdout(), "Run `COMP_INSTALL=0 kpt` to uninstall.\n") - } - // Complete exits if it is called in completion mode, otherwise it is a no-op - cmdcomplete.Complete(cmd, false, nil).Complete("kpt") - h, err := cmd.Flags().GetBool("help") if err != nil { return err @@ -77,20 +63,6 @@ func GetMain(ctx context.Context) *cobra.Command { // create context with associated printer ctx = printer.WithContext(ctx, pr) - cmd.Flags().BoolVar(&installComp, "install-completion", false, - "Install shell completion") - // this command will be invoked by the shell-completion code - cmd.AddCommand(&cobra.Command{ - Use: "kpt", - Hidden: true, - SilenceErrors: true, - SilenceUsage: true, - Run: func(cmd *cobra.Command, args []string) { - // Complete exits if it is called in completion mode, otherwise it is a no-op - cmdcomplete.Complete(cmd.Parent(), false, nil).Complete("kpt") - }, - }) - // find the pager if one exists func() { if val, found := os.LookupEnv("KPT_NO_PAGER_HELP"); !found || val != "1" { diff --git a/scripts/create-licenses.sh b/scripts/create-licenses.sh index 8bffc19d60..3ec8304577 100755 --- a/scripts/create-licenses.sh +++ b/scripts/create-licenses.sh @@ -238,7 +238,7 @@ done >> ${TMP_LICENSE_FILE} cat ${TMP_LICENSE_FILE} > ${VENDOR_LICENSE_FILE} # Create a package of Mozilla repository source code (only go code). -zip -qr $ZIP_FILENAME $mozilla_repos -i '*.go' +[ -n "$mozilla_repos" ] && zip -qr $ZIP_FILENAME $mozilla_repos -i '*.go' # Cleanup vendor directory rm -rf vendor diff --git a/site/book/04-using-functions/01-declarative-function-execution.md b/site/book/04-using-functions/01-declarative-function-execution.md index c2f01cbdde..6bf9e39e09 100644 --- a/site/book/04-using-functions/01-declarative-function-execution.md +++ b/site/book/04-using-functions/01-declarative-function-execution.md @@ -110,13 +110,48 @@ The end result is that: If any of the functions in the pipeline fails for whatever reason, then the entire pipeline is aborted and the local filesystem is left intact. -## Specifying `image` +## Specifying `function` + +### `image` The `image` field specifies the container image for the function. You can specify an image from any container registry. If the registry is omitted, the default container registry for functions catalog (`gcr.io/kpt-fn`) is prepended automatically. For example, `set-labels:v0.1` is automatically expanded to `gcr.io/kpt-fn/set-labels:v0.1`. +### `exec` + +The `exec` field specifies the executable command for the function. You can specify +an executable with arguments. + +Example below uses `sed` executable to replace all occurances of `foo` with `bar` +in the package resources. + +```yaml +# PKG_DIR/Kptfile (Excerpt) +apiVersion: kpt.dev/v1 +kind: Kptfile +metadata: + name: app +pipeline: + mutators: + - exec: "sed -e 's/foo/bar/'" +``` + +Note that you must render the package by allowing executables by specifying `--allow-exec` +command line flag as shown below. + +```shell +$ kpt fn render [PKG_DIR] --allow-exec +``` + +Using `exec` is not recommended for two reasons: + +- It makes the package non-portable since rendering the package requires the + executables to be present on the system. +- Executing binaries is not very secure since they can perform privileged operations + on the system. + ## Specifying `functionConfig` In [Chapter 2], we saw this conceptual representation of a function invocation: diff --git a/site/book/06-deploying-packages/03-handling-dependencies.md b/site/book/06-deploying-packages/03-handling-dependencies.md index 0130a3363a..609aa59d9c 100644 --- a/site/book/06-deploying-packages/03-handling-dependencies.md +++ b/site/book/06-deploying-packages/03-handling-dependencies.md @@ -79,3 +79,8 @@ statefulset.apps/wordpress-mysql deleted service/wordpress-mysql deleted 2 resource(s) deleted, 0 skipped ``` + +See [depends-on] for more information. + +[depends-on]: + /reference/annotations/depends-on/ diff --git a/site/installation/README.md b/site/installation/README.md index 4a783d265a..3d979cc52a 100644 --- a/site/installation/README.md +++ b/site/installation/README.md @@ -27,6 +27,47 @@ Verify the version: $ kpt version ``` +## (Optional) enable shell auto-completion + +kpt provides auto-completion support for several of the common shells. +To see the options for enabling shell auto-completion: +```shell +kpt completion -h +``` + +### Prerequisites +`kpt` depends on `bash-completion` in order to support auto-completion for the +bash shell. If you are using bash as your shell, you will need to install +`bash-completion` in order to use kpt's auto-completion feature. +`bash-completion` is provided by many package managers +(see [here][bash-completion]). + +### Enable kpt auto-completion +The kpt completion script for a shell can be generated with the commands +`kpt completion bash`, `kpt completion zsh`, etc. Sourcing the completion script +in your shell enables auto-completion. + +#### Enable auto-completion for your current shell +bash: +```shell +source <(kpt completion bash) +``` +zsh: +```shell +source <(kpt completion zsh) +``` +etc. +#### Enable kpt completion for all your shell sessions +bash: +```shell +echo 'source <(kpt completion bash)' >> ~/.bashrc +``` +zsh: +```shell +echo 'source <(kpt completion zsh)' >> ~/.zshrc +``` +etc. +