From aa29f77109c7b1f4051047d675b5968c0ace2f9d Mon Sep 17 00:00:00 2001 From: Louis DeJardin Date: Thu, 13 Jan 2022 15:47:42 -0800 Subject: [PATCH] Pull from main into oci-branch (#2643) * design document template (#2563) * updated maintainers * update to the front page * installation paragraph * design doc template * additional process detail * slight updates * update kyaml and cli-utils to latest (#2523) * Do not error while adding merge comment (#2564) * Remove check if repo is checked into git for update (#2569) * Remove check if repo is checked into git for update * Update tests * Update should accept empty path with version (#2572) * Update roadmap for Q4 (#2571) * Update roadmap for Q4 * Suggested changes * Update roadmap (#2574) * Update roadmap * Suggested changes * Update with suggested format * Organize better * update installation doc and Homebrew to use 1.0.0-beta.8 (#2578) * Update kpt book with depends-on annotation (#2565) * Update kpt book with depends-on annotation * Addressed comments * Add snippet for resource dependencies in chapter overview * fix: use KptFileKind variable instead of KptFileName for GVR (#2580) When specifying the Kptfile resource kind, some code were using the KptFileName variable instead of KptFileKind. Both variables currently contain the same value: "Kptfile", but this will be an issue in the future should the KptFileName and KptFileKind variables have different values. * Disable unwrapping to read fnconfig (#2582) * Disable wrapping to read fnconfig * Update index annotation * Update licenses (#2583) - Use go install with explicit tag to avoid modifying go mod - Add licenses to yaml files (new addlicense feature) - Add updated license-check to make all * Revert "update installation doc and Homebrew to use 1.0.0-beta.8 (#2578)" (#2584) This reverts commit 98f029925dc2faaf43115c531e9a318ffc26b5f2. * add tests to exercise modify path annotation (#2586) * Update design doc review process (#2594) * Update beta 9 (#2597) * Add missing json annotation to KptFile (#2598) * Implement metrics annotation (#2588) * Implement metrics annotation * Changes * Suggested changes * Suggested changes * Attributor * Design proposal: kpt live -- Inventory to ResourceGroup (#2576) * DD for kpt live apply from STDIN * resolve comments * Update doc with rg fn interaction Co-authored-by: Phani Teja Marupaka * Add cnrm annotation to merge3 (#2605) * docs: Specify origin in git push command in 3.8 (#2608) Issues: GoogleContainerTools/kpt#2596 * upgrade to kyaml 0.13.0+ (#2603) * upgrade to kyaml 0.13.0+ * use the latest kyaml * --image flag typo fix (#2601) Fix typo in `kpt fn eval`'s --image flag description * Simplify and speed up "kpt live" e2e tests (#2613) * Simplify and speed up live e2e tests * Addressed comments * Use reconcile-timeout for live apply tests (#2618) * Fix flaky e2e test (#2622) * Docs: OCI Support design docs (#2589) * Starting to fill in oci design doc * Updates to design doc * Formatting updates * Updating title * Adding why * Update 01-oci-support.md * Update 01-oci-support.md * Docs: Adding yaml examples to kpt pkg update * Docs: add descriptions of pkg pull/push Also adds kpt pkg copy as an alternative * Update 01-oci-support.md Adding comparison of get vs pull * Update 01-oci-support.md Updating the descriptions of `kpt pkg pull` and `kpt pkg push` Co-authored-by: Louis DeJardin * changed default image pull policy to if-not-present (#2600) * updated installation instructions for kpt 1.0.beta.10 (#2628) * feat: Use fn patch version for cli autocomplete (#2629) This brings the cli function autocompletion in line with other updates to kpt function catalog/documentation, in which we display the fully qualified (patch) version for functions. This is accomplished using the new /catalog-v2.json endpoint which serves an updated schema that includes the latest patch version of each function release. * fix: bash auto completion for CLI flags (#2630) This kpt subcommand is invoked by bash shell completion, which previously returned an error when attempting to perform auto completion for flag values. The actual subcommands and their associated flags are registered with the root Command but not this subcommand, which results in flag validation returning an error for this subcommand. Disabling flag validation fixes this issue and enables auto completion for flags in bash. * Cobra based autocompletion (#2631) * refactor: set klog flags before adding to cmd When flags are set using the Command.Flags().Set() method, the flags are marked as Changed. This results in the Cobra autocompletion logic treating it as though the user passed flags for the Command, ultimately resulting in Cobra not rendering the available subcommands for the root kpt command. This change enables the expected behavior when invoking `kpt ` with the Cobra built-in autocompletion. * refactor: use Cobra built-in autocompletion Prior to this change, the kpt CLI exposed two methods for enabling shell autocompletion: - kpt --install-completion - kpt completion [bash|fish|zsh|powershell] The former was implemented using complete, whereas the latter is provided by Cobra as a default. There is not a clear motivation to offer two methods to provide autocompletion functionality, so this change aims to consolidate these into a single method. This change removes the --install-completion option and applies some fixes to enable proper operation of Cobras autocompletion. The decision was made to favor the Cobra implementation over the custom one in order to reduce maintenance cost and offer similar behavior to other popular CLIs which leverage Cobra (e.g. kubectl) * ci: allow no mozilla repos in create-licenses As of a prior commit there are no dependencies on mozilla repos. This change prevents the script from failing in the case of mozilla_repos being an empty string. * fn render supports function binary executables (#2593) * update base image in Dockerfiles (#2634) * Add annotation reference docs (#2636) * docs: add autocompletion in installation guide (#2635) * Fix go.sum with `go mod tidy` * Fix to optional kpt pkg push argument * Update cli-utils to latest version (#2616) * Remove printers from third_party and use cli-utils (#2627) * docs: fix book hyperlink for depends-on (#2640) The hyperlink was missing a trailing / which was leading to a 404 for the redirected page. * feat: always print truncated function stderr (#2639) * feat: always print truncated function stderr This change makes it so that kpt fn render|eval will always print truncated stderr output from functions. This is intended to improve the experience of debugging functions. * test: add unit test cases for printFnStderr * test: add e2e test for fn eval stderr This test case is intended to check that kpt fn eval outputs the stderr of the function even when the function succeeds. This is doing using a simple function script which emits hello world to stderr. * test: add e2e test for fn render stderr This test case is intended to check that kpt fn render outputs the stderr of the function even when the function succeeds. This is doing using a simple function script which emits hello world to stderr. * Update cli-utils to v0.27.0 (#2644) * Added docs for using exec in fn render (#2642) * Added docs for using exec in fn render * addressed review comment * Removing extra files from merge Co-authored-by: Mike Borozdin Co-authored-by: Natasha Sarkar Co-authored-by: phani Co-authored-by: Mengqi Yu Co-authored-by: Morten Torkildsen Co-authored-by: Ramon Quitales Co-authored-by: Karl Isenberg Co-authored-by: Martin Maly Co-authored-by: Yuwen Ma Co-authored-by: Phani Teja Marupaka Co-authored-by: sdowell Co-authored-by: Ben Congdon Co-authored-by: Louis DeJardin Co-authored-by: Sunil Arora --- docs/design-docs/01-oci-support.md | 474 ++++++++++++++++++ e2e/live/end-to-end-test.sh | 62 +-- .../error-in-pipe/.expected/config.yaml | 7 +- .../.expected/config.yaml | 24 + .../exec-function-stderr/.expected/diff.patch | 21 + .../exec-function-stderr/.expected/exec.sh | 19 + .../fn-eval/exec-function-stderr/.krmignore | 1 + .../fn-eval/exec-function-stderr/function.sh | 21 + .../exec-function-stderr/resources.yaml} | 31 +- .../pkg/.expected/results.yaml | 3 +- .../missing-fn-image/.expected/config.yaml | 2 +- .../.expected/config.yaml | 24 + .../exec-function-stderr/.expected/diff.patch | 21 + .../fn-render/exec-function-stderr/.krmignore | 1 + .../fn-render/exec-function-stderr/Kptfile | 7 + .../exec-function-stderr/function.sh | 21 + .../exec-function-stderr/resources.yaml | 28 ++ .../.expected/config.yaml | 15 + .../.expected/diff.patch | 21 + .../exec-function-with-args/.krmignore | 1 + .../fn-render/exec-function-with-args/Kptfile | 7 + .../exec-function-with-args/resources.yaml | 28 ++ .../.expected/config.yaml | 16 + .../exec-without-permissions/.krmignore | 1 + .../exec-without-permissions/Kptfile | 7 + .../exec-without-permissions/resources.yaml | 28 ++ .../.expected/results.yaml | 3 +- .../missing-fn-image/.expected/config.yaml | 2 +- .../live-apply/apply-depends-on/config.yaml | 11 +- .../live-apply/crd-and-cr/config.yaml | 12 +- .../dry-run-with-install-rg/config.yaml | 4 +- .../install-rg-on-apply/config.yaml | 2 + .../live-apply/json-output/config.yaml | 24 +- .../resources/{third.yaml => dep.yaml} | 8 +- .../live-apply/prune-depends-on/config.yaml | 15 +- go.mod | 8 +- go.sum | 13 +- internal/cmdapply/cmdapply.go | 35 +- internal/cmdcomplete/complete.go | 81 --- internal/cmddestroy/cmddestroy.go | 29 +- internal/cmdfndoc/cmdfndoc.go | 3 + internal/cmdget/cmdget.go | 3 + internal/cmdmigrate/migratecmd.go | 16 +- internal/cmdmigrate/migratecmd_test.go | 6 +- internal/cmdpush/cmdpush.go | 2 +- internal/cmdrender/cmdrender.go | 13 +- internal/cmdrender/executor.go | 44 +- internal/cmdupdate/cmdupdate.go | 3 + internal/docs/generated/fndocs/docs.go | 5 + internal/errors/resolver/live.go | 35 +- internal/errors/resolver/live_test.go | 39 +- internal/fnruntime/container.go | 132 +++-- internal/fnruntime/container_test.go | 4 + internal/fnruntime/fnerrors_test.go | 74 +++ internal/fnruntime/runner.go | 69 ++- internal/fnruntime/runner_test.go | 69 +++ internal/util/cmdutil/cmdutil.go | 16 +- internal/util/cmdutil/cmdutil_test.go | 35 +- main.go | 6 +- pkg/api/kptfile/v1/types.go | 7 + pkg/api/kptfile/v1/validation.go | 25 +- pkg/live/apply-crd-task.go | 6 +- pkg/live/inventoryrg.go | 14 +- pkg/live/load_test.go | 8 +- pkg/live/rgpath_test.go | 6 +- pkg/live/rgstream_test.go | 6 +- pkg/test/live/runner.go | 39 +- pkg/test/runner/config.go | 3 + pkg/test/runner/runner.go | 4 + release/images/Dockerfile | 4 +- release/images/Dockerfile-gcloud | 2 +- run/run.go | 28 -- scripts/create-licenses.sh | 2 +- .../01-declarative-function-execution.md | 37 +- .../03-handling-dependencies.md | 5 + site/installation/README.md | 50 +- site/reference/README.md | 1 + site/reference/annotations/README.md | 21 + .../annotations/apply-time-mutation/README.md | 225 +++++++++ .../annotations/depends-on/README.md | 115 +++++ .../annotations/local-config/README.md | 88 ++++ site/reference/cli/fn/render/README.md | 5 + site/sidebar.md | 4 + thirdparty/cli-utils/flagutils/utils.go | 54 -- thirdparty/cli-utils/flagutils/utils_test.go | 48 -- thirdparty/cli-utils/print/common/ansii.go | 11 - thirdparty/cli-utils/print/common/color.go | 46 -- .../cli-utils/print/common/color_test.go | 70 --- thirdparty/cli-utils/print/list/base.go | 216 -------- thirdparty/cli-utils/print/table/base.go | 153 ------ thirdparty/cli-utils/print/table/base_test.go | 188 ------- .../cli-utils/print/table/columndefs.go | 243 --------- .../cli-utils/print/table/columndefs_test.go | 239 --------- .../cli-utils/printers/events/formatter.go | 147 ------ .../printers/events/formatter_test.go | 240 --------- .../cli-utils/printers/events/printer.go | 19 - thirdparty/cli-utils/printers/json/doc.go | 75 --- .../cli-utils/printers/json/formatter.go | 148 ------ .../cli-utils/printers/json/formatter_test.go | 309 ------------ thirdparty/cli-utils/printers/json/printer.go | 19 - .../cli-utils/printers/printer/printer.go | 13 - thirdparty/cli-utils/printers/printers.go | 48 -- .../cli-utils/printers/table/collector.go | 288 ----------- .../printers/table/collector_test.go | 192 ------- .../cli-utils/printers/table/printer.go | 149 ------ .../cli-utils/printers/table/printer_test.go | 74 --- thirdparty/cli-utils/status/cmdstatus.go | 6 +- .../cli-utils/status/printers/list/base.go | 2 +- .../cli-utils/status/printers/printers.go | 6 +- .../status/printers/table/adapter.go | 2 +- .../status/printers/table/table_printer.go | 2 +- .../cmdconfig/commands/cmdeval/cmdeval.go | 5 +- .../commands/cmdeval/cmdeval_test.go | 6 +- 113 files changed, 1982 insertions(+), 3453 deletions(-) create mode 100644 docs/design-docs/01-oci-support.md create mode 100644 e2e/testdata/fn-eval/exec-function-stderr/.expected/config.yaml create mode 100644 e2e/testdata/fn-eval/exec-function-stderr/.expected/diff.patch create mode 100644 e2e/testdata/fn-eval/exec-function-stderr/.expected/exec.sh create mode 100644 e2e/testdata/fn-eval/exec-function-stderr/.krmignore create mode 100755 e2e/testdata/fn-eval/exec-function-stderr/function.sh rename e2e/testdata/{live-apply/json-output/resources/fourth.yaml => fn-eval/exec-function-stderr/resources.yaml} (57%) create mode 100644 e2e/testdata/fn-render/exec-function-stderr/.expected/config.yaml create mode 100644 e2e/testdata/fn-render/exec-function-stderr/.expected/diff.patch create mode 100644 e2e/testdata/fn-render/exec-function-stderr/.krmignore create mode 100644 e2e/testdata/fn-render/exec-function-stderr/Kptfile create mode 100755 e2e/testdata/fn-render/exec-function-stderr/function.sh create mode 100644 e2e/testdata/fn-render/exec-function-stderr/resources.yaml create mode 100644 e2e/testdata/fn-render/exec-function-with-args/.expected/config.yaml create mode 100644 e2e/testdata/fn-render/exec-function-with-args/.expected/diff.patch create mode 100644 e2e/testdata/fn-render/exec-function-with-args/.krmignore create mode 100644 e2e/testdata/fn-render/exec-function-with-args/Kptfile create mode 100644 e2e/testdata/fn-render/exec-function-with-args/resources.yaml create mode 100644 e2e/testdata/fn-render/exec-without-permissions/.expected/config.yaml create mode 100644 e2e/testdata/fn-render/exec-without-permissions/.krmignore create mode 100644 e2e/testdata/fn-render/exec-without-permissions/Kptfile create mode 100644 e2e/testdata/fn-render/exec-without-permissions/resources.yaml rename e2e/testdata/live-apply/json-output/resources/{third.yaml => dep.yaml} (87%) delete mode 100644 internal/cmdcomplete/complete.go create mode 100644 site/reference/annotations/README.md create mode 100644 site/reference/annotations/apply-time-mutation/README.md create mode 100644 site/reference/annotations/depends-on/README.md create mode 100644 site/reference/annotations/local-config/README.md delete mode 100644 thirdparty/cli-utils/flagutils/utils.go delete mode 100644 thirdparty/cli-utils/flagutils/utils_test.go delete mode 100644 thirdparty/cli-utils/print/common/ansii.go delete mode 100644 thirdparty/cli-utils/print/common/color.go delete mode 100644 thirdparty/cli-utils/print/common/color_test.go delete mode 100644 thirdparty/cli-utils/print/list/base.go delete mode 100644 thirdparty/cli-utils/print/table/base.go delete mode 100644 thirdparty/cli-utils/print/table/base_test.go delete mode 100644 thirdparty/cli-utils/print/table/columndefs.go delete mode 100644 thirdparty/cli-utils/print/table/columndefs_test.go delete mode 100644 thirdparty/cli-utils/printers/events/formatter.go delete mode 100644 thirdparty/cli-utils/printers/events/formatter_test.go delete mode 100644 thirdparty/cli-utils/printers/events/printer.go delete mode 100644 thirdparty/cli-utils/printers/json/doc.go delete mode 100644 thirdparty/cli-utils/printers/json/formatter.go delete mode 100644 thirdparty/cli-utils/printers/json/formatter_test.go delete mode 100644 thirdparty/cli-utils/printers/json/printer.go delete mode 100644 thirdparty/cli-utils/printers/printer/printer.go delete mode 100644 thirdparty/cli-utils/printers/printers.go delete mode 100644 thirdparty/cli-utils/printers/table/collector.go delete mode 100644 thirdparty/cli-utils/printers/table/collector_test.go delete mode 100644 thirdparty/cli-utils/printers/table/printer.go delete mode 100644 thirdparty/cli-utils/printers/table/printer_test.go diff --git a/docs/design-docs/01-oci-support.md b/docs/design-docs/01-oci-support.md new file mode 100644 index 0000000000..11d925d04a --- /dev/null +++ b/docs/design-docs/01-oci-support.md @@ -0,0 +1,474 @@ +# OCI Support + +* Author(s): Louis Dejardin, @loudej +* Approver: \ + +> Every feature will need design sign off an PR approval from a core +> maintainer. If you have not got in touch with anyone yet, you can leave +> this blank and we will try to line someone up for you. + +## Why + +Systems that deal with packaging software or bundling configuration often have +an atomic, versionable artifact. This artifact can exist as a source of +truth independant from the source controlled content from which it was built. + +It is also very common for those packages to have an associated feed or repository +which can receive those packages as they are published, and make them available for +download as needed. In many companies, using `git` as the as the repository and source +of truth for production configuration comes with challenges. + +This design document proposes to add `OCI` as an alternative to `git` for publishing and +distributing Kpt config packages. As a packaging format, it is well understood and documented. +As a repository format, it leverages existing container registries for pushing and pulling +config as image content. For security and production configuration management, if a +company has practices for managing Docker container images in private registries, then +the same practices and security model can be applied to config package images in private +registries. + +https://github.com/GoogleContainerTools/kpt/issues/2300 + +## Design + +### Design Assumptions + +The first stage of OCI support comes from adding support for `oci` in the places +where `git` appears today. + +An image tag is used in the same way a git branch or tag would be used. + +An image digest is used in the same way a git commit would be used. + +The scope of a single image is one root package, with any number of optional sub-packages. + +The structure of the image is a single tar layer. The root `Kptfile` is in the base directory from the tar layer's point of view. It contains only the Kpt package files, no entrypoint or executables. + +A package image should not be confused with a container image. Container images are executable by software, like `Docker`, and package images are purely configuration data. + +### Config chages + +The `Kptfile` structures for `upstream` and `upstreamLock` have `oci` in addition to `git` properties. The `type` property also has the string `oci` added as an accepted value. + +```yaml +upstream: + type: oci + oci: + image: 'IMAGE:TAG' +upstreamLock: + type: oci + oci: + image: 'IMAGE:DIGEST' +``` + +New verions of `kpt` will support existing `Kptfile`. The file structures and `git` functionality is unchanged. + +Existing versions of `kpt` will support `Kptfile` with `upstream` based on `git` for the same reason. The structure and meaning of existing fields is not changed. + +Existing versions of `kpt` will not support `Kptfile` with `upstream` based on `oci`. The `type` value, and missing `git` information will fail validation. The `kpt` binary used will need to be upgraded. + +### Command changes + +### `kpt pkg get` + +The argument that determines upstream today is parsed into `repo`, `ref`, and `path`, and is implicitly a `git` location. + +To support `oci`, it will be necessary to extract different values in a way that's unambiguous. Unfortunately, OCI image names have no Uri prefix, and are indistinguishable from a valid path or file name. + +To solve this, using [Helm](https://helm.sh/docs/topics/registries/#other-subcommands) as an example, the prefix `oci://` can be used. This ensures that selecting `oci` protocol isn't accidental, and it won't collide with other location formats that may be added. + +```shell +# clone package as new folder +kpt pkg get oci://us-docker.pkg.dev/the-project-id/the-repo-name/the-package:v3 my-package +``` + +Because OCI image reference already has a convention for `image:tag` references, using `:v3` should be used instead of `@v3` for version. It will be more intuitive how it relates to the registry, and easier to cut and paste values. + +### `kpt pkg get` sub-packages + +It is possible to use `kpt pkg get` to add sub-packages to a target location. + +Syntax for a sub-package target location is unchanged, it's a normal filesystem path. + +Syntax for an OCI sub-package source location requires the ability to tell when an image name ends and a sub-package path inside that image begins. In `git` this requires an explicit `.git` extension at the transition, and in `.oci` this requires double slash. + +```shell +# clone sub-package as new sub-folder +kpt pkg get oci://us-docker.pkg.dev/the-project-id/the-repo-name/the-package//simple/example:v3 my-package/simple/my-example +``` + +### `kpt pkg update` + +The command for update is not changed, but when the `upstream` is `oci` then the `@VERSION` is used to change the `upstream` image's `tag` or `digest` value. + +To update to an image tag, `kpt pkg update @v14` and `kpt pkg update DIR@v14` will assign the `:v14` tag onto the upstream image. + +```yaml +upstream: + type: oci + oci: + image: us-docker.pkg.dev/the-project-id/the-repo-name/the-package:v14 +``` + +To update to an upstream digest, `kpt pkg update @sha256:{SHA256_HEX}` and `kpt pkg update DIR@sha256:{SHA256_HEX}` will assign `@sha256:{SHA256_HEX}` as the new upstream image digest. + +```yaml +upstream: + type: oci + oci: + image: us-docker.pkg.dev/the-project-id/the-repo-name/the-package@sha256:8815143a333cb9d2cb341f10b984b22f3b8a99fe +``` + +Calling `kpt pkg update` and `kpt pkg update DIR` will perform an update without changing the upstream image name. + +At that point, if the `upstream` is an `image:tag` that is to discover the current `image:digest` for tag, otherwise the `upstream` value for `image:digest` is used. In either case, the `upstreamLock` is changed to point at that new `image:digest`. + +The package contents of the old and new `upstreamLock` image digest are fetched to temp folders, and are the basis of the 3-way merge to update the target package. + +### `kpt pkg diff` + +The `kpt pkg diff` command is identical to `kpt pkg update` in the way that `[PKG_PATH@VERSION]` argument is mapped to OCI concepts. + +### Command additions + +Although it is possible to create and push an OCI image using a combination of commands like `tar` and `gcrane`, that +doesn't provide a very complete end to end experience. Because kpt would already be built with the same OCI go module used +by `gcrane`, it is not difficult to support additional commands to move pull and push package contents from local folders +to remote images and back. + +### ` kpt pkg pull` + +``` +Usage: kpt pkg pull oci://{IMAGE[:TAG|@sha256:DIGEST]} [DIR] + DIR Destination folder for image contents. Default folder name is the last part of the IMAGE path. + IMAGE[:TAG|@sha256:DIGEST] Name of image to pull contents from, with optional TAG or DIGEST. Default TAG is `Latest` +``` + +This command is the reverse of push. An image can be pulled from a repository to a local folder, modified, and pushed +back to the same location, same location with different TAG, or entirely different location. + +The target DIR is optional, following the conventions of `kpg pkg get`, and will default to the final image/path segment. + +`kpt pkg pull` works on git uri as well. This may be used, for example, to mirror a set of known blueprints into a private +oci registry. + +### ` kpt pkg push` + +``` +Usage: kpt pkg push [DIR@VERSION] [--origin oci://{IMAGE[:TAG]}] [--increment] + DIR@VERSION Folder containing package root Kptfile. Default is current directory. + Optional @VERSION changes tag or branch to push onto. Default is most recently pulled or pushed tag. + --origin Name of image to push contents onto, with optional TAG to assign to resulting commit. + Default is to use most recently pulled/pushed image. Required if Kptfile does not have an origin. + --increment Increase the version by 1 while pushing. Default is to leave the origin's TAG or DIR@VERSION unchanged. + The Kptfile's image TAG is also updated to the new value. +``` + +This command will `tar` the contents of the package into a single image layer, and push it into the OCI repository. For +Google Artifact Registry and Google Container Registry, the current `gcloud auth` SSO credentials are used. + +The simplest form of the command is `kpt pkg push` or `kpt pkg push DIR` which will push the current contents back to +the IMAGE:TAG location that was saved when `kpt pkg pull` was run. + +The synxax `kpt pkg push @VERSION` or `kpt pkg push DIR@VERSION` will push back to the image location it came from, but with a new TAG name or version. Examples are `kpt pkt push @draft` or `kpt pkg push @v4` + +If the Kptfile was not obtained by `kpt pkg pull` - for example it's a new package from `kpt pkg init` or `kpt pkg get` - then +the first `kpt pkg push` will require an `--origin IMAGE:TAG` option to provide the target location. It is only necessary on the first +call. + +Finally, if the IMAGE's TAG value is a valid version number, the `--increment` switch can be used to add 1 to the current value before pushing. + +In the simplest case a `v1` is changed to `v2`, and `1` is changed to `2`, but any TAG that is a valid semver (with optional leading 'v') will have the smallest part of the number incremented. So `v1.0` becomes `v1.1`, `v1.0.0` becomes `v1.0.1`, and `v4.1.9-alpha` becomes `v4.1.10-alpha` + +#### Comparison of `pkg get` and `pkg pull` + +Starting with a simple root package, and an orange variant with root as the upstream: + +``` +-- root + \-- orange {upstream: root} +``` + +The purpose of `kpt get` is to create a new leaf node. This is done by creating the initial copy of the new leaf package in a +local folder. This has the side-effects of altering the kptfile name, the upstream values to point at the source, and makes appropriate +changes to sub-package metadata. + +As an example, after running `kpt pkg get scheme://repo/root green` and `kpt pkg get scheme://repo/orange blue` the `green` and +`blue` local folder packages are appended to the inheritance tree like this: + +``` +-- root + \-- orange {upstream: root} + | `-- blue {upstream: orange} ** working copy in ./blue ** + \-- green {upstream: root} ** working copy in ./green ** +``` + +By comparison, `kpt pkg pull` does not create a new package node or identity - it only extracts a copy of existing package +contents to a working directory. In this example, if the user additionally ran `kpt pkg pull scheme://repo/root root` and +`kpt pkg pull scheme://repo/orange orange` the overall state would be this: + +``` +-- root ** working copy in ./root ** + \-- orange {upstream: root} ** working copy in ./orange ** + | `-- blue {upstream: orange} ** working copy in ./blue ** + \-- green {upstream: root} ** working copy in ./green ** +``` + +### Alternatives to push/pull + +There are several ways that pull and push could appear as commands. Those two names are very conventional, but +alternatives to consider could be: + +### `kpt pkg copy` + +``` +Usage: kpt pkg copy {SOURCE} {DEST} + SOURCE Package source location: a local DIR, or `oci://` image, or git repo and path + DEST Package destination: a local DIR, or `oci://` image. +``` + +Puts a copy of the SOURCE package at the DEST location. The package contents would be entirely unchanged by this operation (unlike `kpt pkg get`). + +To pull from remote image to local folder: + +``` +kpt pkg copy \ + oci://us-docker.pkg.dev/the-project-id/the-repo-name/the-package:v14 \ + the-package +``` + +To push from local folder to new remote image tag: + +``` +kpt pkg copy \ + the-package \ + oci://us-docker.pkg.dev/the-project-id/the-repo-name/the-package:v15 +``` + +To copy a package image from one OCI repo to another: + +``` +kpt pkg copy \ + oci://us-docker.pkg.dev/the-project-id/dev-blueprints/the-package:v25 \ + oci://us-docker.pkg.dev/the-project-id/prod-blueprints/the-package:v25 +``` + +To copy a package from a git location to an OCI repo: + +``` +kpt pkg copy \ + https://github.com/GoogleCloudPlatform/blueprints.git/catalog/gke@main \ + oci://us-docker.pkg.dev/the-project-id/gcp-catalog/gke:latest +``` + +## User Guide + +### Creating a package respository + +Before kpt packages can be pushed and pulled as OCI images, a suitable repository +must be created. Google Artifact Registry and Google Container Registry are both +excellent choices. + +```shell +# Choose names and locations +LOCATION="us" +PROJECT_ID="kpt-demo-73823" +REPOSITORY_NAME="blueprints" + +# Base name for any images in this repository +REPOSITORY="${LOCATION}-docker.pkg.dev/${PROJECT_ID}/${REPOSITORY_NAME}" + +# Create the repository +gcloud artifacts repositories create --location="${LOCATION}" --repository-format=docker --project="${PROJECT_ID}" "${REPOSITORY_NAME}" +``` + +### Creating and pushing a new package + +Creating a new package is no different. But when ready to publish, the `kpt pkg push` command +is used instead of source control operations. + +```shell +# A package in a new directory +mkdir hello-world +kpt pkg init hello-world --description="A simple blueprint" + +# Store the contents in the repository, tagged as v1 +kpt pkg push hello-world --image=${REPOSITORY}/hello-world:v1 + +# The local files are not needed any more, pushing has stored them all +rm -r hello-world +``` + +### Pulling and updating a package + +Because the package folder was discarded earlier, the `kpt pkg pull` command +is used to place the contents of a particular version at a location. These set of +commands may be run in `Cloud Build` steps as well, if you are automating the +publication of packages as part of a CI/CD process. + +```shell +# Recreate the folder and extract the pulled image +kpt pkg pull hello-world --image=${REPOSITORY}/hello-world:v1 + +# Add a sub-package from a git repo +kpt pkg get https://github.com/GoogleCloudPlatform/blueprints.git/catalog/bucket hello-world/my-bucket + +# Render to be sure contents are hydrated, and push to a new version tag +kpt pkg render hello-world +kpt pkg push hello-world --image=${REPOSITORY}/hello-world:v2 +``` + +Similar to container image tags, the package image tags like `:v1` and `:v2` above may be used any +number of ways based on your preferred workflows. The tag `:latest` is used by default if all +pulls and pushes should read from and overwrite the same location. Semantic tags like `:draft` and +environmental tags like `:dev`, `:qa`, and `:prod` may be also used. + +No matter what tags are used, the image repository and `kpt` cli will treat them as an alphanumeric +label. + +### Using OCI repository as an upstream + +In addition to providing storage for packages, an OCI registry may also +be used as a source of upstream images to clone. The `oci://` prefix on this +command is required to ensure + +```shell +# Clone the hello-world v1 blueprint into a new folder +kpt pkg get oci://${REPOSITORY}/hello-world:v1 greetings-planet + +# Push the results to the repository, using default `latest` tag in this example +kpt pkg push greetings-planet --image=${REPOSITORY}/greetings-planet +``` + +Looking in the `greetings-planet/Kptfile` at this point will show that +the `hello-world:v1` image is the `upstream`, and the `upstreamLock` will show +exactly the digest that this clone is up-to-date with. + +```yaml +apiVersion: kpt.dev/v1 +kind: Kptfile +metadata: + name: greetings-planet +upstream: + type: oci + oci: + image: us-docker.pkg.dev/kpt-demo-73823/blueprints/hello-world:v1 + updateStrategy: resource-merge +upstreamLock: + type: oci + oci: + image: us-docker.pkg.dev/kpt-demo-73823/blueprints/hello-world@sha256:1632e00af3fe858c5e3b3f9e75c16e6327449155 +``` + +### Adding a subpackage from an OCI upstream subfolder + +Often a folder inside a package is meant to be used as a way to create "more of the same". + +To use an OCI image subfolder as the source of a subpackage, the path is added in a +way that's distinct from the image itself. + +```shell +# Clone the hello-world v1 blueprint into a new folder +kpt pkg get oci://${REPOSITORY}/hello-world//my-bucket:v1 greetings-planet/another-bucket +``` + +The `greetings-planet` package will now contain both a `greetings-planet/my-bucket` as well as a +`greetings-planet/another-bucket` folder. The contents in locations will now both receive changes +when the upstream `hello-world/my-bucket` is updated. + +### Updating package with upstream changes + +The value of the upstream image tag is used to `kpt pkg update` to a specific version. +This works no matter if the tag appears to look like a version number or not. + +```shell +# Update the greetings-planet by applying any differences between the upstreamLock digest and the `v2` tag +kpt pkg update greetings-planet@v2 + +# Overwrite the `greetings-planet:latest` image with the folder contents +kpt pkg push greetings-planet --image=${REPOSITORY}/greetings-planet +``` + +The Kptfile will now show that the `upstream` and `upstreamLock` have both been changed. + +```yaml +apiVersion: kpt.dev/v1 +kind: Kptfile +metadata: + name: greetings-planet +upstream: + type: oci + oci: + image: us-docker.pkg.dev/kpt-demo-73823/blueprints/hello-world:v2 + updateStrategy: resource-merge +upstreamLock: + type: oci + oci: + image: us-docker.pkg.dev/kpt-demo-73823/blueprints/hello-world@sha256:a6f1ed69c6ab51e2a148f6d4926bccb24c843887 +``` + +Just like with a git upstream, it is also possible to `kpt pkg update` without providing a different +tag or version value. This is similar to pulling from a remote branch where the name of the branch does +change but the latest commit on that branch is a different hash. + +In that case the Kptfile `upstream` tag will not change, but if that tag has been overwritten with +new contents then the differences between `upstreamLock` and current contents will be applied to the local copy, +and the `upstreamLock` will be changed tot he current digest. + +### Updating package to a specific upstream push + +Much like you can `kpt pkg update` to a specific commit hash in git, you can update to an +exact image digest with OCI. This is an even more precise reference than by a version number because, +like git commits, the image digest is based on the package file contents and cannot be forged or altered. + +```shell +# Update instread to an exact image digest of the upstream location +kpt pkg update greetings-planet@sha256:3b42daa41102fa83bce07bd82a72edcd691868d6 +``` + +The resulting Kptfile in the local folder will look like this. + +```yaml +apiVersion: kpt.dev/v1 +kind: Kptfile +metadata: + name: greetings-planet +upstream: + type: oci + oci: + image: us-docker.pkg.dev/kpt-demo-73823/blueprints/hello-world@sha256:3b42daa41102fa83bce07bd82a72edcd691868d6 + updateStrategy: resource-merge +upstreamLock: + type: oci + oci: + image: us-docker.pkg.dev/kpt-demo-73823/blueprints/hello-world@sha256:3b42daa41102fa83bce07bd82a72edcd691868d6 +``` + +## Open Issues/Questions + +> Please list any open questions here in the following format: +> +> ### \ +> +> Resolution: Please list the resolution if resolved during the design process or +> specify __Not Yet Resolved__ + +### What additional container registries should be supported? + +The protocol and information is the same. It would mainly be a question +of how the credentials for the call are provided. + +### What commands on exising kpt binary will work on Kptfile with `oci` + +It may be possible `kpt` commands that to not process `upstream` structures +may not require update to work correctly. `kpt fn` and `kpt live` commands +should be tested to see how they behave. + +## Alternatives Considered + +If there is an industry precedent or alternative approaches please list them +here as well as citing *why* you decided not to pursue those paths. + +### \ + +Links and description of the approach, the pros and cons identified during the +design. diff --git a/e2e/live/end-to-end-test.sh b/e2e/live/end-to-end-test.sh index c5ed11102c..06f1c93786 100755 --- a/e2e/live/end-to-end-test.sh +++ b/e2e/live/end-to-end-test.sh @@ -437,11 +437,10 @@ ${BIN_DIR}/kpt live apply e2e/live/testdata/rg-test-case-1a > $OUTPUT_DIR/status # The ResourceGroup inventory CRD is automatically installed on the initial apply. assertContains "installing inventory ResourceGroup CRD" assertContains "namespace/rg-test-namespace unchanged" -assertContains "1 resource(s) applied. 0 created, 1 unchanged, 0 configured, 0 failed" assertContains "pod/pod-a created" assertContains "pod/pod-b created" assertContains "pod/pod-c created" -assertContains "3 resource(s) applied. 3 created, 0 unchanged, 0 configured, 0 failed" +assertContains "4 resource(s) applied. 3 created, 1 unchanged, 0 configured, 0 failed" wait 2 # Validate resources in the cluster # ConfigMap inventory with four inventory items. @@ -451,11 +450,10 @@ assertRGInventory "rg-test-namespace" "4" ${BIN_DIR}/kpt live apply e2e/live/testdata/rg-test-case-1a > $OUTPUT_DIR/status 2>&1 assertNotContains "installing inventory ResourceGroup CRD" # Not applied again assertContains "namespace/rg-test-namespace unchanged" -assertContains "1 resource(s) applied. 0 created, 1 unchanged, 0 configured, 0 failed" assertContains "pod/pod-a unchanged" assertContains "pod/pod-b unchanged" assertContains "pod/pod-c unchanged" -assertContains "3 resource(s) applied. 0 created, 3 unchanged, 0 configured, 0 failed" +assertContains "4 resource(s) applied. 0 created, 4 unchanged, 0 configured, 0 failed" wait 2 printResult @@ -529,12 +527,11 @@ printResult echo "[ResourceGroup] Testing initial apply dry-run" echo "kpt live apply --dry-run e2e/live/testdata/rg-test-case-1a" ${BIN_DIR}/kpt live apply --dry-run e2e/live/testdata/rg-test-case-1a > $OUTPUT_DIR/status -assertContains "namespace/rg-test-namespace created (dry-run)" -assertContains "1 resource(s) applied. 1 created, 0 unchanged, 0 configured, 0 failed (dry-run)" -assertContains "pod/pod-a created (dry-run)" -assertContains "pod/pod-b created (dry-run)" -assertContains "pod/pod-c created (dry-run)" -assertContains "3 resource(s) applied. 3 created, 0 unchanged, 0 configured, 0 failed" +assertContains "namespace/rg-test-namespace created" +assertContains "pod/pod-a created" +assertContains "pod/pod-b created" +assertContains "pod/pod-c created" +assertContains "4 resource(s) applied. 4 created, 0 unchanged, 0 configured, 0 failed" printResult # Test: Basic kpt live apply @@ -545,11 +542,10 @@ ${BIN_DIR}/kpt live apply e2e/live/testdata/rg-test-case-1a > $OUTPUT_DIR/status # The ResourceGroup CRD is already installed. assertNotContains "installing inventory ResourceGroup CRD" assertContains "namespace/rg-test-namespace unchanged" -assertContains "1 resource(s) applied. 0 created, 1 unchanged, 0 configured, 0 failed" assertContains "pod/pod-a created" assertContains "pod/pod-b created" assertContains "pod/pod-c created" -assertContains "3 resource(s) applied. 3 created, 0 unchanged, 0 configured, 0 failed" +assertContains "4 resource(s) applied. 3 created, 1 unchanged, 0 configured, 0 failed" wait 2 # Validate resources in the cluster # ConfigMap inventory with four inventory items. @@ -565,11 +561,10 @@ ${BIN_DIR}/kpt live apply link-to-rg-test-case-1a > $OUTPUT_DIR/status # The ResourceGroup CRD is already installed. assertNotContains "installing inventory ResourceGroup CRD" assertContains "namespace/rg-test-namespace unchanged" -assertContains "1 resource(s) applied. 0 created, 1 unchanged, 0 configured, 0 failed" assertContains "pod/pod-a unchanged" assertContains "pod/pod-b unchanged" assertContains "pod/pod-c unchanged" -assertContains "3 resource(s) applied. 0 created, 3 unchanged, 0 configured, 0 failed" +assertContains "4 resource(s) applied. 0 created, 4 unchanged, 0 configured, 0 failed" wait 2 # Validate resources in the cluster # ConfigMap inventory with four inventory items. @@ -602,14 +597,13 @@ echo "[ResourceGroup] Testing basic apply dry-run" echo "kpt live apply --dry-run e2e/live/testdata/rg-test-case-1b" cp -f e2e/live/testdata/rg-test-case-1a/Kptfile e2e/live/testdata/rg-test-case-1b ${BIN_DIR}/kpt live apply --dry-run e2e/live/testdata/rg-test-case-1b > $OUTPUT_DIR/status -assertContains "namespace/rg-test-namespace configured (dry-run)" -assertContains "1 resource(s) applied. 0 created, 0 unchanged, 1 configured, 0 failed (dry-run)" -assertContains "pod/pod-b configured (dry-run)" -assertContains "pod/pod-c configured (dry-run)" -assertContains "pod/pod-d created (dry-run)" -assertContains "3 resource(s) applied. 1 created, 0 unchanged, 2 configured, 0 failed (dry-run)" -assertContains "pod/pod-a pruned (dry-run)" -assertContains "1 resource(s) pruned, 0 skipped, 0 failed (dry-run)" +assertContains "namespace/rg-test-namespace configured" +assertContains "pod/pod-b configured" +assertContains "pod/pod-c configured" +assertContains "pod/pod-d created" +assertContains "4 resource(s) applied. 1 created, 0 unchanged, 3 configured, 0 failed" +assertContains "pod/pod-a pruned" +assertContains "1 resource(s) pruned, 0 skipped, 0 failed" wait 2 # Validate resources in the cluster # ConfigMap inventory with four inventory items. @@ -626,11 +620,10 @@ echo "kpt live apply e2e/live/testdata/rg-test-case-1b" ${BIN_DIR}/kpt live apply e2e/live/testdata/rg-test-case-1b > $OUTPUT_DIR/status assertNotContains "installing inventory ResourceGroup CRD" # CRD already installed assertContains "namespace/rg-test-namespace unchanged" -assertContains "1 resource(s) applied. 0 created, 1 unchanged, 0 configured, 0 failed" assertContains "pod/pod-b unchanged" assertContains "pod/pod-c unchanged" assertContains "pod/pod-d created" -assertContains "3 resource(s) applied. 1 created, 2 unchanged, 0 configured, 0 failed" +assertContains "4 resource(s) applied. 1 created, 3 unchanged, 0 configured, 0 failed" assertContains "pod/pod-a pruned" assertContains "1 resource(s) pruned, 0 skipped, 0 failed" wait 2 @@ -647,12 +640,11 @@ printResult echo "[ResourceGroup] Testing basic destroy dry-run" echo "kpt live destroy --dry-run e2e/live/testdata/rg-test-case-1b" ${BIN_DIR}/kpt live destroy --dry-run e2e/live/testdata/rg-test-case-1b > $OUTPUT_DIR/status -assertContains "pod/pod-d deleted (dry-run)" -assertContains "pod/pod-c deleted (dry-run)" -assertContains "pod/pod-b deleted (dry-run)" -assertContains "3 resource(s) deleted, 0 skipped (dry-run)" -assertContains "namespace/rg-test-namespace deleted (dry-run)" -assertContains "1 resource(s) deleted, 0 skipped (dry-run)" +assertContains "pod/pod-d deleted" +assertContains "pod/pod-c deleted" +assertContains "pod/pod-b deleted" +assertContains "namespace/rg-test-namespace deleted" +assertContains "4 resource(s) deleted, 0 skipped" # Validate resources NOT DESTROYED in the cluster assertPodExists "pod-b" "rg-test-namespace" assertPodExists "pod-c" "rg-test-namespace" @@ -667,9 +659,8 @@ ${BIN_DIR}/kpt live destroy e2e/live/testdata/rg-test-case-1b > $OUTPUT_DIR/stat assertContains "pod/pod-d deleted" assertContains "pod/pod-c deleted" assertContains "pod/pod-b deleted" -assertContains "3 resource(s) deleted, 0 skipped" assertContains "namespace/rg-test-namespace deleted" -assertContains "1 resource(s) deleted, 0 skipped" +assertContains "4 resource(s) deleted, 0 skipped" # Validate resources NOT in the cluster assertPodNotExists "pod-b" "rg-test-namespace" assertPodNotExists "pod-c" "rg-test-namespace" @@ -684,7 +675,7 @@ cat e2e/live/testdata/stdin-test/pods.yaml | ${BIN_DIR}/kpt live apply - > $OUTP assertContains "pod/pod-a created" assertContains "pod/pod-b created" assertContains "pod/pod-c created" -assertContains "3 resource(s) applied. 3 created, 0 unchanged, 0 configured, 0 failed" +assertContains "4 resource(s) applied. 3 created, 1 unchanged, 0 configured, 0 failed" printResult echo "cat e2e/live/testdata/stdin-test/pods.yaml | kpt live status -" cat e2e/live/testdata/stdin-test/pods.yaml | ${BIN_DIR}/kpt live status - > $OUTPUT_DIR/status 2>&1 @@ -698,7 +689,7 @@ cat e2e/live/testdata/stdin-test/pods.yaml | ${BIN_DIR}/kpt live destroy - > $OU assertContains "pod/pod-a deleted" assertContains "pod/pod-b deleted" assertContains "pod/pod-c deleted" -assertContains "3 resource(s) deleted, 0 skipped" +assertContains "4 resource(s) deleted, 0 skipped" printResult # Test: kpt live apply continue-on-error @@ -829,11 +820,10 @@ echo "kpt live apply e2e/live/testdata/migrate-case-1b" cp -f e2e/live/testdata/migrate-case-1a/Kptfile e2e/live/testdata/migrate-case-1b ${BIN_DIR}/kpt live apply e2e/live/testdata/migrate-case-1b > $OUTPUT_DIR/status assertContains "namespace/test-rg-namespace unchanged" -assertContains "1 resource(s) applied. 0 created, 1 unchanged, 0 configured, 0 failed" assertContains "pod/pod-b unchanged" assertContains "pod/pod-c unchanged" assertContains "pod/pod-d created" -assertContains "3 resource(s) applied. 1 created, 2 unchanged, 0 configured, 0 failed" +assertContains "4 resource(s) applied. 1 created, 3 unchanged, 0 configured, 0 failed" assertContains "pod/pod-a pruned" assertContains "1 resource(s) pruned, 0 skipped, 0 failed" wait 2 diff --git a/e2e/testdata/fn-eval/error-in-pipe/.expected/config.yaml b/e2e/testdata/fn-eval/error-in-pipe/.expected/config.yaml index 214804680f..77df92ae74 100644 --- a/e2e/testdata/fn-eval/error-in-pipe/.expected/config.yaml +++ b/e2e/testdata/fn-eval/error-in-pipe/.expected/config.yaml @@ -21,8 +21,11 @@ stdErr: | Stderr: "[error] /// : failed to configure function: input namespace cannot be empty" Exit code: 1 - + [RUNNING] "gcr.io/kpt-fn/dne" [FAIL] "gcr.io/kpt-fn/dne" in 0s - Error: Function image "gcr.io/kpt-fn/dne" doesn't exist \ No newline at end of file + Stderr: + "docker: Error response from daemon: manifest for gcr.io/kpt-fn/dne:latest not found: manifest unknown: Failed to fetch \"latest\" from request \"/v2/kpt-fn/dne/manifests/latest\"." + "See 'docker run --help'." + Exit code: 125 \ No newline at end of file diff --git a/e2e/testdata/fn-eval/exec-function-stderr/.expected/config.yaml b/e2e/testdata/fn-eval/exec-function-stderr/.expected/config.yaml new file mode 100644 index 0000000000..f45537b8a7 --- /dev/null +++ b/e2e/testdata/fn-eval/exec-function-stderr/.expected/config.yaml @@ -0,0 +1,24 @@ +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +testType: eval +stdErr: | + [RUNNING] "./function.sh" + [PASS] "./function.sh" in 0s + Stderr: + "Hello world 0!" + "Hello world 1!" + "Hello world 2!" + "Hello world 3!" + ...(18 line(s) truncated, use '--truncate-output=false' to disable) diff --git a/e2e/testdata/fn-eval/exec-function-stderr/.expected/diff.patch b/e2e/testdata/fn-eval/exec-function-stderr/.expected/diff.patch new file mode 100644 index 0000000000..cc985fa320 --- /dev/null +++ b/e2e/testdata/fn-eval/exec-function-stderr/.expected/diff.patch @@ -0,0 +1,21 @@ +diff --git a/resources.yaml b/resources.yaml +index e8ae6bb..297b99f 100644 +--- a/resources.yaml ++++ b/resources.yaml +@@ -15,7 +15,7 @@ apiVersion: apps/v1 + kind: Deployment + metadata: + name: nginx-deployment +- namespace: foo ++ namespace: bar + spec: + replicas: 3 + --- +@@ -23,6 +23,6 @@ apiVersion: custom.io/v1 + kind: Custom + metadata: + name: custom +- namespace: foo ++ namespace: bar + spec: + image: nginx:1.2.3 diff --git a/e2e/testdata/fn-eval/exec-function-stderr/.expected/exec.sh b/e2e/testdata/fn-eval/exec-function-stderr/.expected/exec.sh new file mode 100644 index 0000000000..4a180de52c --- /dev/null +++ b/e2e/testdata/fn-eval/exec-function-stderr/.expected/exec.sh @@ -0,0 +1,19 @@ +#! /bin/bash +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -eo pipefail + +kpt fn source \ + | kpt fn eval --exec ./function.sh diff --git a/e2e/testdata/fn-eval/exec-function-stderr/.krmignore b/e2e/testdata/fn-eval/exec-function-stderr/.krmignore new file mode 100644 index 0000000000..9d7a4007d6 --- /dev/null +++ b/e2e/testdata/fn-eval/exec-function-stderr/.krmignore @@ -0,0 +1 @@ +.expected diff --git a/e2e/testdata/fn-eval/exec-function-stderr/function.sh b/e2e/testdata/fn-eval/exec-function-stderr/function.sh new file mode 100755 index 0000000000..c9cb9d7bbe --- /dev/null +++ b/e2e/testdata/fn-eval/exec-function-stderr/function.sh @@ -0,0 +1,21 @@ +#! /usr/bin/env bash +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +sed -e 's/foo/bar/' + +for i in {0..20} +do + >&2 echo "Hello world $i!" +done diff --git a/e2e/testdata/live-apply/json-output/resources/fourth.yaml b/e2e/testdata/fn-eval/exec-function-stderr/resources.yaml similarity index 57% rename from e2e/testdata/live-apply/json-output/resources/fourth.yaml rename to e2e/testdata/fn-eval/exec-function-stderr/resources.yaml index 48e7567715..e8ae6bbaf6 100644 --- a/e2e/testdata/live-apply/json-output/resources/fourth.yaml +++ b/e2e/testdata/fn-eval/exec-function-stderr/resources.yaml @@ -11,27 +11,18 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. - apiVersion: apps/v1 kind: Deployment metadata: - name: fourth-nginx - namespace: json-output - annotations: - config.k8s.io/owning-inventory: json-output - config.kubernetes.io/depends-on: apps/namespaces/json-output/Deployment/third-nginx + name: nginx-deployment + namespace: foo spec: - replicas: 1 - selector: - matchLabels: - app: fourth-nginx - template: - metadata: - labels: - app: fourth-nginx - spec: - containers: - - name: nginx - image: nginx:1.14.2 - ports: - - containerPort: 80 \ No newline at end of file + replicas: 3 +--- +apiVersion: custom.io/v1 +kind: Custom +metadata: + name: custom + namespace: foo +spec: + image: nginx:1.2.3 diff --git a/e2e/testdata/fn-eval/fn-success-with-stderr/pkg/.expected/results.yaml b/e2e/testdata/fn-eval/fn-success-with-stderr/pkg/.expected/results.yaml index cd73b3c490..67f0c37a90 100644 --- a/e2e/testdata/fn-eval/fn-success-with-stderr/pkg/.expected/results.yaml +++ b/e2e/testdata/fn-eval/fn-success-with-stderr/pkg/.expected/results.yaml @@ -5,6 +5,5 @@ metadata: exitCode: 0 items: - image: gcr.io/kpt-fn/starlark:v0.2 - stderr: | - function succeeded, reporting it on stderr + stderr: function succeeded, reporting it on stderr exitCode: 0 diff --git a/e2e/testdata/fn-eval/missing-fn-image/.expected/config.yaml b/e2e/testdata/fn-eval/missing-fn-image/.expected/config.yaml index cbef6dd615..fe98cda9a6 100644 --- a/e2e/testdata/fn-eval/missing-fn-image/.expected/config.yaml +++ b/e2e/testdata/fn-eval/missing-fn-image/.expected/config.yaml @@ -17,4 +17,4 @@ exitCode: 1 image: gcr.io/kpt-fn/dne # non-existing image args: namespace: staging -stdErr: 'Function image "gcr.io/kpt-fn/dne" doesn''t exist' +stdErr: 'gcr.io/kpt-fn/dne:latest not found' diff --git a/e2e/testdata/fn-render/exec-function-stderr/.expected/config.yaml b/e2e/testdata/fn-render/exec-function-stderr/.expected/config.yaml new file mode 100644 index 0000000000..9183295c7b --- /dev/null +++ b/e2e/testdata/fn-render/exec-function-stderr/.expected/config.yaml @@ -0,0 +1,24 @@ +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +allowExec: true +stdErr: | + [RUNNING] "./testdata/fn-render/exec-function-stderr/function.sh" + [PASS] "./testdata/fn-render/exec-function-stderr/function.sh" in 0s + Stderr: + "Hello world 0!" + "Hello world 1!" + "Hello world 2!" + "Hello world 3!" + ...(18 line(s) truncated, use '--truncate-output=false' to disable) diff --git a/e2e/testdata/fn-render/exec-function-stderr/.expected/diff.patch b/e2e/testdata/fn-render/exec-function-stderr/.expected/diff.patch new file mode 100644 index 0000000000..cc985fa320 --- /dev/null +++ b/e2e/testdata/fn-render/exec-function-stderr/.expected/diff.patch @@ -0,0 +1,21 @@ +diff --git a/resources.yaml b/resources.yaml +index e8ae6bb..297b99f 100644 +--- a/resources.yaml ++++ b/resources.yaml +@@ -15,7 +15,7 @@ apiVersion: apps/v1 + kind: Deployment + metadata: + name: nginx-deployment +- namespace: foo ++ namespace: bar + spec: + replicas: 3 + --- +@@ -23,6 +23,6 @@ apiVersion: custom.io/v1 + kind: Custom + metadata: + name: custom +- namespace: foo ++ namespace: bar + spec: + image: nginx:1.2.3 diff --git a/e2e/testdata/fn-render/exec-function-stderr/.krmignore b/e2e/testdata/fn-render/exec-function-stderr/.krmignore new file mode 100644 index 0000000000..9d7a4007d6 --- /dev/null +++ b/e2e/testdata/fn-render/exec-function-stderr/.krmignore @@ -0,0 +1 @@ +.expected diff --git a/e2e/testdata/fn-render/exec-function-stderr/Kptfile b/e2e/testdata/fn-render/exec-function-stderr/Kptfile new file mode 100644 index 0000000000..6f2fe1122f --- /dev/null +++ b/e2e/testdata/fn-render/exec-function-stderr/Kptfile @@ -0,0 +1,7 @@ +apiVersion: kpt.dev/v1 +kind: Kptfile +metadata: + name: app +pipeline: + mutators: + - exec: "./testdata/fn-render/exec-function-stderr/function.sh" diff --git a/e2e/testdata/fn-render/exec-function-stderr/function.sh b/e2e/testdata/fn-render/exec-function-stderr/function.sh new file mode 100755 index 0000000000..c9cb9d7bbe --- /dev/null +++ b/e2e/testdata/fn-render/exec-function-stderr/function.sh @@ -0,0 +1,21 @@ +#! /usr/bin/env bash +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +sed -e 's/foo/bar/' + +for i in {0..20} +do + >&2 echo "Hello world $i!" +done diff --git a/e2e/testdata/fn-render/exec-function-stderr/resources.yaml b/e2e/testdata/fn-render/exec-function-stderr/resources.yaml new file mode 100644 index 0000000000..e8ae6bbaf6 --- /dev/null +++ b/e2e/testdata/fn-render/exec-function-stderr/resources.yaml @@ -0,0 +1,28 @@ +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: nginx-deployment + namespace: foo +spec: + replicas: 3 +--- +apiVersion: custom.io/v1 +kind: Custom +metadata: + name: custom + namespace: foo +spec: + image: nginx:1.2.3 diff --git a/e2e/testdata/fn-render/exec-function-with-args/.expected/config.yaml b/e2e/testdata/fn-render/exec-function-with-args/.expected/config.yaml new file mode 100644 index 0000000000..1794ec551a --- /dev/null +++ b/e2e/testdata/fn-render/exec-function-with-args/.expected/config.yaml @@ -0,0 +1,15 @@ +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +allowExec: true diff --git a/e2e/testdata/fn-render/exec-function-with-args/.expected/diff.patch b/e2e/testdata/fn-render/exec-function-with-args/.expected/diff.patch new file mode 100644 index 0000000000..cc985fa320 --- /dev/null +++ b/e2e/testdata/fn-render/exec-function-with-args/.expected/diff.patch @@ -0,0 +1,21 @@ +diff --git a/resources.yaml b/resources.yaml +index e8ae6bb..297b99f 100644 +--- a/resources.yaml ++++ b/resources.yaml +@@ -15,7 +15,7 @@ apiVersion: apps/v1 + kind: Deployment + metadata: + name: nginx-deployment +- namespace: foo ++ namespace: bar + spec: + replicas: 3 + --- +@@ -23,6 +23,6 @@ apiVersion: custom.io/v1 + kind: Custom + metadata: + name: custom +- namespace: foo ++ namespace: bar + spec: + image: nginx:1.2.3 diff --git a/e2e/testdata/fn-render/exec-function-with-args/.krmignore b/e2e/testdata/fn-render/exec-function-with-args/.krmignore new file mode 100644 index 0000000000..9d7a4007d6 --- /dev/null +++ b/e2e/testdata/fn-render/exec-function-with-args/.krmignore @@ -0,0 +1 @@ +.expected diff --git a/e2e/testdata/fn-render/exec-function-with-args/Kptfile b/e2e/testdata/fn-render/exec-function-with-args/Kptfile new file mode 100644 index 0000000000..0d98dbb169 --- /dev/null +++ b/e2e/testdata/fn-render/exec-function-with-args/Kptfile @@ -0,0 +1,7 @@ +apiVersion: kpt.dev/v1 +kind: Kptfile +metadata: + name: app +pipeline: + mutators: + - exec: "sed -e 's/foo/bar/'" diff --git a/e2e/testdata/fn-render/exec-function-with-args/resources.yaml b/e2e/testdata/fn-render/exec-function-with-args/resources.yaml new file mode 100644 index 0000000000..e8ae6bbaf6 --- /dev/null +++ b/e2e/testdata/fn-render/exec-function-with-args/resources.yaml @@ -0,0 +1,28 @@ +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: nginx-deployment + namespace: foo +spec: + replicas: 3 +--- +apiVersion: custom.io/v1 +kind: Custom +metadata: + name: custom + namespace: foo +spec: + image: nginx:1.2.3 diff --git a/e2e/testdata/fn-render/exec-without-permissions/.expected/config.yaml b/e2e/testdata/fn-render/exec-without-permissions/.expected/config.yaml new file mode 100644 index 0000000000..840b9e0a49 --- /dev/null +++ b/e2e/testdata/fn-render/exec-without-permissions/.expected/config.yaml @@ -0,0 +1,16 @@ +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +exitCode: 1 +stdErr: 'Error: must run with `--allow-exec` option to allow running function binaries' \ No newline at end of file diff --git a/e2e/testdata/fn-render/exec-without-permissions/.krmignore b/e2e/testdata/fn-render/exec-without-permissions/.krmignore new file mode 100644 index 0000000000..9d7a4007d6 --- /dev/null +++ b/e2e/testdata/fn-render/exec-without-permissions/.krmignore @@ -0,0 +1 @@ +.expected diff --git a/e2e/testdata/fn-render/exec-without-permissions/Kptfile b/e2e/testdata/fn-render/exec-without-permissions/Kptfile new file mode 100644 index 0000000000..0d98dbb169 --- /dev/null +++ b/e2e/testdata/fn-render/exec-without-permissions/Kptfile @@ -0,0 +1,7 @@ +apiVersion: kpt.dev/v1 +kind: Kptfile +metadata: + name: app +pipeline: + mutators: + - exec: "sed -e 's/foo/bar/'" diff --git a/e2e/testdata/fn-render/exec-without-permissions/resources.yaml b/e2e/testdata/fn-render/exec-without-permissions/resources.yaml new file mode 100644 index 0000000000..e8ae6bbaf6 --- /dev/null +++ b/e2e/testdata/fn-render/exec-without-permissions/resources.yaml @@ -0,0 +1,28 @@ +# Copyright 2021 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: nginx-deployment + namespace: foo +spec: + replicas: 3 +--- +apiVersion: custom.io/v1 +kind: Custom +metadata: + name: custom + namespace: foo +spec: + image: nginx:1.2.3 diff --git a/e2e/testdata/fn-render/fn-success-with-stderr/.expected/results.yaml b/e2e/testdata/fn-render/fn-success-with-stderr/.expected/results.yaml index cd73b3c490..67f0c37a90 100644 --- a/e2e/testdata/fn-render/fn-success-with-stderr/.expected/results.yaml +++ b/e2e/testdata/fn-render/fn-success-with-stderr/.expected/results.yaml @@ -5,6 +5,5 @@ metadata: exitCode: 0 items: - image: gcr.io/kpt-fn/starlark:v0.2 - stderr: | - function succeeded, reporting it on stderr + stderr: function succeeded, reporting it on stderr exitCode: 0 diff --git a/e2e/testdata/fn-render/missing-fn-image/.expected/config.yaml b/e2e/testdata/fn-render/missing-fn-image/.expected/config.yaml index fab1748ae6..7bee7e3546 100644 --- a/e2e/testdata/fn-render/missing-fn-image/.expected/config.yaml +++ b/e2e/testdata/fn-render/missing-fn-image/.expected/config.yaml @@ -13,4 +13,4 @@ # limitations under the License. exitCode: 1 -stdErr: 'Error: Function image "gcr.io/kpt-fn/dne" doesn''t exist' +stdErr: 'gcr.io/kpt-fn/dne:latest not found' diff --git a/e2e/testdata/live-apply/apply-depends-on/config.yaml b/e2e/testdata/live-apply/apply-depends-on/config.yaml index 0a7223d53e..b954988009 100644 --- a/e2e/testdata/live-apply/apply-depends-on/config.yaml +++ b/e2e/testdata/live-apply/apply-depends-on/config.yaml @@ -14,11 +14,18 @@ parallel: true +kptArgs: + - "--reconcile-timeout=2m" + stdOut: | deployment.apps/first-nginx created - 1 resource(s) applied. 1 created, 0 unchanged, 0 configured, 0 failed + deployment.apps/first-nginx reconcile pending + deployment.apps/first-nginx reconciled deployment.apps/second-nginx created - 1 resource(s) applied. 1 created, 0 unchanged, 0 configured, 0 failed + 2 resource(s) applied. 2 created, 0 unchanged, 0 configured, 0 failed + deployment.apps/second-nginx reconcile pending + deployment.apps/second-nginx reconciled + 2 resource(s) reconciled, 0 skipped, 0 failed to reconcile, 0 timed out inventory: - group: apps diff --git a/e2e/testdata/live-apply/crd-and-cr/config.yaml b/e2e/testdata/live-apply/crd-and-cr/config.yaml index 96141a5fda..a0e5ef4a3b 100644 --- a/e2e/testdata/live-apply/crd-and-cr/config.yaml +++ b/e2e/testdata/live-apply/crd-and-cr/config.yaml @@ -13,11 +13,19 @@ # limitations under the License. parallel: true + +kptArgs: + - "--reconcile-timeout=1m" + stdOut: | customresourcedefinition.apiextensions.k8s.io/customs.kpt.dev created - 1 resource(s) applied. 1 created, 0 unchanged, 0 configured, 0 failed + customresourcedefinition.apiextensions.k8s.io/customs.kpt.dev reconcile pending + customresourcedefinition.apiextensions.k8s.io/customs.kpt.dev reconciled custom.kpt.dev/cr created - 1 resource(s) applied. 1 created, 0 unchanged, 0 configured, 0 failed + 2 resource(s) applied. 2 created, 0 unchanged, 0 configured, 0 failed + custom.kpt.dev/cr reconcile pending + custom.kpt.dev/cr reconciled + 2 resource(s) reconciled, 0 skipped, 0 failed to reconcile, 0 timed out inventory: - group: apiextensions.k8s.io kind: CustomResourceDefinition diff --git a/e2e/testdata/live-apply/dry-run-with-install-rg/config.yaml b/e2e/testdata/live-apply/dry-run-with-install-rg/config.yaml index 5f8ab9af10..add5fb5e03 100644 --- a/e2e/testdata/live-apply/dry-run-with-install-rg/config.yaml +++ b/e2e/testdata/live-apply/dry-run-with-install-rg/config.yaml @@ -18,8 +18,8 @@ kptArgs: - "--dry-run" - "--install-resource-group" stdOut: | - deployment.apps/nginx-deployment created (dry-run) - 1 resource(s) applied. 1 created, 0 unchanged, 0 configured, 0 failed (dry-run) + deployment.apps/nginx-deployment created + 1 resource(s) applied. 1 created, 0 unchanged, 0 configured, 0 failed stdErr: | installing inventory ResourceGroup CRD. exitCode: 0 \ No newline at end of file diff --git a/e2e/testdata/live-apply/install-rg-on-apply/config.yaml b/e2e/testdata/live-apply/install-rg-on-apply/config.yaml index 6ea6d5df3c..b0019b96ee 100644 --- a/e2e/testdata/live-apply/install-rg-on-apply/config.yaml +++ b/e2e/testdata/live-apply/install-rg-on-apply/config.yaml @@ -14,6 +14,8 @@ parallel: false noResourceGroup: true +kptArgs: + - "--reconcile-timeout=1m" stdOut: | deployment.apps/nginx-deployment created 1 resource(s) applied. 1 created, 0 unchanged, 0 configured, 0 failed diff --git a/e2e/testdata/live-apply/json-output/config.yaml b/e2e/testdata/live-apply/json-output/config.yaml index 892c7b859a..2a5526d4a8 100644 --- a/e2e/testdata/live-apply/json-output/config.yaml +++ b/e2e/testdata/live-apply/json-output/config.yaml @@ -15,25 +15,29 @@ parallel: true kptArgs: - "--output=json" + - "--reconcile-timeout=2m" stdOut: | {"eventType":"resourceApplied","group":"","kind":"ConfigMap","name":"cm","namespace":"json-output","operation":"Created","timestamp":"","type":"apply"} - {"eventType":"resourceApplied","group":"apps","kind":"Deployment","name":"third-nginx","namespace":"json-output","operation":"Created","timestamp":"","type":"apply"} + {"eventType":"resourceReconciled","group":"","kind":"ConfigMap","name":"cm","namespace":"json-output","operation":"Pending","timestamp":"","type":"wait"} + {"eventType":"resourceReconciled","group":"","kind":"ConfigMap","name":"cm","namespace":"json-output","operation":"Reconciled","timestamp":"","type":"wait"} + {"eventType":"resourceApplied","group":"apps","kind":"Deployment","name":"nginx","namespace":"json-output","operation":"Created","timestamp":"","type":"apply"} {"configuredCount":0,"count":2,"createdCount":2,"eventType":"completed","failedCount":0,"serverSideCount":0,"timestamp":"","type":"apply","unchangedCount":0} - {"eventType":"resourceApplied","group":"apps","kind":"Deployment","name":"fourth-nginx","namespace":"json-output","operation":"Created","timestamp":"","type":"apply"} - {"configuredCount":0,"count":1,"createdCount":1,"eventType":"completed","failedCount":0,"serverSideCount":0,"timestamp":"","type":"apply","unchangedCount":0} + {"eventType":"resourceReconciled","group":"apps","kind":"Deployment","name":"nginx","namespace":"json-output","operation":"Pending","timestamp":"","type":"wait"} + {"eventType":"resourceReconciled","group":"apps","kind":"Deployment","name":"nginx","namespace":"json-output","operation":"Reconciled","timestamp":"","type":"wait"} {"eventType":"resourcePruned","group":"apps","kind":"Deployment","name":"second-nginx","namespace":"json-output","operation":"Pruned","timestamp":"","type":"prune"} - {"eventType":"completed","pruned":1,"skipped":0,"timestamp":"","type":"prune"} + {"eventType":"resourceReconciled","group":"apps","kind":"Deployment","name":"second-nginx","namespace":"json-output","operation":"Pending","timestamp":"","type":"wait"} + {"eventType":"resourceReconciled","group":"apps","kind":"Deployment","name":"second-nginx","namespace":"json-output","operation":"Reconciled","timestamp":"","type":"wait"} {"eventType":"resourcePruned","group":"apps","kind":"Deployment","name":"first-nginx","namespace":"json-output","operation":"Pruned","timestamp":"","type":"prune"} - {"eventType":"completed","pruned":1,"skipped":0,"timestamp":"","type":"prune"} + {"eventType":"completed","failed":0,"pruned":2,"skipped":0,"timestamp":"","type":"prune"} + {"eventType":"resourceReconciled","group":"apps","kind":"Deployment","name":"first-nginx","namespace":"json-output","operation":"Pending","timestamp":"","type":"wait"} + {"eventType":"resourceReconciled","group":"apps","kind":"Deployment","name":"first-nginx","namespace":"json-output","operation":"Reconciled","timestamp":"","type":"wait"} + {"eventType":"completed","failed":0,"reconciled":4,"skipped":0,"timeout":0,"timestamp":"","type":"wait"} + inventory: - kind: ConfigMap name: cm namespace: json-output - group: apps kind: Deployment - name: third-nginx - namespace: json-output - - group: apps - kind: Deployment - name: fourth-nginx + name: nginx namespace: json-output \ No newline at end of file diff --git a/e2e/testdata/live-apply/json-output/resources/third.yaml b/e2e/testdata/live-apply/json-output/resources/dep.yaml similarity index 87% rename from e2e/testdata/live-apply/json-output/resources/third.yaml rename to e2e/testdata/live-apply/json-output/resources/dep.yaml index 123b05bf0f..eb0129b45c 100644 --- a/e2e/testdata/live-apply/json-output/resources/third.yaml +++ b/e2e/testdata/live-apply/json-output/resources/dep.yaml @@ -15,19 +15,19 @@ apiVersion: apps/v1 kind: Deployment metadata: - name: third-nginx + name: nginx namespace: json-output annotations: - config.k8s.io/owning-inventory: json-output + config.kubernetes.io/depends-on: apps/namespaces/json-output/ConfigMap/cm spec: replicas: 1 selector: matchLabels: - app: third-nginx + app: nginx template: metadata: labels: - app: third-nginx + app: nginx spec: containers: - name: nginx diff --git a/e2e/testdata/live-apply/prune-depends-on/config.yaml b/e2e/testdata/live-apply/prune-depends-on/config.yaml index a30d858ff0..2f85051b72 100644 --- a/e2e/testdata/live-apply/prune-depends-on/config.yaml +++ b/e2e/testdata/live-apply/prune-depends-on/config.yaml @@ -13,13 +13,24 @@ # limitations under the License. parallel: true + +kptArgs: + - "--reconcile-timeout=1m" + stdOut: | configmap/cm created 1 resource(s) applied. 1 created, 0 unchanged, 0 configured, 0 failed + configmap/cm reconcile pending + configmap/cm reconciled deployment.apps/second-nginx pruned - 1 resource(s) pruned, 0 skipped, 0 failed + deployment.apps/second-nginx reconcile pending + deployment.apps/second-nginx reconciled deployment.apps/first-nginx pruned - 1 resource(s) pruned, 0 skipped, 0 failed + 2 resource(s) pruned, 0 skipped, 0 failed to prune + deployment.apps/first-nginx reconcile pending + deployment.apps/first-nginx reconciled + 3 resource(s) reconciled, 0 skipped, 0 failed to reconcile, 0 timed out + inventory: - kind: ConfigMap name: cm diff --git a/go.mod b/go.mod index 489136b8c9..42323e643a 100644 --- a/go.mod +++ b/go.mod @@ -11,13 +11,12 @@ require ( github.com/igorsobreira/titlecase v0.0.0-20140109233139-4156b5b858ac github.com/otiai10/copy v1.6.0 github.com/philopon/go-toposort v0.0.0-20170620085441-9be86dbd762f - github.com/posener/complete/v2 v2.0.1-alpha.12 github.com/spf13/cobra v1.2.1 - github.com/spf13/pflag v1.0.5 github.com/stretchr/testify v1.7.0 github.com/xlab/treeprint v0.0.0-20181112141820-a009c3971eca + golang.org/x/mod v0.5.1 gotest.tools v2.2.0+incompatible - k8s.io/api v0.22.3 + k8s.io/api v0.22.3 // indirect k8s.io/apiextensions-apiserver v0.22.2 k8s.io/apimachinery v0.22.3 k8s.io/cli-runtime v0.22.2 @@ -25,8 +24,7 @@ require ( k8s.io/klog/v2 v2.10.0 k8s.io/kube-openapi v0.0.0-20211109043139-026bd182f079 // indirect k8s.io/kubectl v0.22.2 - k8s.io/utils v0.0.0-20210820185131-d34e5cb4466e - sigs.k8s.io/cli-utils v0.26.1-0.20211020064957-d62b5c62002d + sigs.k8s.io/cli-utils v0.27.0 sigs.k8s.io/kustomize/api v0.8.11 sigs.k8s.io/kustomize/kyaml v0.13.1-0.20211203194734-cd2c6a1ad117 ) diff --git a/go.sum b/go.sum index 973bff733f..96ca9d5ca6 100644 --- a/go.sum +++ b/go.sum @@ -486,13 +486,11 @@ github.com/grpc-ecosystem/grpc-gateway v1.16.0/go.mod h1:BDjrQk3hbvj6Nolgz8mAMFb github.com/hashicorp/consul/api v1.1.0/go.mod h1:VmuI/Lkw1nC05EYQWNKwWGbkg+FbDBtguAZLlVdkD9Q= github.com/hashicorp/consul/sdk v0.1.1/go.mod h1:VKf9jXwCTEY1QZP2MOLRhb5i/I/ssyNV1vwHyQBF0x8= github.com/hashicorp/errwrap v0.0.0-20141028054710-7554cd9344ce/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4= -github.com/hashicorp/errwrap v1.0.0 h1:hLrqtEDnRye3+sgx6z4qVLNuviH3MR5aQ0ykNJa/UYA= github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4= github.com/hashicorp/go-cleanhttp v0.5.1/go.mod h1:JpRdi6/HCYpAwUzNwuwqhbovhLtngrth3wmdIIUrZ80= github.com/hashicorp/go-immutable-radix v1.0.0/go.mod h1:0y9vanUI8NX6FsYoO3zeMjhV/C5i9g4Q3DwcSNZ4P60= github.com/hashicorp/go-msgpack v0.5.3/go.mod h1:ahLV/dePpqEmjfWmKiqvPkv/twdG7iPBM1vqhUKIvfM= github.com/hashicorp/go-multierror v0.0.0-20161216184304-ed905158d874/go.mod h1:JMRHfdO9jKNzS/+BTlxCjKNQHg/jZAft8U7LloJvN7I= -github.com/hashicorp/go-multierror v1.0.0 h1:iVjPR7a6H0tWELX5NxNe7bYopibicUzc7uPribsnS6o= github.com/hashicorp/go-multierror v1.0.0/go.mod h1:dHtQlpGsu+cZNNAkkCN/P3hoUDHhCYQXV3UM06sGGrk= github.com/hashicorp/go-rootcerts v1.0.0/go.mod h1:K6zTfqpRlCUIjkwsN4Z+hiSfzSTQa6eBIzfwKfwNnHU= github.com/hashicorp/go-sockaddr v1.0.0/go.mod h1:7Xibr9yA9JjQq1JpNB2Vw7kxv8xerXegt+ozgdvDeDU= @@ -697,12 +695,7 @@ github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINE github.com/pkg/sftp v1.10.1/go.mod h1:lYOWFsE0bwd1+KfKJaKeuokY15vzFx25BLbzYYoAxZI= github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= -github.com/posener/complete v1.1.1 h1:ccV59UEOTzVDnDUEFdT95ZzHVZ+5+158q8+SJb2QV5w= github.com/posener/complete v1.1.1/go.mod h1:em0nMJCgc9GFtwrmVmEMR/ZL6WyhyjMBndrE9hABlRI= -github.com/posener/complete/v2 v2.0.1-alpha.12 h1:0wvkuDfHb5vSZlNBYgpEH4XQHpF46MjLPHav8XC77Nc= -github.com/posener/complete/v2 v2.0.1-alpha.12/go.mod h1://JlL91cS2JV7rOl6LVHrRqBXoBUecJu3ILQPgbJiMQ= -github.com/posener/script v1.0.4 h1:nSuXW5ZdmFnQIueLB2s0qvs4oNsUloM1Zydzh75v42w= -github.com/posener/script v1.0.4/go.mod h1:Rg3ijooqulo05aGLyGsHoLmIOUzHUVK19WVgrYBPU/E= github.com/pquerna/cachecontrol v0.0.0-20171018203845-0dec1b30a021/go.mod h1:prYjPmNq4d1NPVmpShWobRqXY3q7Vp+80DqgxxUrUIA= github.com/prometheus/client_golang v0.0.0-20180209125602-c332b6f63c06/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw= github.com/prometheus/client_golang v0.9.1/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw= @@ -943,6 +936,8 @@ golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.4.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.4.1/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= +golang.org/x/mod v0.5.1 h1:OJxoQ/rynoF0dcCdI7cLPktw/hR2cueqYfjm43oqK38= +golang.org/x/mod v0.5.1/go.mod h1:5OXOZSfqPIIbmVBIIKWRFfZjPR0E5r58TLhUjH0a2Ro= golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= @@ -1434,8 +1429,8 @@ rsc.io/sampler v1.3.0/go.mod h1:T1hPZKmBbMNahiBKFy5HrXp6adAjACjK9JXDnKaTXpA= sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.0.14/go.mod h1:LEScyzhFmoF5pso/YSeBstl57mOzx9xlU9n85RGrDQg= sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.0.15/go.mod h1:LEScyzhFmoF5pso/YSeBstl57mOzx9xlU9n85RGrDQg= sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.0.22/go.mod h1:LEScyzhFmoF5pso/YSeBstl57mOzx9xlU9n85RGrDQg= -sigs.k8s.io/cli-utils v0.26.1-0.20211020064957-d62b5c62002d h1:yxJZ6HujyxXTLuHuZ8/HkzWy6g+eTpslhUzAzPpA9dE= -sigs.k8s.io/cli-utils v0.26.1-0.20211020064957-d62b5c62002d/go.mod h1:8ll2fyx+bzjbwmwUnKBQU+2LDbMDsxy44DiDZ+drALg= +sigs.k8s.io/cli-utils v0.27.0 h1:BxI7lPNn0fBZa5g4UwR+ShJyL4CCxELA6tLHbr2YrpU= +sigs.k8s.io/cli-utils v0.27.0/go.mod h1:8ll2fyx+bzjbwmwUnKBQU+2LDbMDsxy44DiDZ+drALg= sigs.k8s.io/controller-runtime v0.10.1 h1:+eLHgY/VrJWnfg6iXUqhCUqNXgPH1NZeP9drNAAgWlg= sigs.k8s.io/controller-runtime v0.10.1/go.mod h1:CQp8eyUQZ/Q7PJvnIrB6/hgfTC1kBkGylwsLgOQi1WY= sigs.k8s.io/kustomize/api v0.8.11 h1:LzQzlq6Z023b+mBtc6v72N2mSHYmN8x7ssgbf/hv0H8= diff --git a/internal/cmdapply/cmdapply.go b/internal/cmdapply/cmdapply.go index ffc4ff9d26..c7611b0a58 100644 --- a/internal/cmdapply/cmdapply.go +++ b/internal/cmdapply/cmdapply.go @@ -25,17 +25,16 @@ import ( "github.com/GoogleContainerTools/kpt/internal/util/argutil" "github.com/GoogleContainerTools/kpt/internal/util/strings" "github.com/GoogleContainerTools/kpt/pkg/live" - "github.com/GoogleContainerTools/kpt/thirdparty/cli-utils/flagutils" - "github.com/GoogleContainerTools/kpt/thirdparty/cli-utils/printers" "github.com/spf13/cobra" v1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" "k8s.io/cli-runtime/pkg/genericclioptions" "k8s.io/kubectl/pkg/cmd/util" + "sigs.k8s.io/cli-utils/cmd/flagutils" "sigs.k8s.io/cli-utils/pkg/apply" "sigs.k8s.io/cli-utils/pkg/common" "sigs.k8s.io/cli-utils/pkg/inventory" - status "sigs.k8s.io/cli-utils/pkg/util/factory" + "sigs.k8s.io/cli-utils/pkg/printers" ) // NewRunner returns a command runner @@ -81,6 +80,8 @@ func NewRunner(ctx context.Context, factory util.Factory, "If true, install the inventory ResourceGroup CRD before applying.") c.Flags().BoolVar(&r.dryRun, "dry-run", false, "dry-run apply for the resources in the package.") + c.Flags().BoolVar(&r.printStatusEvents, "status-events", false, + "Print status events (always enabled for table output)") return r } @@ -106,6 +107,7 @@ type Runner struct { pruneTimeout time.Duration inventoryPolicyString string dryRun bool + printStatusEvents bool inventoryPolicy inventory.InventoryPolicy prunePropPolicy v1.DeletionPropagation @@ -208,33 +210,36 @@ func runApply(r *Runner, invInfo inventory.InventoryInfo, objs []*unstructured.U // Run the applier. It will return a channel where we can receive updates // to keep track of progress and any issues. - poller, err := status.NewStatusPoller(r.factory) - if err != nil { - return err - } invClient, err := inventory.NewInventoryClient(r.factory, live.WrapInventoryObj, live.InvToUnstructuredFunc) if err != nil { return err } - applier, err := apply.NewApplier(r.factory, invClient, poller) + applier, err := apply.NewApplier(r.factory, invClient) if err != nil { return err } ch := applier.Run(r.ctx, invInfo, objs, apply.Options{ - ServerSideOptions: r.serverSideOptions, - PollInterval: r.period, - ReconcileTimeout: r.reconcileTimeout, - // If we are not waiting for status, tell the applier to not - // emit the events. - EmitStatusEvents: r.reconcileTimeout != time.Duration(0) || r.pruneTimeout != time.Duration(0), + ServerSideOptions: r.serverSideOptions, + PollInterval: r.period, + ReconcileTimeout: r.reconcileTimeout, + EmitStatusEvents: true, // We are always waiting for reconcile. DryRunStrategy: dryRunStrategy, PrunePropagationPolicy: r.prunePropPolicy, PruneTimeout: r.pruneTimeout, InventoryPolicy: r.inventoryPolicy, }) + // Print the preview strategy unless the output format is json. + if dryRunStrategy.ClientOrServerDryRun() && r.output != printers.JSONPrinter { + if dryRunStrategy.ServerDryRun() { + fmt.Println("Dry-run strategy: server") + } else { + fmt.Println("Dry-run strategy: client") + } + } + // The printer will print updates from the channel. It will block // until the channel is closed. printer := printers.GetPrinter(r.output, r.ioStreams) - return printer.Print(ch, dryRunStrategy) + return printer.Print(ch, dryRunStrategy, r.printStatusEvents) } diff --git a/internal/cmdcomplete/complete.go b/internal/cmdcomplete/complete.go deleted file mode 100644 index d77a3d52dd..0000000000 --- a/internal/cmdcomplete/complete.go +++ /dev/null @@ -1,81 +0,0 @@ -// Copyright 2019 Google LLC -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -// Package cmdcomplete contains the completion command -package cmdcomplete - -import ( - "strings" - - "github.com/GoogleContainerTools/kpt/internal/util/cmdutil" - kptfilev1 "github.com/GoogleContainerTools/kpt/pkg/api/kptfile/v1" - "github.com/posener/complete/v2" - "github.com/posener/complete/v2/predict" - "github.com/spf13/cobra" - "github.com/spf13/pflag" -) - -type VisitFlags func(cmd *cobra.Command, flag *pflag.Flag, cc *complete.Command) - -// Complete returns a completion command for a cobra command -func Complete(cmd *cobra.Command, skipHelp bool, visitFlags VisitFlags) *complete.Command { - cc := &complete.Command{ - Flags: map[string]complete.Predictor{}, - Sub: map[string]*complete.Command{}, - } - if strings.Contains(cmd.Use, "DIR") { - // if usage contains directory, then use a file predictor - cc.Args = predict.Dirs("*") - } - - // add each command - if !skipHelp { - cc.Sub["help"] = &complete.Command{Sub: map[string]*complete.Command{}} - } - for i := range cmd.Commands() { - c := cmd.Commands()[i] - if c.Hidden || c.Deprecated != "" { - continue - } - name := strings.Split(c.Use, " ")[0] - cc.Sub[name] = Complete(c, true, visitFlags) - if !skipHelp { - cc.Sub["help"].Sub[name] = cc.Sub[name] - } - } - - cmd.Flags().VisitAll(func(flag *pflag.Flag) { - if visitFlags != nil { - // extension support for other commands that embed this one - visitFlags(cmd, flag, cc) - } - if flag.Name == "strategy" { - cc.Flags[flag.Name] = predict.Options(predict.OptValues(kptfilev1.UpdateStrategiesAsStrings()...)) - return - } - if flag.Name == "pattern" { - cc.Flags[flag.Name] = predict.Options(predict.OptValues("%k_%n.yaml")) - return - } - if flag.Name == "image" || flag.Shorthand == "i" { - fnImages := cmdutil.FetchFunctionImages() - cc.Flags[flag.Name] = predict.Options(predict.OptValues(fnImages...)) - cc.Flags[flag.Shorthand] = predict.Options(predict.OptValues(fnImages...)) - return - } - cc.Flags[flag.Name] = predict.Nothing - }) - - return cc -} diff --git a/internal/cmddestroy/cmddestroy.go b/internal/cmddestroy/cmddestroy.go index 5ae7930e17..928fa082af 100644 --- a/internal/cmddestroy/cmddestroy.go +++ b/internal/cmddestroy/cmddestroy.go @@ -23,15 +23,14 @@ import ( "github.com/GoogleContainerTools/kpt/internal/util/argutil" "github.com/GoogleContainerTools/kpt/internal/util/strings" "github.com/GoogleContainerTools/kpt/pkg/live" - "github.com/GoogleContainerTools/kpt/thirdparty/cli-utils/flagutils" - "github.com/GoogleContainerTools/kpt/thirdparty/cli-utils/printers" "github.com/spf13/cobra" "k8s.io/cli-runtime/pkg/genericclioptions" "k8s.io/kubectl/pkg/cmd/util" + "sigs.k8s.io/cli-utils/cmd/flagutils" "sigs.k8s.io/cli-utils/pkg/apply" "sigs.k8s.io/cli-utils/pkg/common" "sigs.k8s.io/cli-utils/pkg/inventory" - status "sigs.k8s.io/cli-utils/pkg/util/factory" + "sigs.k8s.io/cli-utils/pkg/printers" ) func NewRunner(ctx context.Context, factory util.Factory, @@ -60,6 +59,8 @@ func NewRunner(ctx context.Context, factory util.Factory, fmt.Sprintf("%q and %q.", flagutils.InventoryPolicyStrict, flagutils.InventoryPolicyAdopt)) c.Flags().BoolVar(&r.dryRun, "dry-run", false, "dry-run apply for the resources in the package.") + c.Flags().BoolVar(&r.printStatusEvents, "status-events", false, + "Print status events (always enabled for table output)") return r } @@ -81,6 +82,7 @@ type Runner struct { output string inventoryPolicyString string dryRun bool + printStatusEvents bool inventoryPolicy inventory.InventoryPolicy @@ -154,26 +156,31 @@ func (r *Runner) runE(c *cobra.Command, args []string) error { func runDestroy(r *Runner, inv inventory.InventoryInfo, dryRunStrategy common.DryRunStrategy) error { // Run the destroyer. It will return a channel where we can receive updates // to keep track of progress and any issues. - poller, err := status.NewStatusPoller(r.factory) - if err != nil { - return err - } invClient, err := inventory.NewInventoryClient(r.factory, live.WrapInventoryObj, live.InvToUnstructuredFunc) if err != nil { return err } - destroyer, err := apply.NewDestroyer(r.factory, invClient, poller) + destroyer, err := apply.NewDestroyer(r.factory, invClient) if err != nil { return err } options := apply.DestroyerOptions{ - InventoryPolicy: r.inventoryPolicy, - DryRunStrategy: dryRunStrategy, + InventoryPolicy: r.inventoryPolicy, + DryRunStrategy: dryRunStrategy, + EmitStatusEvents: true, } ch := destroyer.Run(context.Background(), inv, options) + // Print the preview strategy unless the output format is json. + if dryRunStrategy.ClientOrServerDryRun() && r.output != printers.JSONPrinter { + if dryRunStrategy.ServerDryRun() { + fmt.Println("Dry-run strategy: server") + } else { + fmt.Println("Dry-run strategy: client") + } + } // The printer will print updates from the channel. It will block // until the channel is closed. printer := printers.GetPrinter(r.output, r.ioStreams) - return printer.Print(ch, dryRunStrategy) + return printer.Print(ch, dryRunStrategy, r.printStatusEvents) } diff --git a/internal/cmdfndoc/cmdfndoc.go b/internal/cmdfndoc/cmdfndoc.go index ae3c11516b..11e9324581 100644 --- a/internal/cmdfndoc/cmdfndoc.go +++ b/internal/cmdfndoc/cmdfndoc.go @@ -43,6 +43,9 @@ func NewRunner(ctx context.Context, parent string) *Runner { } r.Command = c c.Flags().StringVarP(&r.Image, "image", "i", "", "kpt function image name") + _ = r.Command.RegisterFlagCompletionFunc("image", func(cmd *cobra.Command, args []string, toComplete string) ([]string, cobra.ShellCompDirective) { + return cmdutil.FetchFunctionImages(), cobra.ShellCompDirectiveDefault + }) cmdutil.FixDocs("kpt", parent, c) return r } diff --git a/internal/cmdget/cmdget.go b/internal/cmdget/cmdget.go index 43aee5340d..9cb9e14aba 100644 --- a/internal/cmdget/cmdget.go +++ b/internal/cmdget/cmdget.go @@ -53,6 +53,9 @@ func NewRunner(ctx context.Context, parent string) *Runner { c.Flags().StringVar(&r.strategy, "strategy", string(kptfilev1.ResourceMerge), "update strategy that should be used when updating this package -- must be one of: "+ strings.Join(kptfilev1.UpdateStrategiesAsStrings(), ",")) + _ = c.RegisterFlagCompletionFunc("strategy", func(cmd *cobra.Command, args []string, toComplete string) ([]string, cobra.ShellCompDirective) { + return kptfilev1.UpdateStrategiesAsStrings(), cobra.ShellCompDirectiveDefault + }) return r } diff --git a/internal/cmdmigrate/migratecmd.go b/internal/cmdmigrate/migratecmd.go index ff06693d70..51981dbf9f 100644 --- a/internal/cmdmigrate/migratecmd.go +++ b/internal/cmdmigrate/migratecmd.go @@ -231,16 +231,18 @@ func (mr *MigrateRunner) retrieveConfigMapInv(reader io.Reader, args []string) ( if err != nil { return nil, err } - cmInv, _, err := mr.cmLoader.InventoryInfo(objs) + cmInvObj, _, err := inventory.SplitUnstructureds(objs) if err != nil { + return nil, err + } + if cmInvObj == nil { // No ConfigMap inventory means the migration has already run before. - if _, ok := err.(inventory.NoInventoryObjError); ok { //nolint - fmt.Fprintln(mr.ioStreams.Out, "no ConfigMap inventory...completed") - } - } else { - fmt.Fprintf(mr.ioStreams.Out, "success (inventory-id: %s)\n", cmInv.ID()) + fmt.Fprintln(mr.ioStreams.Out, "no ConfigMap inventory...completed") + return nil, inventory.NoInventoryObjError{} } - return cmInv, err + cmInv := inventory.WrapInventoryInfoObj(cmInvObj) + fmt.Fprintf(mr.ioStreams.Out, "success (inventory-id: %s)\n", cmInv.ID()) + return cmInv, nil } // retrieveInvObjs returns the object references from the passed diff --git a/internal/cmdmigrate/migratecmd_test.go b/internal/cmdmigrate/migratecmd_test.go index e83eabf2d5..76c999e1a3 100644 --- a/internal/cmdmigrate/migratecmd_test.go +++ b/internal/cmdmigrate/migratecmd_test.go @@ -286,14 +286,14 @@ func TestKptMigrate_migrateObjs(t *testing.T) { }, "One migrate object is valid": { invObj: kptfileStr, - objs: []object.ObjMetadata{object.UnstructuredToObjMetaOrDie(pod1)}, + objs: []object.ObjMetadata{object.UnstructuredToObjMetadata(pod1)}, isError: false, }, "Multiple migrate objects are valid": { invObj: kptfileStr, objs: []object.ObjMetadata{ - object.UnstructuredToObjMetaOrDie(pod1), - object.UnstructuredToObjMetaOrDie(pod2), + object.UnstructuredToObjMetadata(pod1), + object.UnstructuredToObjMetadata(pod2), }, isError: false, }, diff --git a/internal/cmdpush/cmdpush.go b/internal/cmdpush/cmdpush.go index 33e9c8a49a..20654e9e05 100644 --- a/internal/cmdpush/cmdpush.go +++ b/internal/cmdpush/cmdpush.go @@ -121,7 +121,7 @@ func (r *Runner) preRunE(_ *cobra.Command, args []string) error { } if r.Origin != "" { - _, err := parse.ParseArgs(r.ctx, args, parse.Options{ + _, err := parse.ParseArgs(r.ctx, []string{r.Origin, path}, parse.Options{ SetOci: func(oci *kptfilev1.Oci) error { r.Push.Origin = remote.NewOciOrigin(oci) return nil diff --git a/internal/cmdrender/cmdrender.go b/internal/cmdrender/cmdrender.go index ee5bca1de4..2faf44be65 100644 --- a/internal/cmdrender/cmdrender.go +++ b/internal/cmdrender/cmdrender.go @@ -45,8 +45,10 @@ func NewRunner(ctx context.Context, parent string) *Runner { "path to a directory to save function results") c.Flags().StringVarP(&r.dest, "output", "o", "", fmt.Sprintf("output resources are written to provided location. Allowed values: %s|%s|", cmdutil.Stdout, cmdutil.Unwrap)) - c.Flags().StringVar(&r.imagePullPolicy, "image-pull-policy", string(fnruntime.AlwaysPull), + c.Flags().StringVar(&r.imagePullPolicy, "image-pull-policy", string(fnruntime.IfNotPresentPull), fmt.Sprintf("pull image before running the container. It must be one of %s, %s and %s.", fnruntime.AlwaysPull, fnruntime.IfNotPresentPull, fnruntime.NeverPull)) + c.Flags().BoolVar(&r.allowExec, "allow-exec", false, + "allow binary executable to be run during pipeline execution.") cmdutil.FixDocs("kpt", parent, c) r.Command = c return r @@ -61,6 +63,7 @@ type Runner struct { pkgPath string resultsDirPath string imagePullPolicy string + allowExec bool dest string Command *cobra.Command ctx context.Context @@ -98,10 +101,6 @@ func (r *Runner) preRunE(c *cobra.Command, args []string) error { } func (r *Runner) runE(c *cobra.Command, _ []string) error { - err := cmdutil.DockerCmdAvailable() - if err != nil { - return err - } var output io.Writer outContent := bytes.Buffer{} if r.dest != "" { @@ -114,9 +113,9 @@ func (r *Runner) runE(c *cobra.Command, _ []string) error { ResultsDirPath: r.resultsDirPath, Output: output, ImagePullPolicy: cmdutil.StringToImagePullPolicy(r.imagePullPolicy), + AllowExec: r.allowExec, } - err = executor.Execute(r.ctx) - if err != nil { + if err := executor.Execute(r.ctx); err != nil { return err } diff --git a/internal/cmdrender/executor.go b/internal/cmdrender/executor.go index b321a90ea3..8489e73db5 100644 --- a/internal/cmdrender/executor.go +++ b/internal/cmdrender/executor.go @@ -28,6 +28,7 @@ import ( "github.com/GoogleContainerTools/kpt/internal/printer" "github.com/GoogleContainerTools/kpt/internal/types" "github.com/GoogleContainerTools/kpt/internal/util/attribution" + "github.com/GoogleContainerTools/kpt/internal/util/cmdutil" "github.com/GoogleContainerTools/kpt/internal/util/printerutil" fnresult "github.com/GoogleContainerTools/kpt/pkg/api/fnresult/v1" kptfilev1 "github.com/GoogleContainerTools/kpt/pkg/api/kptfile/v1" @@ -37,12 +38,15 @@ import ( "sigs.k8s.io/kustomize/kyaml/yaml" ) +var errAllowedExecNotSpecified error = fmt.Errorf("must run with `--allow-exec` option to allow running function binaries") + // Executor hydrates a given pkg. type Executor struct { PkgPath string ResultsDirPath string Output io.Writer ImagePullPolicy fnruntime.ImagePullPolicy + AllowExec bool } // Execute runs a pipeline. @@ -62,6 +66,7 @@ func (e *Executor) Execute(ctx context.Context) error { pkgs: map[types.UniquePath]*pkgNode{}, fnResults: fnresult.NewResultList(), imagePullPolicy: e.ImagePullPolicy, + allowExec: e.AllowExec, } if _, err = hydrate(ctx, root, hctx); err != nil { @@ -162,6 +167,15 @@ type hydrationContext struct { // imagePullPolicy controls the image pulling behavior. imagePullPolicy fnruntime.ImagePullPolicy + + // allowExec determines if function binary executable are allowed + // to be run during pipeline execution. Running function binaries is a + // privileged operation, so explicit permission is required. + allowExec bool + + // bookkeeping to ensure docker command availability check is done once + // during rendering + dockerCheckDone bool } // @@ -426,11 +440,22 @@ func (pn *pkgNode) runValidators(ctx context.Context, hctx *hydrationContext, in if err != nil { return err } + var validator kio.Filter displayResourceCount := false if len(fn.Selectors) > 0 { displayResourceCount = true } - validator, err := fnruntime.NewContainerRunner(ctx, &fn, pn.pkg.UniquePath, hctx.fnResults, hctx.imagePullPolicy, displayResourceCount) + if fn.Exec != "" && !hctx.allowExec { + return errAllowedExecNotSpecified + } + if fn.Image != "" && !hctx.dockerCheckDone { + err := cmdutil.DockerCmdAvailable() + if err != nil { + return err + } + hctx.dockerCheckDone = true + } + validator, err = fnruntime.NewRunner(ctx, &fn, pn.pkg.UniquePath, hctx.fnResults, hctx.imagePullPolicy, displayResourceCount) if err != nil { return err } @@ -527,17 +552,28 @@ func pathRelToRoot(rootPkgPath, subPkgPath, resourcePath string) (relativePath s func fnChain(ctx context.Context, hctx *hydrationContext, pkgPath types.UniquePath, fns []kptfilev1.Function) ([]kio.Filter, error) { var runners []kio.Filter for i := range fns { + var err error + var runner kio.Filter fn := fns[i] - fn.Image = fnruntime.AddDefaultImagePathPrefix(fn.Image) displayResourceCount := false if len(fn.Selectors) > 0 { displayResourceCount = true } - r, err := fnruntime.NewContainerRunner(ctx, &fn, pkgPath, hctx.fnResults, hctx.imagePullPolicy, displayResourceCount) + if fn.Exec != "" && !hctx.allowExec { + return nil, errAllowedExecNotSpecified + } + if fn.Image != "" && !hctx.dockerCheckDone { + err := cmdutil.DockerCmdAvailable() + if err != nil { + return nil, err + } + hctx.dockerCheckDone = true + } + runner, err = fnruntime.NewRunner(ctx, &fn, pkgPath, hctx.fnResults, hctx.imagePullPolicy, displayResourceCount) if err != nil { return nil, err } - runners = append(runners, r) + runners = append(runners, runner) } return runners, nil } diff --git a/internal/cmdupdate/cmdupdate.go b/internal/cmdupdate/cmdupdate.go index 722cda2b3c..546d945c1a 100644 --- a/internal/cmdupdate/cmdupdate.go +++ b/internal/cmdupdate/cmdupdate.go @@ -53,6 +53,9 @@ func NewRunner(ctx context.Context, parent string) *Runner { "the update strategy that will be used when updating the package. This will change "+ "the default strategy for the package -- must be one of: "+ strings.Join(kptfilev1.UpdateStrategiesAsStrings(), ",")) + _ = c.RegisterFlagCompletionFunc("strategy", func(cmd *cobra.Command, args []string, toComplete string) ([]string, cobra.ShellCompDirective) { + return kptfilev1.UpdateStrategiesAsStrings(), cobra.ShellCompDirectiveDefault + }) cmdutil.FixDocs("kpt", parent, c) r.Command = c return r diff --git a/internal/docs/generated/fndocs/docs.go b/internal/docs/generated/fndocs/docs.go index f37a533c7f..e8d677b96a 100644 --- a/internal/docs/generated/fndocs/docs.go +++ b/internal/docs/generated/fndocs/docs.go @@ -239,6 +239,11 @@ Args: Flags: + --allow-exec: + Allow executable binaries to run as function. Note that executable binaries + can perform privileged operations on your system, so ensure that binaries + referred in the pipeline are trusted and safe to execute. + --image-pull-policy: If the image should be pulled before rendering the package(s). It can be set to one of always, ifNotPresent, never. If unspecified, always will be the diff --git a/internal/errors/resolver/live.go b/internal/errors/resolver/live.go index 00f42ff892..d01c6dbddd 100644 --- a/internal/errors/resolver/live.go +++ b/internal/errors/resolver/live.go @@ -19,9 +19,9 @@ import ( "github.com/GoogleContainerTools/kpt/internal/cmdutil" "github.com/GoogleContainerTools/kpt/internal/errors" "github.com/GoogleContainerTools/kpt/pkg/live" - "sigs.k8s.io/cli-utils/pkg/apply/taskrunner" "sigs.k8s.io/cli-utils/pkg/inventory" "sigs.k8s.io/cli-utils/pkg/manifestreader" + "sigs.k8s.io/cli-utils/pkg/print/common" ) //nolint:gochecknoinits @@ -42,14 +42,6 @@ package or automatically deleting omitted resources (pruning). Error: Package has multiple inventory object templates. The package should have one and only one inventory object template. -` - //nolint:lll - timeoutErrorMsg = ` -Error: Timeout after {{printf "%.0f" .err.Timeout.Seconds}} seconds waiting for {{printf "%d" (len .err.TimedOutResources)}} out of {{printf "%d" (len .err.Identifiers)}} resources to reach condition {{ .err.Condition}}:{{ printf "\n" }} - -{{- range .err.TimedOutResources}} -{{printf "%s/%s %s %s" .Identifier.GroupKind.Kind .Identifier.Name .Status .Message }} -{{- end}} ` resourceGroupCRDInstallErrorMsg = ` @@ -85,15 +77,13 @@ Details: ` unknownTypesMsg = ` -Error: {{ printf "%d" (len .err.GroupKinds) }} resource types could not be found in the cluster or as CRDs among the applied resources. +Error: {{ printf "%d" (len .err.GroupVersionKinds) }} resource types could not be found in the cluster or as CRDs among the applied resources. Resource types: -{{- range .err.GroupKinds}} +{{- range .err.GroupVersionKinds}} {{ printf "%s" .String }} {{- end}} ` - - TimeoutErrorExitCode = 3 ) // liveErrorResolver is an implementation of the ErrorResolver interface @@ -119,16 +109,6 @@ func (*liveErrorResolver) Resolve(err error) (ResolvedResult, bool) { }, true } - var timeoutError *taskrunner.TimeoutError - if errors.As(err, &timeoutError) { - return ResolvedResult{ - Message: ExecuteTemplate(timeoutErrorMsg, map[string]interface{}{ - "err": *timeoutError, - }), - ExitCode: TimeoutErrorExitCode, - }, true - } - var resourceGroupCRDInstallError *cmdutil.ResourceGroupCRDInstallError if errors.As(err, &resourceGroupCRDInstallError) { return ResolvedResult{ @@ -182,5 +162,14 @@ func (*liveErrorResolver) Resolve(err error) (ResolvedResult, bool) { }), }, true } + + var resultError *common.ResultError + if errors.As(err, &resultError) { + return ResolvedResult{ + Message: resultError.Error(), + ExitCode: 3, + }, true + } + return ResolvedResult{}, false } diff --git a/internal/errors/resolver/live_test.go b/internal/errors/resolver/live_test.go index 08164610df..b19f563055 100644 --- a/internal/errors/resolver/live_test.go +++ b/internal/errors/resolver/live_test.go @@ -17,14 +17,11 @@ package resolver import ( "strings" "testing" - "time" "github.com/GoogleContainerTools/kpt/internal/errors" "github.com/stretchr/testify/assert" "k8s.io/apimachinery/pkg/runtime/schema" - "sigs.k8s.io/cli-utils/pkg/apply/taskrunner" - "sigs.k8s.io/cli-utils/pkg/kstatus/status" - "sigs.k8s.io/cli-utils/pkg/object" + "sigs.k8s.io/cli-utils/pkg/manifestreader" ) func TestLiveErrorResolver(t *testing.T) { @@ -34,39 +31,21 @@ func TestLiveErrorResolver(t *testing.T) { }{ "nested timeoutError": { err: &errors.Error{ - Err: &taskrunner.TimeoutError{ - Identifiers: []object.ObjMetadata{ + Err: &manifestreader.UnknownTypesError{ + GroupVersionKinds: []schema.GroupVersionKind{ { - GroupKind: schema.GroupKind{ - Group: "apps", - Kind: "Deployment", - }, - Name: "test", - Namespace: "test-ns", - }, - }, - Condition: taskrunner.AllCurrent, - Timeout: 3 * time.Second, - TimedOutResources: []taskrunner.TimedOutResource{ - { - Identifier: object.ObjMetadata{ - GroupKind: schema.GroupKind{ - Group: "apps", - Kind: "Deployment", - }, - Name: "test", - Namespace: "test-ns", - }, - Status: status.InProgressStatus, - Message: "this is a test", + Group: "apps", + Version: "v1", + Kind: "Deployment", }, }, }, }, expected: ` -Error: Timeout after 3 seconds waiting for 1 out of 1 resources to reach condition AllCurrent: +Error: 1 resource types could not be found in the cluster or as CRDs among the applied resources. -Deployment/test InProgress this is a test +Resource types: +apps/v1, Kind=Deployment `, }, } diff --git a/internal/fnruntime/container.go b/internal/fnruntime/container.go index 4eca4358f7..092a97f9ea 100644 --- a/internal/fnruntime/container.go +++ b/internal/fnruntime/container.go @@ -15,6 +15,7 @@ package fnruntime import ( + "bufio" "bytes" "context" goerrors "errors" @@ -34,11 +35,10 @@ import ( type containerNetworkName string const ( - networkNameNone containerNetworkName = "none" - networkNameHost containerNetworkName = "host" - defaultLongTimeout time.Duration = 5 * time.Minute - defaultShortTimeout time.Duration = 5 * time.Second - dockerBin string = "docker" + networkNameNone containerNetworkName = "none" + networkNameHost containerNetworkName = "host" + defaultLongTimeout time.Duration = 5 * time.Minute + dockerBin string = "docker" AlwaysPull ImagePullPolicy = "Always" IfNotPresentPull ImagePullPolicy = "IfNotPresent" @@ -85,13 +85,6 @@ type ContainerFn struct { // It reads the input from the given reader and writes the output // to the provided writer. func (f *ContainerFn) Run(reader io.Reader, writer io.Writer) error { - // check and pull image before running to avoid polluting CLI - // output - err := f.prepareImage() - if err != nil { - return err - } - errSink := bytes.Buffer{} cmd, cancel := f.getDockerCmd() defer cancel() @@ -105,7 +98,7 @@ func (f *ContainerFn) Run(reader io.Reader, writer io.Writer) error { return &ExecError{ OriginalErr: exitErr, ExitCode: exitErr.ExitCode(), - Stderr: errSink.String(), + Stderr: filterDockerCLIOutput(&errSink), TruncateOutput: printer.TruncateOutput, } } @@ -113,7 +106,7 @@ func (f *ContainerFn) Run(reader io.Reader, writer io.Writer) error { } if errSink.Len() > 0 { - f.FnResult.Stderr = errSink.String() + f.FnResult.Stderr = filterDockerCLIOutput(&errSink) } return nil } @@ -135,8 +128,16 @@ func (f *ContainerFn) getDockerCmd() (*exec.Cmd, context.CancelFunc) { "--user", uidgid, "--security-opt=no-new-privileges", } - if f.ImagePullPolicy == NeverPull { + + switch f.ImagePullPolicy { + case NeverPull: args = append(args, "--pull", "never") + case AlwaysPull: + args = append(args, "--pull", "pull") + case IfNotPresentPull: + args = append(args, "--pull", "missing") + default: + args = append(args, "--pull", "missing") } for _, storageMount := range f.StorageMounts { args = append(args, "--mount", storageMount.String()) @@ -173,62 +174,6 @@ func NewContainerEnvFromStringSlice(envStr []string) *runtimeutil.ContainerEnv { return ce } -// prepareImage will check local images and pull it if it doesn't -// exist. -func (f *ContainerFn) prepareImage() error { - // If ImagePullPolicy is set to "never", we don't need to do anything here. - if f.ImagePullPolicy == NeverPull { - return nil - } - - // If ImagePullPolicy is set to "ifNotPresent", we scan the local images - // first. If there is a match, we just return. This can be useful for local - // development to prevent the remote image to accidentally override the - // local image when they use the same name and tag. - if f.ImagePullPolicy == IfNotPresentPull { - if foundInLocalCache := f.checkImageExistence(); foundInLocalCache { - return nil - } - } - - // If ImagePullPolicy is set to always (which is the default), we will try - // to pull the image regardless if the tag has been seen in the local cache. - // This can help to ensure we have the latest release for "moving tags" like - // v1 and v1.2. The performance cost is very minimal, since `docker pull` - // checks the SHA first and only pull the missing docker layer(s). - args := []string{"image", "pull", f.Image} - // setup timeout - timeout := defaultLongTimeout - if f.Timeout != 0 { - timeout = f.Timeout - } - ctx, cancel := context.WithTimeout(context.Background(), timeout) - defer cancel() - cmd := exec.CommandContext(ctx, dockerBin, args...) - output, err := cmd.CombinedOutput() - if err != nil { - return &ContainerImageError{ - Image: f.Image, - Output: string(output), - } - } - return nil -} - -// checkImageExistence returns true if the image does exist in -// local cache -func (f *ContainerFn) checkImageExistence() bool { - args := []string{"image", "inspect", f.Image} - ctx, cancel := context.WithTimeout(context.Background(), defaultShortTimeout) - defer cancel() - cmd := exec.CommandContext(ctx, dockerBin, args...) - if _, err := cmd.CombinedOutput(); err == nil { - // image exists locally - return true - } - return false -} - // AddDefaultImagePathPrefix adds default gcr.io/kpt-fn/ path prefix to image if only image name is specified func AddDefaultImagePathPrefix(image string) string { if !strings.Contains(image, "/") { @@ -250,3 +195,48 @@ func (e *ContainerImageError) Error() string { "Error: Function image %q doesn't exist remotely. If you are developing new functions locally, you can choose to set the image pull policy to ifNotPresent or never.\n%v", e.Image, e.Output) } + +// filterDockerCLIOutput filters out docker CLI messages +// from the given buffer. +func filterDockerCLIOutput(in io.Reader) string { + s := bufio.NewScanner(in) + var lines []string + + for s.Scan() { + txt := s.Text() + if !isdockerCLIoutput(txt) { + lines = append(lines, txt) + } + } + return strings.Join(lines, "\n") +} + +// isdockerCLIoutput is helper method to determine if +// the given string is a docker CLI output message. +// Example docker output: +// "Unable to find image 'gcr.io/kpt-fn/starlark:v0.3' locally" +// "v0.3: Pulling from kpt-fn/starlark" +// "4e9f2cdf4387: Already exists" +// "aafbf7df3ddf: Pulling fs layer" +// "aafbf7df3ddf: Verifying Checksum" +// "aafbf7df3ddf: Download complete" +// "6b759ab96cb2: Waiting" +// "aafbf7df3ddf: Pull complete" +// "Digest: sha256:c347e28606fa1a608e8e02e03541a5a46e4a0152005df4a11e44f6c4ab1edd9a" +// "Status: Downloaded newer image for gcr.io/kpt-fn/starlark:v0.3" +// +func isdockerCLIoutput(s string) bool { + if strings.Contains(s, ": Already exists") || + strings.Contains(s, ": Pulling fs layer") || + strings.Contains(s, ": Verifying Checksum") || + strings.Contains(s, ": Download complete") || + strings.Contains(s, ": Pulling from") || + strings.Contains(s, ": Waiting") || + strings.Contains(s, ": Pull complete") || + strings.Contains(s, "Digest: sha256") || + strings.Contains(s, "Status: Downloaded newer image") || + strings.Contains(s, "Unable to find image") { + return true + } + return false +} diff --git a/internal/fnruntime/container_test.go b/internal/fnruntime/container_test.go index 11161f1cd5..f6ac50ac7b 100644 --- a/internal/fnruntime/container_test.go +++ b/internal/fnruntime/container_test.go @@ -24,6 +24,7 @@ import ( "github.com/GoogleContainerTools/kpt/internal/fnruntime" "github.com/GoogleContainerTools/kpt/internal/printer" + fnresult "github.com/GoogleContainerTools/kpt/pkg/api/fnresult/v1" "github.com/stretchr/testify/assert" ) @@ -54,6 +55,9 @@ func TestContainerFn(t *testing.T) { instance := fnruntime.ContainerFn{ Ctx: printer.WithContext(ctx, printer.New(nil, errBuff)), Image: tt.image, + FnResult: &fnresult.Result{ + Image: tt.image, + }, } input := bytes.NewBufferString(tt.input) output := &bytes.Buffer{} diff --git a/internal/fnruntime/fnerrors_test.go b/internal/fnruntime/fnerrors_test.go index b022723b3b..b308d72120 100644 --- a/internal/fnruntime/fnerrors_test.go +++ b/internal/fnruntime/fnerrors_test.go @@ -15,6 +15,7 @@ package fnruntime import ( + "bytes" "testing" "github.com/stretchr/testify/assert" @@ -138,3 +139,76 @@ error message`, }) } } + +func TestDockerCLIOutputFilter(t *testing.T) { + + testcases := []struct { + name string + input string + expected string + }{ + { + name: "should filter docker CLI output successfully", + input: `Unable to find image 'gcr.io/kpt-fn/starlark:v0.3' locally +v0.3: Pulling from kpt-fn/starlark +4e9f2cdf4387: Already exists +aafbf7df3ddf: Pulling fs layer +aafbf7df3ddf: Verifying Checksum +aafbf7df3ddf: Download complete +aafbf7df3ddf: Pull complete +6b759ab96cb2: Waiting +Digest: sha256:c347e28606fa1a608e8e02e03541a5a46e4a0152005df4a11e44f6c4ab1edd9a +Status: Downloaded newer image for gcr.io/kpt-fn/starlark:v0.3 +`, + expected: "", + }, + { + name: "should filter docker messages and shouldn't truncate trailing lines", + input: `Unable to find image 'gcr.io/kpt-fn/starlark:v0.3' locally +v0.3: Pulling from kpt-fn/starlark +4e9f2cdf4387: Already exists +aafbf7df3ddf: Pulling fs layer +aafbf7df3ddf: Verifying Checksum +aafbf7df3ddf: Download complete +aafbf7df3ddf: Pull complete +6b759ab96cb2: Waiting +Digest: sha256:c347e28606fa1a608e8e02e03541a5a46e4a0152005df4a11e44f6c4ab1edd9a +Status: Downloaded newer image for gcr.io/kpt-fn/starlark:v0.3 +line before last line +lastline + +`, + expected: `line before last line +lastline +`, + }, + { + name: "should filter interleaved docker messages", + input: `firstline +Unable to find image 'gcr.io/kpt-fn/starlark:v0.3' locally +v0.3: Pulling from kpt-fn/starlark +4e9f2cdf4387: Already exists +aafbf7df3ddf: Pulling fs layer +aafbf7df3ddf: Verifying Checksum +line in the middle +aafbf7df3ddf: Download complete +aafbf7df3ddf: Pull complete +6b759ab96cb2: Waiting +Digest: sha256:c347e28606fa1a608e8e02e03541a5a46e4a0152005df4a11e44f6c4ab1edd9a +Status: Downloaded newer image for gcr.io/kpt-fn/starlark:v0.3 +lastline +`, + expected: `firstline +line in the middle +lastline`, + }, + } + + for _, tc := range testcases { + tc := tc + t.Run(tc.name, func(t *testing.T) { + s := filterDockerCLIOutput(bytes.NewBufferString(tc.input)) + assert.Equal(t, tc.expected, s) + }) + } +} diff --git a/internal/fnruntime/runner.go b/internal/fnruntime/runner.go index bdbaa1a3ed..051a533032 100644 --- a/internal/fnruntime/runner.go +++ b/internal/fnruntime/runner.go @@ -31,6 +31,7 @@ import ( "github.com/GoogleContainerTools/kpt/internal/types" fnresult "github.com/GoogleContainerTools/kpt/pkg/api/fnresult/v1" kptfilev1 "github.com/GoogleContainerTools/kpt/pkg/api/kptfile/v1" + "github.com/google/shlex" "sigs.k8s.io/kustomize/kyaml/fn/framework" "sigs.k8s.io/kustomize/kyaml/fn/runtime/runtimeutil" "sigs.k8s.io/kustomize/kyaml/kio" @@ -38,33 +39,62 @@ import ( "sigs.k8s.io/kustomize/kyaml/yaml" ) -// NewContainerRunner returns a kio.Filter given a specification of a container function +// NewRunner returns a kio.Filter given a specification of a function // and it's config. -func NewContainerRunner( +func NewRunner( ctx context.Context, f *kptfilev1.Function, pkgPath types.UniquePath, fnResults *fnresult.ResultList, imagePullPolicy ImagePullPolicy, displayResourceCount bool) (kio.Filter, error) { + config, err := newFnConfig(f, pkgPath) if err != nil { return nil, err } + if f.Image != "" { + f.Image = AddDefaultImagePathPrefix(f.Image) + } fnResult := &fnresult.Result{ - Image: f.Image, + Image: f.Image, + ExecPath: f.Exec, // TODO(droot): This is required for making structured results subpackage aware. // Enable this once test harness supports filepath based assertions. // Pkg: string(pkgPath), } - cfn := &ContainerFn{ - Path: pkgPath, - Image: f.Image, - ImagePullPolicy: imagePullPolicy, - Ctx: ctx, - FnResult: fnResult, - } - fltr := &runtimeutil.FunctionFilter{ - Run: cfn.Run, - FunctionConfig: config, + + fltr := &runtimeutil.FunctionFilter{FunctionConfig: config} + switch { + case f.Image != "": + cfn := &ContainerFn{ + Path: pkgPath, + Image: f.Image, + ImagePullPolicy: imagePullPolicy, + Ctx: ctx, + FnResult: fnResult, + } + fltr.Run = cfn.Run + case f.Exec != "": + var execArgs []string + // assuming exec here + s, err := shlex.Split(f.Exec) + if err != nil { + return nil, fmt.Errorf("exec command %q must be valid: %w", f.Exec, err) + } + execPath := f.Exec + if len(s) > 0 { + execPath = s[0] + } + if len(s) > 1 { + execArgs = s[1:] + } + eFn := &ExecFn{ + Path: execPath, + Args: execArgs, + FnResult: fnResult, + } + fltr.Run = eFn.Run + default: + return nil, fmt.Errorf("must specify `exec` or `image` to execute a function") } return NewFunctionRunner(ctx, fltr, pkgPath, fnResult, fnResults, true, displayResourceCount) } @@ -135,6 +165,7 @@ func (fr *FunctionRunner) Filter(input []*yaml.RNode) (output []*yaml.RNode, err if !fr.disableCLIOutput { pr.Printf("[PASS] %q in %v\n", fr.name, time.Since(t0).Truncate(time.Millisecond*100)) printFnResult(fr.ctx, fr.fnResult, printer.NewOpt()) + printFnStderr(fr.ctx, fr.fnResult.Stderr) } return output, err } @@ -315,16 +346,22 @@ func printFnResult(ctx context.Context, fnResult *fnresult.Result, opt *printer. // on kpt CLI. func printFnExecErr(ctx context.Context, fnErr *ExecError) { pr := printer.FromContextOrDie(ctx) - if len(fnErr.Stderr) > 0 { + printFnStderr(ctx, fnErr.Stderr) + pr.Printf(" Exit code: %d\n\n", fnErr.ExitCode) +} + +// printFnStderr prints given stdErr in a user friendly format on kpt CLI. +func printFnStderr(ctx context.Context, stdErr string) { + pr := printer.FromContextOrDie(ctx) + if len(stdErr) > 0 { errLines := &multiLineFormatter{ Title: "Stderr", - Lines: strings.Split(fnErr.Stderr, "\n"), + Lines: strings.Split(stdErr, "\n"), UseQuote: true, TruncateOutput: printer.TruncateOutput, } pr.Printf("%s", errLines.String()) } - pr.Printf(" Exit code: %d\n\n", fnErr.ExitCode) } // path (location) of a KRM resources is tracked in a special key in diff --git a/internal/fnruntime/runner_test.go b/internal/fnruntime/runner_test.go index 8c3e74942e..97fd2b1fec 100644 --- a/internal/fnruntime/runner_test.go +++ b/internal/fnruntime/runner_test.go @@ -18,12 +18,14 @@ package fnruntime import ( "bytes" + "context" "io/ioutil" "os" "path" "strings" "testing" + "github.com/GoogleContainerTools/kpt/internal/printer" "github.com/GoogleContainerTools/kpt/internal/types" kptfilev1 "github.com/GoogleContainerTools/kpt/pkg/api/kptfile/v1" "github.com/stretchr/testify/assert" @@ -577,3 +579,70 @@ file: assert.Equal(t, tc.expected, string(out)) } } + +func TestPrintFnStderr(t *testing.T) { + tests := map[string]struct { + input string // input + truncateOutput bool // whether to truncate output + expected string // expected result + }{ + "no output": { + input: ``, + truncateOutput: true, + expected: ``, + }, + "truncated output": { + input: `0 +1 +2 +3 +4 +5`, + truncateOutput: true, + expected: ` Stderr: + "0" + "1" + "2" + "3" + ...(2 line(s) truncated, use '--truncate-output=false' to disable) +`, + }, + "non-truncated output": { + input: `0 +1 +2 +3 +4 +5`, + truncateOutput: false, + expected: ` Stderr: + "0" + "1" + "2" + "3" + "4" + "5" +`, + }, + } + cleanupFunc := func() func() { + origTruncateOutput := printer.TruncateOutput + return func() { + printer.TruncateOutput = origTruncateOutput + } + }() + defer cleanupFunc() + for testName, tc := range tests { + t.Run(testName, func(t *testing.T) { + printer.TruncateOutput = tc.truncateOutput + out := &bytes.Buffer{} + err := &bytes.Buffer{} + ctx := printer.WithContext(context.Background(), printer.New(out, err)) + + printFnStderr(ctx, tc.input) + + assert.Equal(t, tc.expected, err.String()) + assert.Equal(t, "", out.String()) + }) + } +} diff --git a/internal/util/cmdutil/cmdutil.go b/internal/util/cmdutil/cmdutil.go index 8fda513e9d..fa5ffd752e 100644 --- a/internal/util/cmdutil/cmdutil.go +++ b/internal/util/cmdutil/cmdutil.go @@ -29,6 +29,7 @@ import ( "github.com/GoogleContainerTools/kpt/internal/fnruntime" "github.com/GoogleContainerTools/kpt/internal/util/httputil" "github.com/spf13/cobra" + "golang.org/x/mod/semver" "sigs.k8s.io/kustomize/kyaml/kio" "sigs.k8s.io/kustomize/kyaml/kio/kioutil" ) @@ -39,7 +40,7 @@ const ( Stdout = "stdout" Unwrap = "unwrap" dockerVersionTimeout time.Duration = 5 * time.Second - FunctionsCatalogURL = "https://catalog.kpt.dev/catalog.json" + FunctionsCatalogURL = "https://catalog.kpt.dev/catalog-v2.json" ) // FixDocs replaces instances of old with new in the docs for c @@ -196,18 +197,25 @@ func FetchFunctionImages() []string { return listImages(content) } +// fnName -> v. -> catalogEntry +type catalogV2 map[string]map[string]struct { + LatestPatchVersion string + Examples interface{} +} + // listImages returns the list of latest images from the input catalog content func listImages(content string) []string { var result []string - jsonData := map[string]map[string]interface{}{} + var jsonData catalogV2 err := json.Unmarshal([]byte(content), &jsonData) if err != nil { return result } for fnName, fnInfo := range jsonData { var latestVersion string - for version := range fnInfo { - if latestVersion < version { + for _, catalogEntry := range fnInfo { + version := catalogEntry.LatestPatchVersion + if semver.Compare(version, latestVersion) == 1 { latestVersion = version } } diff --git a/internal/util/cmdutil/cmdutil_test.go b/internal/util/cmdutil/cmdutil_test.go index 8fe1a91dcd..c73792b56c 100644 --- a/internal/util/cmdutil/cmdutil_test.go +++ b/internal/util/cmdutil/cmdutil_test.go @@ -316,30 +316,39 @@ func TestListImages(t *testing.T) { result := listImages(`{ "apply-setters": { "v0.1": { - "apply-setters-simple": { - "LocalExamplePath": "/apply-setters/v0.1/apply-setters-simple", - "RemoteExamplePath": "https://github.com/GoogleContainerTools/kpt-functions-catalog/tree/apply-setters/v0.1/examples/apply-setters-simple", - "RemoteSourcePath": "https://github.com/GoogleContainerTools/kpt-functions-catalog/tree/apply-setters/v0.1/functions/go/apply-setters" + "LatestPatchVersion": "v0.1.1", + "Examples": { + "apply-setters-simple": { + "LocalExamplePath": "/apply-setters/v0.1/apply-setters-simple", + "RemoteExamplePath": "https://github.com/GoogleContainerTools/kpt-functions-catalog/tree/apply-setters/v0.1/examples/apply-setters-simple", + "RemoteSourcePath": "https://github.com/GoogleContainerTools/kpt-functions-catalog/tree/apply-setters/v0.1/functions/go/apply-setters" + } } } }, "gatekeeper": { "v0.1": { - "gatekeeper-warning-only": { - "LocalExamplePath": "/gatekeeper/v0.1/gatekeeper-warning-only", - "RemoteExamplePath": "https://github.com/GoogleContainerTools/kpt-functions-catalog/tree/gatekeeper/v0.1/examples/gatekeeper-warning-only", - "RemoteSourcePath": "https://github.com/GoogleContainerTools/kpt-functions-catalog/tree/gatekeeper/v0.1/functions/go/gatekeeper" + "LatestPatchVersion": "v0.1.3", + "Examples": { + "gatekeeper-warning-only": { + "LocalExamplePath": "/gatekeeper/v0.1/gatekeeper-warning-only", + "RemoteExamplePath": "https://github.com/GoogleContainerTools/kpt-functions-catalog/tree/gatekeeper/v0.1/examples/gatekeeper-warning-only", + "RemoteSourcePath": "https://github.com/GoogleContainerTools/kpt-functions-catalog/tree/gatekeeper/v0.1/functions/go/gatekeeper" + } } }, "v0.2": { - "gatekeeper-warning-only": { - "LocalExamplePath": "/gatekeeper/v0.2/gatekeeper-warning-only", - "RemoteExamplePath": "https://github.com/GoogleContainerTools/kpt-functions-catalog/tree/gatekeeper/v0.2/examples/gatekeeper-warning-only", - "RemoteSourcePath": "https://github.com/GoogleContainerTools/kpt-functions-catalog/tree/gatekeeper/v0.2/functions/go/gatekeeper" + "LatestPatchVersion": "v0.2.1", + "Examples": { + "gatekeeper-warning-only": { + "LocalExamplePath": "/gatekeeper/v0.2/gatekeeper-warning-only", + "RemoteExamplePath": "https://github.com/GoogleContainerTools/kpt-functions-catalog/tree/gatekeeper/v0.2/examples/gatekeeper-warning-only", + "RemoteSourcePath": "https://github.com/GoogleContainerTools/kpt-functions-catalog/tree/gatekeeper/v0.2/functions/go/gatekeeper" + } } } } }`) sort.Strings(result) - assert.Equal(t, []string{"apply-setters:v0.1", "gatekeeper:v0.2"}, result) + assert.Equal(t, []string{"apply-setters:v0.1.1", "gatekeeper:v0.2.1"}, result) } diff --git a/main.go b/main.go index 4e1053af11..311b9ad510 100644 --- a/main.go +++ b/main.go @@ -60,10 +60,10 @@ func runMain() int { // Note(droot): There are too many flags exposed that makes the command // usage verbose but couldn't find a way to make it less verbose. klog.InitFlags(&logFlags) - cmd.Flags().AddGoFlagSet(&logFlags) // By default klog v1 logs to stderr, switch that off - _ = cmd.Flags().Set("logtostderr", "false") - _ = cmd.Flags().Set("alsologtostderr", "false") + _ = logFlags.Set("logtostderr", "false") + _ = logFlags.Set("alsologtostderr", "true") + cmd.Flags().AddGoFlagSet(&logFlags) err = cmd.Execute() if err != nil { diff --git a/pkg/api/kptfile/v1/types.go b/pkg/api/kptfile/v1/types.go index 8f668ab788..ecabd4d9be 100644 --- a/pkg/api/kptfile/v1/types.go +++ b/pkg/api/kptfile/v1/types.go @@ -315,6 +315,13 @@ type Function struct { // image: set-labels Image string `yaml:"image,omitempty" json:"image,omitempty"` + // Exec specifies the function binary executable. + // The executable can be fully qualified or it must exists in the $PATH e.g: + // + // exec: set-namespace + // exec: /usr/local/bin/my-custom-fn + Exec string `yaml:"exec,omitempty" json:"exec,omitempty"` + // `ConfigPath` specifies a slash-delimited relative path to a file in the current directory // containing a KRM resource used as the function config. This resource is // excluded when resolving 'sources', and as a result cannot be operated on diff --git a/pkg/api/kptfile/v1/validation.go b/pkg/api/kptfile/v1/validation.go index 41eb9e8488..6489afd7f3 100644 --- a/pkg/api/kptfile/v1/validation.go +++ b/pkg/api/kptfile/v1/validation.go @@ -67,14 +67,29 @@ func (p *Pipeline) validate(pkgPath types.UniquePath) error { } func (f *Function) validate(fnType string, idx int, pkgPath types.UniquePath) error { - err := ValidateFunctionImageURL(f.Image) - if err != nil { + if f.Image == "" && f.Exec == "" { + return &ValidateError{ + Field: fmt.Sprintf("pipeline.%s[%d]", fnType, idx), + Reason: "must specify a functon (`image` or `exec`) to execute", + } + } + if f.Image != "" && f.Exec != "" { return &ValidateError{ - Field: fmt.Sprintf("pipeline.%s[%d].image", fnType, idx), - Value: f.Image, - Reason: err.Error(), + Field: fmt.Sprintf("pipeline.%s[%d]", fnType, idx), + Reason: "must not specify both `image` and `exec` at the same time", + } + } + if f.Image != "" { + err := ValidateFunctionImageURL(f.Image) + if err != nil { + return &ValidateError{ + Field: fmt.Sprintf("pipeline.%s[%d].image", fnType, idx), + Value: f.Image, + Reason: err.Error(), + } } } + // TODO(droot): validate the exec if len(f.ConfigMap) != 0 && f.ConfigPath != "" { return &ValidateError{ diff --git a/pkg/live/apply-crd-task.go b/pkg/live/apply-crd-task.go index 2a1399ea74..2fe40fe258 100644 --- a/pkg/live/apply-crd-task.go +++ b/pkg/live/apply-crd-task.go @@ -41,7 +41,7 @@ func (a *ApplyCRDTask) Action() event.ResourceAction { } func (a *ApplyCRDTask) Identifiers() object.ObjMetadataSet { - return object.UnstructuredsToObjMetasOrDie([]*unstructured.Unstructured{a.crd}) + return object.UnstructuredSetToObjMetadataSet([]*unstructured.Unstructured{a.crd}) } // NewApplyCRDTask returns a pointer to an ApplyCRDTask struct, @@ -89,4 +89,6 @@ func (a *ApplyCRDTask) Start(taskContext *taskrunner.TaskContext) { }() } -func (a *ApplyCRDTask) ClearTimeout() {} +func (a *ApplyCRDTask) Cancel(_ *taskrunner.TaskContext) {} + +func (a *ApplyCRDTask) StatusUpdate(_ *taskrunner.TaskContext, _ object.ObjMetadata) {} diff --git a/pkg/live/inventoryrg.go b/pkg/live/inventoryrg.go index ae9a708845..7d10f269b0 100644 --- a/pkg/live/inventoryrg.go +++ b/pkg/live/inventoryrg.go @@ -31,8 +31,9 @@ import ( "sigs.k8s.io/cli-utils/pkg/apply/taskrunner" "sigs.k8s.io/cli-utils/pkg/common" "sigs.k8s.io/cli-utils/pkg/inventory" + "sigs.k8s.io/cli-utils/pkg/kstatus/polling" + "sigs.k8s.io/cli-utils/pkg/kstatus/polling/engine" "sigs.k8s.io/cli-utils/pkg/object" - utilfactory "sigs.k8s.io/cli-utils/pkg/util/factory" "sigs.k8s.io/kustomize/kyaml/yaml" ) @@ -149,9 +150,10 @@ func (icm *InventoryResourceGroup) Load() (object.ObjMetadataSet, error) { Kind: strings.TrimSpace(kind), } klog.V(4).Infof("creating obj metadata: %s/%s/%s", namespace, name, groupKind) - objMeta, err := object.CreateObjMetadata(namespace, name, groupKind) - if err != nil { - return []object.ObjMetadata{}, err + objMeta := object.ObjMetadata{ + GroupKind: groupKind, + Name: name, + Namespace: namespace, } objs = append(objs, objMeta) } @@ -266,7 +268,7 @@ func InstallResourceGroupCRD(factory cmdutil.Factory) error { } // Create the task to apply the ResourceGroup CRD. applyRGTask := NewApplyCRDTask(factory, crd) - objs := object.UnstructuredsToObjMetasOrDie([]*unstructured.Unstructured{crd}) + objs := object.UnstructuredSetToObjMetadataSet([]*unstructured.Unstructured{crd}) // Create the tasks to apply the ResourceGroup CRD. tasks := []taskrunner.Task{ applyRGTask, @@ -278,7 +280,7 @@ func InstallResourceGroupCRD(factory cmdutil.Factory) error { for _, t := range tasks { taskQueue <- t } - statusPoller, err := utilfactory.NewStatusPoller(factory) + statusPoller, err := polling.NewStatusPollerFromFactory(factory, []engine.StatusReader{}) if err != nil { handleError(eventChannel, err) return diff --git a/pkg/live/load_test.go b/pkg/live/load_test.go index 919eaca0d5..4562d10a08 100644 --- a/pkg/live/load_test.go +++ b/pkg/live/load_test.go @@ -34,7 +34,7 @@ func TestLoad_LocalDisk(t *testing.T) { testCases := map[string]struct { pkg *pkgbuilder.RootPkg namespace string - expectedObjs []object.ObjMetadata + expectedObjs object.ObjMetadataSet expectedInv kptfile.Inventory expectedErrMsg string }{ @@ -184,7 +184,7 @@ func TestLoad_LocalDisk(t *testing.T) { } assert.NoError(t, err) - objMetas := object.UnstructuredsToObjMetasOrDie(objs) + objMetas := object.UnstructuredSetToObjMetadataSet(objs) sort.Slice(objMetas, func(i, j int) bool { return objMetas[i].String() < objMetas[j].String() }) @@ -199,7 +199,7 @@ func TestLoad_StdIn(t *testing.T) { testCases := map[string]struct { pkg *pkgbuilder.RootPkg namespace string - expectedObjs []object.ObjMetadata + expectedObjs object.ObjMetadataSet expectedInv kptfile.Inventory expectedErrMsg string }{ @@ -343,7 +343,7 @@ func TestLoad_StdIn(t *testing.T) { } assert.NoError(t, err) - objMetas := object.UnstructuredsToObjMetasOrDie(objs) + objMetas := object.UnstructuredSetToObjMetadataSet(objs) sort.Slice(objMetas, func(i, j int) bool { return objMetas[i].String() < objMetas[j].String() }) diff --git a/pkg/live/rgpath_test.go b/pkg/live/rgpath_test.go index 2e529e00cc..4130671a9f 100644 --- a/pkg/live/rgpath_test.go +++ b/pkg/live/rgpath_test.go @@ -23,7 +23,7 @@ func TestPathManifestReader_Read(t *testing.T) { testCases := map[string]struct { manifests map[string]string namespace string - expectedObjs []object.ObjMetadata + expectedObjs object.ObjMetadataSet expectedErrMsg string }{ "Empty package is ok": { @@ -151,7 +151,7 @@ func TestPathManifestReader_Read(t *testing.T) { "cr.yaml": cr, }, namespace: "test-namespace", - expectedErrMsg: "unknown resource types: Custom.custom.io", + expectedErrMsg: "unknown resource types: custom.io/v1/Custom", }, "local-config is filtered out": { manifests: map[string]string{ @@ -211,7 +211,7 @@ func TestPathManifestReader_Read(t *testing.T) { } assert.NoError(t, err) - readObjMetas := object.UnstructuredsToObjMetasOrDie(readObjs) + readObjMetas := object.UnstructuredSetToObjMetadataSet(readObjs) sort.Slice(readObjMetas, func(i, j int) bool { return readObjMetas[i].String() < readObjMetas[j].String() diff --git a/pkg/live/rgstream_test.go b/pkg/live/rgstream_test.go index 15ddf9ad6c..18d59923fc 100644 --- a/pkg/live/rgstream_test.go +++ b/pkg/live/rgstream_test.go @@ -22,7 +22,7 @@ func TestResourceStreamManifestReader_Read(t *testing.T) { testCases := map[string]struct { manifests map[string]string namespace string - expectedObjs []object.ObjMetadata + expectedObjs object.ObjMetadataSet expectedErrMsg string }{ "Kptfile is excluded": { @@ -99,7 +99,7 @@ func TestResourceStreamManifestReader_Read(t *testing.T) { "cr.yaml": cr, }, namespace: "test-namespace", - expectedErrMsg: "unknown resource types: Custom.custom.io", + expectedErrMsg: "unknown resource types: custom.io/v1/Custom", }, } @@ -137,7 +137,7 @@ func TestResourceStreamManifestReader_Read(t *testing.T) { } assert.NoError(t, err) - readObjMetas := object.UnstructuredsToObjMetasOrDie(readObjs) + readObjMetas := object.UnstructuredSetToObjMetadataSet(readObjs) sort.Slice(readObjMetas, func(i, j int) bool { return readObjMetas[i].String() < readObjMetas[j].String() diff --git a/pkg/test/live/runner.go b/pkg/test/live/runner.go index 7ee8bd11b7..304644ab01 100644 --- a/pkg/test/live/runner.go +++ b/pkg/test/live/runner.go @@ -15,6 +15,7 @@ package live import ( + "bufio" "bytes" "errors" "os" @@ -26,6 +27,7 @@ import ( "testing" "github.com/stretchr/testify/assert" + "sigs.k8s.io/cli-utils/pkg/kstatus/status" "sigs.k8s.io/kustomize/kyaml/yaml" ) @@ -96,11 +98,15 @@ func (r *Runner) VerifyExitCode(t *testing.T, err error) { } func (r *Runner) VerifyStdout(t *testing.T, stdout string) { - assert.Equal(t, strings.TrimSpace(r.Config.StdOut), strings.TrimSpace(substituteTimestamps(stdout))) + assert.Equal(t, strings.TrimSpace(r.Config.StdOut), prepOutput(t, stdout)) } func (r *Runner) VerifyStderr(t *testing.T, stderr string) { - assert.Equal(t, strings.TrimSpace(r.Config.StdErr), strings.TrimSpace(substituteTimestamps(stderr))) + assert.Equal(t, strings.TrimSpace(r.Config.StdErr), prepOutput(t, stderr)) +} + +func prepOutput(t *testing.T, s string) string { + return strings.TrimSpace(substituteTimestamps(removeStatusEvents(t, s))) } func (r *Runner) VerifyInventory(t *testing.T, name, namespace string) { @@ -173,3 +179,32 @@ var timestampRegexp = regexp.MustCompile(`\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z`) func substituteTimestamps(text string) string { return timestampRegexp.ReplaceAllString(text, "") } + +var statuses = []status.Status{ + status.InProgressStatus, + status.CurrentStatus, + status.FailedStatus, + status.TerminatingStatus, + status.UnknownStatus, + status.NotFoundStatus, +} + +func removeStatusEvents(t *testing.T, text string) string { + scanner := bufio.NewScanner(strings.NewReader(text)) + var lines []string + +scan: + for scanner.Scan() { + line := scanner.Text() + for _, s := range statuses { + if strings.Contains(line, s.String()) { + continue scan + } + } + lines = append(lines, line) + } + if err := scanner.Err(); err != nil { + t.Fatalf("error scanning output: %v", err) + } + return strings.Join(lines, "\n") +} diff --git a/pkg/test/runner/config.go b/pkg/test/runner/config.go index 44fc75d985..57b3c3035c 100644 --- a/pkg/test/runner/config.go +++ b/pkg/test/runner/config.go @@ -71,6 +71,9 @@ type TestCaseConfig struct { // be the same as the CLI flag. ImagePullPolicy string `json:"imagePullPolicy,omitempty" yaml:"imagePullPolicy,omitempty"` + // AllowExec determines if `fn render` needs to be invoked with `--allow-exec` flag + AllowExec bool `json:"allowExec,omitempty" yaml:"allowExec,omitempty"` + // Skip means should this test case be skipped. Default: false Skip bool `json:"skip,omitempty" yaml:"skip,omitempty"` diff --git a/pkg/test/runner/runner.go b/pkg/test/runner/runner.go index 4309cab839..1754a77672 100644 --- a/pkg/test/runner/runner.go +++ b/pkg/test/runner/runner.go @@ -349,6 +349,10 @@ func (r *Runner) runFnRender() error { kptArgs = append(kptArgs, "--image-pull-policy", r.testCase.Config.ImagePullPolicy) } + if r.testCase.Config.AllowExec { + kptArgs = append(kptArgs, "--allow-exec") + } + if r.testCase.Config.DisableOutputTruncate { kptArgs = append(kptArgs, "--truncate-output=false") } diff --git a/release/images/Dockerfile b/release/images/Dockerfile index 6ebcf4a7cf..cc34596e1a 100644 --- a/release/images/Dockerfile +++ b/release/images/Dockerfile @@ -12,9 +12,9 @@ # See the License for the specific language governing permissions and # limitations under the License. -FROM alpine:3.11 +FROM alpine:3.13 RUN apk update && apk upgrade && \ - apk add --no-cache git less man diffutils bash openssh docker-cli && \ + apk add --no-cache git less mandoc diffutils bash openssh docker-cli && \ rm -rf /var/lib/apt/lists/* && \ rm /var/cache/apk/* # This is set up for the Dockerfile to be used by goreleaser: https://goreleaser.com/customization/docker/ diff --git a/release/images/Dockerfile-gcloud b/release/images/Dockerfile-gcloud index 4cbb114703..5f3c16c76d 100644 --- a/release/images/Dockerfile-gcloud +++ b/release/images/Dockerfile-gcloud @@ -12,7 +12,7 @@ # See the License for the specific language governing permissions and # limitations under the License. -FROM gcr.io/google.com/cloudsdktool/cloud-sdk:355.0.0-alpine +FROM gcr.io/google.com/cloudsdktool/cloud-sdk:367.0.0-alpine RUN apk update && apk upgrade && \ apk add --no-cache git less diffutils bash openssh docker-cli && \ rm -rf /var/lib/apt/lists/* && \ diff --git a/run/run.go b/run/run.go index 176a898220..36ab589647 100644 --- a/run/run.go +++ b/run/run.go @@ -24,7 +24,6 @@ import ( "strings" kptcommands "github.com/GoogleContainerTools/kpt/commands" - "github.com/GoogleContainerTools/kpt/internal/cmdcomplete" "github.com/GoogleContainerTools/kpt/internal/docs/generated/overview" "github.com/GoogleContainerTools/kpt/internal/printer" "github.com/GoogleContainerTools/kpt/internal/util/cmdutil" @@ -36,7 +35,6 @@ var pgr []string func GetMain(ctx context.Context) *cobra.Command { os.Setenv(commandutil.EnableAlphaCommmandsEnvName, "true") - installComp := false cmd := &cobra.Command{ Use: "kpt", Short: overview.CliShort, @@ -46,18 +44,6 @@ func GetMain(ctx context.Context) *cobra.Command { // adjust the error message coming from libraries SilenceErrors: true, RunE: func(cmd *cobra.Command, args []string) error { - if installComp { - os.Setenv("COMP_INSTALL", "1") - os.Setenv("COMP_YES", "1") - fmt.Fprint(cmd.OutOrStdout(), "Installing shell completion...\n") - fmt.Fprint(cmd.OutOrStdout(), - "This will add 'complete -C /Users/$USER/go/bin/kpt kpt' to "+ - ".bashrc, .bash_profile, etc\n") - fmt.Fprint(cmd.OutOrStdout(), "Run `COMP_INSTALL=0 kpt` to uninstall.\n") - } - // Complete exits if it is called in completion mode, otherwise it is a no-op - cmdcomplete.Complete(cmd, false, nil).Complete("kpt") - h, err := cmd.Flags().GetBool("help") if err != nil { return err @@ -77,20 +63,6 @@ func GetMain(ctx context.Context) *cobra.Command { // create context with associated printer ctx = printer.WithContext(ctx, pr) - cmd.Flags().BoolVar(&installComp, "install-completion", false, - "Install shell completion") - // this command will be invoked by the shell-completion code - cmd.AddCommand(&cobra.Command{ - Use: "kpt", - Hidden: true, - SilenceErrors: true, - SilenceUsage: true, - Run: func(cmd *cobra.Command, args []string) { - // Complete exits if it is called in completion mode, otherwise it is a no-op - cmdcomplete.Complete(cmd.Parent(), false, nil).Complete("kpt") - }, - }) - // find the pager if one exists func() { if val, found := os.LookupEnv("KPT_NO_PAGER_HELP"); !found || val != "1" { diff --git a/scripts/create-licenses.sh b/scripts/create-licenses.sh index 8bffc19d60..3ec8304577 100755 --- a/scripts/create-licenses.sh +++ b/scripts/create-licenses.sh @@ -238,7 +238,7 @@ done >> ${TMP_LICENSE_FILE} cat ${TMP_LICENSE_FILE} > ${VENDOR_LICENSE_FILE} # Create a package of Mozilla repository source code (only go code). -zip -qr $ZIP_FILENAME $mozilla_repos -i '*.go' +[ -n "$mozilla_repos" ] && zip -qr $ZIP_FILENAME $mozilla_repos -i '*.go' # Cleanup vendor directory rm -rf vendor diff --git a/site/book/04-using-functions/01-declarative-function-execution.md b/site/book/04-using-functions/01-declarative-function-execution.md index c2f01cbdde..6bf9e39e09 100644 --- a/site/book/04-using-functions/01-declarative-function-execution.md +++ b/site/book/04-using-functions/01-declarative-function-execution.md @@ -110,13 +110,48 @@ The end result is that: If any of the functions in the pipeline fails for whatever reason, then the entire pipeline is aborted and the local filesystem is left intact. -## Specifying `image` +## Specifying `function` + +### `image` The `image` field specifies the container image for the function. You can specify an image from any container registry. If the registry is omitted, the default container registry for functions catalog (`gcr.io/kpt-fn`) is prepended automatically. For example, `set-labels:v0.1` is automatically expanded to `gcr.io/kpt-fn/set-labels:v0.1`. +### `exec` + +The `exec` field specifies the executable command for the function. You can specify +an executable with arguments. + +Example below uses `sed` executable to replace all occurances of `foo` with `bar` +in the package resources. + +```yaml +# PKG_DIR/Kptfile (Excerpt) +apiVersion: kpt.dev/v1 +kind: Kptfile +metadata: + name: app +pipeline: + mutators: + - exec: "sed -e 's/foo/bar/'" +``` + +Note that you must render the package by allowing executables by specifying `--allow-exec` +command line flag as shown below. + +```shell +$ kpt fn render [PKG_DIR] --allow-exec +``` + +Using `exec` is not recommended for two reasons: + +- It makes the package non-portable since rendering the package requires the + executables to be present on the system. +- Executing binaries is not very secure since they can perform privileged operations + on the system. + ## Specifying `functionConfig` In [Chapter 2], we saw this conceptual representation of a function invocation: diff --git a/site/book/06-deploying-packages/03-handling-dependencies.md b/site/book/06-deploying-packages/03-handling-dependencies.md index 0130a3363a..609aa59d9c 100644 --- a/site/book/06-deploying-packages/03-handling-dependencies.md +++ b/site/book/06-deploying-packages/03-handling-dependencies.md @@ -79,3 +79,8 @@ statefulset.apps/wordpress-mysql deleted service/wordpress-mysql deleted 2 resource(s) deleted, 0 skipped ``` + +See [depends-on] for more information. + +[depends-on]: + /reference/annotations/depends-on/ diff --git a/site/installation/README.md b/site/installation/README.md index 4a783d265a..3d979cc52a 100644 --- a/site/installation/README.md +++ b/site/installation/README.md @@ -27,6 +27,47 @@ Verify the version: $ kpt version ``` +## (Optional) enable shell auto-completion + +kpt provides auto-completion support for several of the common shells. +To see the options for enabling shell auto-completion: +```shell +kpt completion -h +``` + +### Prerequisites +`kpt` depends on `bash-completion` in order to support auto-completion for the +bash shell. If you are using bash as your shell, you will need to install +`bash-completion` in order to use kpt's auto-completion feature. +`bash-completion` is provided by many package managers +(see [here][bash-completion]). + +### Enable kpt auto-completion +The kpt completion script for a shell can be generated with the commands +`kpt completion bash`, `kpt completion zsh`, etc. Sourcing the completion script +in your shell enables auto-completion. + +#### Enable auto-completion for your current shell +bash: +```shell +source <(kpt completion bash) +``` +zsh: +```shell +source <(kpt completion zsh) +``` +etc. +#### Enable kpt completion for all your shell sessions +bash: +```shell +echo 'source <(kpt completion bash)' >> ~/.bashrc +``` +zsh: +```shell +echo 'source <(kpt completion zsh)' >> ~/.zshrc +``` +etc. +