Separate documents issue.

[mikaelcabot]

kubectl apply accepts yaml documents with multiple docs without --- in head.
But without this seperator in head of each template the generated template from kontemplate becomes invalid.

[tazjin]

This can become an issue when users split resources across multiple files and don't start the resources with ---.

Inside of a single file containing multiple resources I expect people to be aware of this, but a good thing to do here would be to issue a warning to users.

What I'm thinking is logic such as the following (pseudocode):

if (moreThanOneTemplateInResourceSet) {
  foreach (file in yamlFiles) {
    if (file.firstLineThatIsntAComment != "---\n" ) {
      warn("The file ${file.name} does not start with a YAML resource separator ('---')!")
    }
  }
}

[tazjin]

Note that I'd prefer not to add any logic to actually parse & validate YAML itself in Kontemplate.

[mikaelcabot]

Agree with not adding logic for validating the YAML.
It's a matter of how far you want to "pollute" your code I guess, but a warning like that could sure help, but then again you would have to check if that document contained more objects ---. - And if not then it would be ok if the first line didn't start with ---. Hmmmm...

[tazjin]

but then again you would have to check if that document contained more objects

No, inside of a file it's up to the user to write valid YAML ;-)

[mikaelcabot]

Ok, maybe it's just me then :) - I didn't know you had to start every YAML file with --- for it to be valid ...

[tazjin]

You don't, but it's only Kontemplate's responsibility to warn the user if it's at the start of the file (because it's kontemplate that concatenates the files).

[tazjin]

Other fix idea: What happens if kubectl is sent something like this

---
# no actual resource here
---

?

If that is valid then I can apply an actual "fix" of just prefixing each YAML file with --- when passing through to kubectl.

[mikaelcabot]

How about just checking if the document contains multiple objects seperated by ---. And if so warn if the document does not start with --- ?

[mikaelcabot]

I'm really not so sure anymore - as I guess this would be a validator's job ...

[tazjin]

How about just checking if the document contains multiple objects seperated by ---.

This can still cause the same issue if people include multiple resource sets in one run that have one YAML file with one resource each (without ---).

In fact that makes this a bug.

[mikaelcabot]

heh, good that got cleared out then 👍

[tazjin]

Yep, thanks for noticing!

[tazjin]

Multiple object separators seem to work without any issues:

vincent@urdhva ~ % cat | kubectl create -f -
---
---
apiVersion: v1
kind: Namespace
metadata:
  name: yaml-test
namespace "yaml-test" created

[mikaelcabot]

Then that seems to be the easiest fix

[tazjin]

I've decided agains the "easy" fix.

@heavenlyhash suggested that kontemplate should potentially execute kubectl multiple times (once per resource set), this way the resource sets internally have to be consistent but including multiple resource sets won't conflict if the user leaves out the ---.

That's enough to fix the specific bug I have and in addition error messages could potentially be a bit nicer!

[tazjin]

The new plan is as follows (and I hope to implement it in the next days because vacation is freeing up my time atm!):

1. Refactor templater to introduce a new RenderedResourceSet type, this will contain the fully-rendered templates but also retain information about the enclosing resource set, e.g.:

type RenderedResource struct {
  Filename string
  Output string // rendered template
}

type RenderedResourceSet struct {
  Name string // Resource set name/path (for output)
  Resources []RenderedResource
}

The kubectl-calling code will then be refactored to execute kubectl once per resource set, while providing some additional logging to stderr that can be useful when debugging issues.

As a sanity check particular to the problem described in this issue I may get around to adding logic that checks if multiple YAML-files are in a single resource set and, if yes, whether they contain --- somewhere. If not a warning will be issued, but kontemplate keeps doing its thing anyways.