Internal keys / non-prompted values

[verdaatt]

Is your feature request related to a problem? Please describe.

Trying to build a template for Terraform structure, which means a lot of the file tree structure depends on one or more of the inputs. Using a "choice" type here makes a lot more sense from a usability standpoint than using multiple competing boolean keys. For example:

Example copier.yaml

flavor:
  type: str
  choices:
    - Docker
    - Instances
    - Kubernetes
    - None

But that means having to do string comparison if-statements in directory names to create conditional directories. And everywhere else when this value is evaluated. If multiple answers are correct for some conditional directories then that results in even longer if-and statements:

Example filetree

.
├── {% if flavor == 'docker' %}ecs{% endif %}
├── {% if flavor == 'kubernetes' %}eks{% endif %}
├── {% if flavor == 'instance' %}bastion{% endif %}
├── {% if flavor == 'docker' or flavor == 'kubernetes' %}portainer{% endif %}
└── {% if flavor != 'none' %}vpc{% endif %}

This can make the directory names quite long and makes everything poorly readable. It also results in a lot of Jinja repetition across the template.

Describe the solution you'd like

I was hoping to create keys derived from the choice key behind the scenes, e.g. without prompting the user for input. In my scenario these would be booleans, but it could also be useful for concatenating strings or other post-input processing purposes that deduplicates a lot of code from the template files.

Example of how I'd love for this to work for my use case:

Future copier.yaml

flavor:
  type: str
  choices:
    - Docker
    - Instances
    - Kubernetes
    - None

# Internal variables
isContainer:
  type: bool
  value: "{% if flavor == 'docker' or flavor == 'kubernetes' %}true{% else %}false{% endif %}"

isDocker:
  type: bool
  value: "{% if flavor == 'docker' %}true{% else %}false{% endif %}"

isInstance:
  type: bool
  value: "{% if flavor == 'instance' %}true{% else %}false{% endif %}"

isKubernetes:
  type: bool
  value: "{% if flavor == 'kubernetes' %}true{% else %}false{% endif %}"

isLite:
  type: bool
  value: "{% if flavor == 'none' %}true{% else %}false{% endif %}"

Future filetree

.
├── {% if isDocker %}ecs{% endif %}
├── {% if isKubernetes %}eks{% endif %}
├── {% if isInstance %}bastion{% endif %}
├── {% if isContainer %}portainer{% endif %}
└── {% if isLite %}vpc{% endif %}

For this to work the value: key would have to be added and if this key is set then prompting the user for input would always be skipped. Alternatively prompt: false could be used in combination with setting the value using default: similar to what was suggested in #229 before.

If preferred these "internal variables" could also be moved one layer down under an identifier:

flavor:
  type: str
  choices:
    - Docker
    - Instances
    - Kubernetes
    - None

_derived_keys:
  isContainer:
    type: bool
    value: "{% if flavor == 'docker' or flavor == 'kubernetes' %}true{% else %}false{% endif %}"
  isDocker:
    type: bool
    value: "{% if flavor == 'docker' %}true{% else %}false{% endif %}"
  isInstance:
    type: bool
    value: "{% if flavor == 'instance' %}true{% else %}false{% endif %}"
  isKubernetes:
    type: bool
    value: "{% if flavor == 'kubernetes' %}true{% else %}false{% endif %}"
  isLite:
    type: bool
    value: "{% if flavor == 'none' %}true{% else %}false{% endif %}"

Describe alternatives you've considered

I've tried creating the variable like this, but setting when: false doesn't set the variable at all, resulting in the boolean always resolving to true:

isLite:
  type: bool
  when: false
  default: "{% if flavor == 'none' %}true{% else %}false{% endif %}"

Additional context

[yajo]

It seems to me like you need to template the subdirectory:

_subdirectory: "{{flavor}}"
flavor:
  type: str
  choices:
    - Docker
    - Instances
    - Kubernetes
    - None

Then add one folder per option. Copier will use it as template source.

Would that help?

[pawamoy]

That would serve only one particular use-case (a single top-level directory to render). Many other nested files/folders could need the same feature, without an option like _subdirectory to work around it.

[yajo]

What about adding a task that removes all dirs except the one you'll use?:

flavor:
  type: str
  choices:
    - Docker
    - Instances
    - Kubernetes
    - None
_tasks:
  - rm -rf {{ "Docker Instances Kubernetes None"|replace(flavor, "") }}

[pawamoy]

Clever use of replace 😄

In this particular case the directory names are not the same as the flavor value, so it'd need something more complex, probably a shell script (cross-platform compatibility issues) or a Python script: - {{ _copier_python }} posthook.py" (#612 ❤️).

So yeah, tasks can always be used when the other feature are not enough. But then @verdaatt would still have to repeat {% if flavor == 'docker' or flavor == 'kubernetes' %} over and over again in his templated files. And note that this is a simple condition. It could be way more complex.

Generally, this issue and the older ones mentioned simply show the usefulness of being able to augment the context based on the answers. I know you refused in the past to add such feature, because as you demonstrated, one can always use tasks or macros, etc., so I won't push for it myself, especially since I have my ContextHook extension to do the job 🙂 Seeing that this feature requests pops from time to time, maybe we should dedicate a paragraph in the docs to it, with workarounds, and possibly an example using my ContextHook extension.

[yajo]

The idea of adding your suggestion to FAQ is good.

[verdaatt]

Hi @yajo and @pawamoy. Thanks for your alternative suggestions.

I believe that most of these suggestions won't be a very convenient solution for us though. We need to template folders at several layers of the directory tree, which makes the rm post-task not very practical, even though I agree it was a clever use of replace. It also kind of pollutes the output of your update runs, right?

This also makes templating the subdirectories not really feasible. We'd get lots and lots of repetition to cover all scenarios. All this duplication isn't exactly DRY and doesn't improve code maintainability.

In addition we've also turned a lot of Terraform files into Jinja templates because there are lots of variable output dependencies that also depend on our flavor value. So that would still have these long if-statements.

Looks like the ContextHook extension would do the job I requested. I will look into using this. Totally support the suggestion to add this to the FAQ because as a new user I was unaware of this.

I still believe implementing this feature request would be great for usability. Using the hooks to augment variables feels like a complicated way of doing something that should be extremely easy and part of core functionality. Things that are set in hooks are less transparent to new developers working on the template than stuff that's set in copier.yaml itself. But that's just my two cents and I respect the decision of the people that put in the work to write the code ;-)

[yajo]

Thanks everybody for the interesting and relaxed talk about this subject. 😊

The main problem with this feature is that, although it's true that for file names is a bit cumbersome, it is also true that we'd mostly be adding syntactic sugar. However, inside the files themselves, we already have Jinja doing a great job with macros, imports and includes, so there's no need there.

There could be some key like this:

# copier.yml
_header: |
  {% set isDocker = flavor == 'docker' %}

... and we could prepend that template code to all other templates (filenames or not).

However, that would make file templates less explicit than what they are today, and I kinda prefer explicit over implicit (python zen BTW).

I'd prefer those kind of things to be handled by an extension if really needed.

Another option would be to ask via booleans, but use when to avoid asking if user has chosen already:

# copier.yaml
isDocker:
  type: bool

isKubernetes:
  type: bool
  when: "{{ not isDocker }}"

isInstance:
  type: bool
  when: "{{ not isDocker and not isKubernetes }}"

If you're worried about filename readability, you might find these names more readable: {{ flavor if flavor == 'docker' else '' }}.

Finally, if you really need to make the user choose among templates... then have you considered just doing several templates? So, the user does:

copier copy https://github.com/example/docker-template.git ./docker-dir
copier copy https://github.com/example/k8s-template.git ./kubernetes-dir
...

You can share code between templates using git submodules, and you can include some standard questionary in all of them using sections in copier.yml and including the external section from the submodule.

You can even apply several templates to the same repository:

copier -a .docker.copier-answers.yml https://github.com/example/docker-template.git .
copier -a .k8s.copier-answers.yml https://github.com/example/k8s-template.git .

(Note that the default value for a template's answer file can exist in the template, so you can save the -a part).

Well, I'm just giving you some options. The point is that what you want to achieve is perfectly doable, just you have to deal with that syntax in the file names, but there's really a lot of potential!

I'd really like to pave the way out for newcomers without having to support this extra case, so I'd be glad if you could tell me which of the ideas have been more interesting to you, so we can improve the docs. Actually, if you could open the PR to the docs, that'd be awesome!

[verdaatt]

Quick update: I tried copier-templates-extensions and used the ContextHook extension to implement this and it works perfectly. Adopted slugify too while I was at it which makes life easier as well!

My context.py for reference for anyone that might stumble upon this ticket while searching for something similar:

# https://github.com/pawamoy/copier-templates-extensions#context-hook-extension
from copier_templates_extensions import ContextHook


class ContextUpdater(ContextHook):
    update = False

    def hook(self, context):
        # Multiple-choice answer conversion to boolean variables
        context["isDocker"]    = context["flavor"] == 'docker'
        context["isK8s"]       = context["flavor"] == 'kubernetes'
        context["isInstances"] = context["flavor"] == 'instances'
        context["isLite"]      = context["flavor"] == 'none'

        context["isNotDocker"]    = context["flavor"] != 'docker'
        context["isNotK8s"]       = context["flavor"] != 'kubernetes'
        context["isNotInstances"] = context["flavor"] != 'instances'
        context["isNotLite"]      = context["flavor"] != 'none'

        # Derived variables
        context["hasContainers"] = (context["flavor"] == 'docker' or context["flavor"] == 'kubernetes')

        # Remove variables that should not be used in templating directly
        # del context["flavor"]

I've also gained the new insight that it makes sense for @yajo to want to prevent scope creep of Copier itself. Perfectly fine for this kind of stuff to live in extensions.

But..... I only found out about copier-templates-extensions through the discussion on this PR. It's not mentioned in the official Copier docs! I didn't know this was possible and available! Which is too bad because it's a great feature!

Can I recommend at the least explaining Copier extendability and referencing copier-templates-extensions in the official Copier docs?

Even better would be to make copier-templates-extensions an official part of the Copier stack and moving it into the copier-org GitHub organization for improved visibility and governance - if @pawamoy agrees to transfer this to the copier-org umbrella of course. This could enable more features being developed as optional extensions out over time.

[pawamoy]

It's mentioned in the _jinja_extensions option docs (hint), but I agree that it tells nothing of its capabilities 😅 I'll try to add a paragraph to the docs somewhere, to explain this use-case and how this extension comes in handy. I also don't mind transfering the project to the copier-org, if that's something @yajo is okay with. I'd still maintain it of course.

Thanks for the update @verdaatt!

[yajo]

Both things sound good to me. 🙂 El mié., 13 abr. 2022 15:26, Timothée Mazzucotelli ***@***.***> escribió:

…

It's mentioned in the _jinja_extensions option docs (hint) <https://copier.readthedocs.io/en/latest/configuring/#jinja_extensions>, but I agree that it tells nothing of its capabilities 😅 I'll try to add a paragraph to the docs somewhere, to explain this use-case and how this extensions comes in handy. I also don't mind transfering the project to the copier-org, if that's something @yajo <https://github.com/Yajo> is okay with. I'd still maintain it of course. — Reply to this email directly, view it on GitHub <#629 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHNXDKAADNRWICVF3PZ2CLVE3KSXANCNFSM5R7M2LSA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[verdaatt]

Oh I didn't see that part! I've read the jinja_extensions section again and as a new user I didn't really understand it and its use case all that well initially. It's a bit brief and indeed misses some example use cases. I considered this a feature for advanced use cases that surely I wouldn't need because this was something basic to me.

Thanks @pawamoy for volunteering to ad that info. I think it would be really insightful to new users to explain how they can extend Copier functionality this way, and how they can use copier-template-extensions to manipulate and enhance data even before Jinja runs.

Also great that you two agree that bringing this under the Copier umbrella makes the product better. Looking forward to the move.

Should I leave this issue open until the docs are updated and maybe copier-template-extensions is moved?

[pawamoy]

Extension already moved 🙂
And I'm working on the docs! I should be able to open a draft PR tonight.

[pawamoy]

PR sent, let me know what you think @verdaatt @yajo 🙂