failedTemplate error not exists or not selected in test suite

[AmitBenAmi]

Hi

I have a test which I'm trying to test that it is failing with

This is my _helpers.tpl:

{{- define "my-chart.checkNameOverrideLength" -}}
{{- if .Values.nameOverride -}}
{{- if gt (len .Values.nameOverride) 54 -}}
{{- fail "nameOverride cannot be longer than 54 characters" -}}
{{- end -}}
{{- end -}}
{{- end -}}

{{- define "my-chart.name" -}}
{{- include "web-api.checkNameOverrideLength" . -}}
{{- .Values.nameOverride -}}
{{- end -}}

This is my test:

suite: test
templates:
  - deployment.yaml
tests:
  - it: should fail since the nameOverride is too long
    values:
      - ../../values.yaml
    set:
      nameOverride: too-long-of-a-name-override-that-should-fail-the-template-immediately
    asserts:
      - failedTemplate:
          errorMessage: nameOverride cannot be longer than 54 characters

As you can see, all I try to check is that I get the error from the template, but I keep getting this error:

 FAIL  test     helm-unittest/common/values_test.yaml
        - should fail since the nameOverride is too long

                - asserts[0] `failedTemplate` fail

                        Error:
                                template "my-chart/templates/deployment.yaml" not exists or not selected in test suite


Charts:      1 failed, 0 passed, 1 total
Test Suites: 1 failed, 0 passed, 1 total
Tests:       1 failed, 0 passed, 1 total
Snapshot:    0 passed, 0 total
Time:        18.780023ms

Error: plugin "unittest" exited with error

Any reason why it could happen and how can I resolve it?
Thanks

[jonastr]

+1

i am having the same problem when using istio 1.3.3 as dependent chart. it works with istio 1.4.8.

interestingly enough, helm template works just fine. i am suspecting some rendering quirk inside helm-unittest to be the culprit.

[quintush]

+1

i am having the same problem when using istio 1.3.3 as dependent chart. it works with istio 1.4.8.

interestingly enough, helm template works just fine. i am suspecting some rendering quirk inside helm-unittest to be the culprit.

Thanks for the update. I'm currently working on a fix regarding this issue. Seems that failing validations within template files (e.g. _helpers.tpl) have a different result in Helm3 than in Helm2.

A workaround could be by preventing the failure to come, using predefined values 😁

[jonastr]

@quintush thanks for the quick reply!
only to add to my previous report - to make it more precise: in our case, it's this part of the changeset between istio 1.4.7 and 1.4.8

diff --git a/install/kubernetes/helm/istio/charts/gateways/templates/role.yaml b/install/kubernetes/helm/istio/charts/gateways/templates/role.yaml
index 37bdf3ef87..6ebfb3504e 100644
--- a/install/kubernetes/helm/istio/charts/gateways/templates/role.yaml
+++ b/install/kubernetes/helm/istio/charts/gateways/templates/role.yaml
@@ -1,7 +1,8 @@
 {{- range $key, $spec := .Values }}
 {{- if ne $key "enabled" }}
 {{- if $spec.enabled }}
-{{- if ($spec.sds) and (eq $spec.sds.enabled true) }}
+{{- if $spec.sds }}
+{{- if eq $spec.sds.enabled true }}
 apiVersion: rbac.authorization.k8s.io/v1
 kind: Role
 metadata:
@@ -16,3 +17,4 @@ rules:
 {{- end }}
 {{- end }}
 {{- end }}
+{{- end }}

interestingly enough, helm3 also complains about the and in the condition {{- if ($spec.sds) and (eq $spec.sds.enabled true) }}.

[quintush]

Thanks for the additional details.
It will help solve the issue.

[quintush]

@quintush thanks for the quick reply!
only to add to my previous report - to make it more precise: in our case, it's this part of the changeset between istio 1.4.7 and 1.4.8

diff --git a/install/kubernetes/helm/istio/charts/gateways/templates/role.yaml b/install/kubernetes/helm/istio/charts/gateways/templates/role.yaml
index 37bdf3ef87..6ebfb3504e 100644
--- a/install/kubernetes/helm/istio/charts/gateways/templates/role.yaml
+++ b/install/kubernetes/helm/istio/charts/gateways/templates/role.yaml
@@ -1,7 +1,8 @@
 {{- range $key, $spec := .Values }}
 {{- if ne $key "enabled" }}
 {{- if $spec.enabled }}
-{{- if ($spec.sds) and (eq $spec.sds.enabled true) }}
+{{- if $spec.sds }}
+{{- if eq $spec.sds.enabled true }}
 apiVersion: rbac.authorization.k8s.io/v1
 kind: Role
 metadata:
@@ -16,3 +17,4 @@ rules:
 {{- end }}
 {{- end }}
 {{- end }}
+{{- end }}

interestingly enough, helm3 also complains about the and in the condition {{- if ($spec.sds) and (eq $spec.sds.enabled true) }}.

After seeing the code, i noticed that the and is positioned wrong.
I would expect the following condition would work, as the setup is based on the golang template package:

{{- if and ($spec.sds) (eq $spec.sds.enabled true) }}

Similar setup is created in one of the samples in the unittest plugin.

Greetings,
@quintush

[jonastr]

@quintush yes and no, i would say. :)

generally, my expectation towards helm unittest would be that any code that passes helm template (and helm lint for the matter) would also be accepted by helm unittest as long as the same helm version are used.

so, yes, changing the syntax on the above example makes the unittests work. however, helm template also works with 'wrong' syntax. so why is it that this code only fails with helm unittest?

cheers,
@jonastr

[jonastr]

@quintush some more thoughts below.

as far as i understood from the code of helm unittest it's essentially deferring templating to helm libraries. it is therefore odd that helm unittest shows different behavior (like the failure we are looking at) than helm itself.

some ideas:

• are there minor/patch version of the used helm templating library that helm unittest is missing out on, which contain a fix for this?
• is helm unittest doing something different w.r.t. templating when compared to helm itself?

cheers
@jonastr

[quintush]

Hello @jonastr ,

Indeed the helm unittest is deferring templating to helm libraries. I wanted to use the rendering from Helm itself as much as possible, as some template functions are only available within Helm, which otherwise needed to be developed separatly in the helm-unittest.

Currently i use the following versions to render the helm charts:
version 2.16.10 for helm 2
version 3.3.1 for helm 3

So i will update to the latest versions to verify if it will solve the error.
Otherwise i will investigate what i'm missing in the rendering that the unittest plugin is not rendering the same as helm (as i goal was that it should)

Greetings,
@quintush

[jonastr]

hi @quintush,

yeah, deferring makes a lot of sense to me as well. i wasn't aware though that unittest would eventually use a different version of helm libraries than the version of helm installed.

for example, i am using helm version v2.16.3 here. so that's actually an older version and templating works fine. this could imply that a bug has been introduced in newer versions of helm ;)

to get consistent results, it would make sense to me to always use the same version. i therefore wonder if it'd be easily doable to change helm unittest such that it defers templating to the installed version of helm - rather than tying it to a specific version of helm used during build time of helm unittest.

---

if we can confirm/reject that this error is related to the current helm version used in helm unittest, we know better how to fix it. :)

---

cheers,
@jonastr

[AmitBenAmi]

But what about the fail templating in Helm?
Helm template will fail this, but helm-unittest is failing itself to make the tests