Template merge functionality

[rahulguptajss]

For this discussion purpose default template is lun.yaml and custom template is custom_lun.yaml

Current behaviour of custom template creation is documented at https://github.com/NetApp/harvest/blob/main/cmd/collectors/zapiperf/README.md#creatingediting-subtemplates

This approach requires copying all contents from default and then modify the custom template. This only supports complete overwrite of default templates. There are problems with this approach.

1. Any change in parameters to default templates from release to release will need manual diffing to maintain changes in custom template.
2. This is not good user experience to copy all contents from default template to custom template.

Proposed Change in Functionality:
Proposed solution intend to only change when object key defined in default.yaml and custom.yaml are same.
Solution is to overwrite top level parameters defined in custom template. All other params should be taken from default template.

Definition of top level parameters:
name,query,object,counters,plugins,export_options are the top level elements in below example.

lun.yaml

name:                       Lun
query:                      lun-get-iter
object:                     lun

counters:
  lun-info:
    - ^node
    - ^path
    - ^qtree
    - size
    - size-used
    - ^state
    - ^^uuid
    - ^volume
    - ^vserver => svm

plugins:
  - LabelAgent:
    value_mapping: status state online `1`
    # metric label zapi_value rest_value `default_value`
    value_to_num: new_status state online online `0`
    # path is something like "/vol/vol_georg_fcp401/lun401"
    # we only want lun name, which is 4th element
    split: path `/` ,,,lun

export_options:
  instance_keys:
    - node
    - qtree
    - lun
    - volume
    - svm
  instance_labels:
    - state

custom_lun.yaml

plugins:
  - LabelAgent:
    # my custom status
    value_mapping: custom_status state online `0`

Post merge lun.yaml

name:                       Lun
query:                      lun-get-iter
object:                     lun

counters:
  lun-info:
    - ^node
    - ^path
    - ^qtree
    - size
    - size-used
    - ^state
    - ^^uuid
    - ^volume
    - ^vserver => svm

plugins:
  - LabelAgent:
    # my custom status
    value_mapping: custom_status state online `0`

export_options:
  instance_keys:
    - node
    - qtree
    - lun
    - volume
    - svm
  instance_labels:
    - state

From above example plugins (top level param) got overwritten.

Advantages of this approach:

1. Better user experience.
2. Easy to compare and overwrite default templates.
3. Compare and merging all child elements inside top level elements may lead to confusion and manual comparision.

[hashi825]

If you aren't merging child elements of the dict keys then if you want to add one extra counter into your custom template you'd have to copy the whole contents of the counters dict again which is essentially back to the same issue of redefining the entire template, unless I'm reading it wrong?

For example above the post merge Lun.yaml loses important value mappings from the plugins dict being overwritten, if it's the default you'd want that merged with the custom template instead.

[rahulguptajss]

Intention of not merging child element is to avoid confusion where one would have to think about final merged template. If we go with child merging as well, we can build a cli to show the final merged template for debugging purpose.

[hashi825]

It would be definitely easier without child merging to avoid confusion but need to figure out the templating use case where you want the majority of the default template but want to add an extra counter and export a new label for example, without merging that requires copying most of the existing template, which then creates issues if the default template gets upgraded and we are back to square one.

[cgrinds]

thanks for clarifying @hashi825 - rahul and I discussed yesterday and agree that we need better support for the use-cases you mentioned above: adding counter, export new label, etc. He's working a PR that satisfies those cases as well as supporting the overwriting use-cases. stay tuned

[rahulguptajss]

• Template Merge Feature
• Add a cli to print merged template to console for debugging
• update docs

[cgrinds]

Verified, commit: c37b608

• verified adding a new object template (minor bug with docs docs: sensor docs need to match product #633)
• verified extending an existing object template (two small doc changes)
  • bc721d6
  • 300871d