Skip to content

Commit

Permalink
fix: improve template yaml parsing (#710)
Browse files Browse the repository at this point in the history 
* fix: improve template yaml parsing

Fixes #704

* fix: invalid yaml in testcases
* fix: remove earlier yaml parser
  • Loading branch information
@cgrinds
cgrinds authored Nov 30, 2021
1 parent 1b358d1 commit b09df58
Show file tree
Hide file tree
Showing 15 changed files with 154 additions and 271 deletions.
12 changes: 6 additions & 6 deletions cmd/poller/collector/helpers.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -29,12 +29,12 @@ import (
)

// ImportTemplate retrieves the config (template) of a collector, arguments are:
// @confDir - path of Harvest config durectory (usually /etc/harvest)
// @confDir - path of Harvest config directory (usually /etc/harvest)
// @confFn - template filename (e.g. default.yaml or custom.yaml)
// @collectorName - name of the collector
func ImportTemplate(confPath, confFn, collectorName string) (*node.Node, error) {
fp := path.Join(confPath, "conf/", strings.ToLower(collectorName), confFn)
return tree.Import("yaml", fp)
return tree.ImportYaml(fp)
}

// ImportSubTemplate retrieves the best matching subtemplate of a collector object.
Expand Down Expand Up @@ -105,7 +105,7 @@ func (c *AbstractCollector) ImportSubTemplate(model, filename string, ver [3]int
verS, err := version.NewVersion(verWithDots)
if err != nil {
c.Logger.Trace().Msgf("error parsing ONTAP version: %s err: %s", verWithDots, err)
return nil, errors.New("No best-fitting subtemplate version found")
return nil, errors.New("no best-fitting subtemplate version found")
}
// get closest index
idx := getClosestIndex(versions, verS)
Expand All @@ -115,18 +115,18 @@ func (c *AbstractCollector) ImportSubTemplate(model, filename string, ver [3]int
}

if selectedVersion == "" {
return nil, errors.New("No best-fit template found")
return nil, errors.New("no best-fit template found")
}

subTemplateFp = path.Join(pathPrefix, selectedVersion, f)
c.Logger.Info().Msgf("best-fit template [%s] for [%s]", subTemplateFp, verWithDots)
if finalTemplate == nil {
finalTemplate, err = tree.Import("yaml", subTemplateFp)
finalTemplate, err = tree.ImportYaml(subTemplateFp)
if err == nil {
finalTemplate.PreprocessTemplate()
}
} else {
tempTemplate, err = tree.Import("yaml", subTemplateFp)
tempTemplate, err = tree.ImportYaml(subTemplateFp)
if err == nil {
tempTemplate.PreprocessTemplate()
// merge templates
Expand Down
37 changes: 18 additions & 19 deletions cmd/poller/plugin/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ plugins:
# will aggregate a new Matrix based on target label LABEL
```

If you want to speficy which labels should be included in the new instances, you can add those space-seperated after `LABEL`:
If you want to specify which labels should be included in the new instances, you can add those space-seperated after `LABEL`:

```yaml
- LABEL LABEL1,LABEL2
Expand All @@ -47,7 +47,7 @@ Or include all labels:
- LABEL ...
# copy all labels of the original instance
```
By default aggregated metrics will be prefixed with `LABEL`. For example if the object of the original Matrix is `volume` (meaning metrics are prefixed with `volume_`) and `LABEL` is `aggr`, then the metric `volume_read_ops` will become `aggr_volume_read_ops`, etc. You can override this by providing the `<>OBJ` using the following syntax:
By default, aggregated metrics will be prefixed with `LABEL`. For example if the object of the original Matrix is `volume` (meaning metrics are prefixed with `volume_`) and `LABEL` is `aggr`, then the metric `volume_read_ops` will become `aggr_volume_read_ops`, etc. You can override this by providing the `<>OBJ` using the following syntax:

```yaml
- LABEL<>OBJ
Expand All @@ -71,14 +71,13 @@ Examples:
```yaml
plugins:
Aggregator:
# will aggregate metrics of the aggregate. The labels "node" and "type" are included in the new instances
- aggr node type
# will aggregate metrics of Anggregate level and make sure
# labels "node" and "type" are included in the new instances
# aggregate instances if label "type" has value "flexgroup"
# include all original labels
- type<flexgroup> ...
# aggregate instances if label "type" has value "flexgroup"
# include all original labels
# aggregate all instances if value of "volume" ends with underscore and 4 digits
- volume<`_\d{4}$`>
# aggregate all instances if value of "volume" ends with underscore and 4 digits
```
### Aggregation rules
Expand All @@ -94,7 +93,7 @@ The plugin tries to intelligently aggregate metrics based on a few rules:
- **Ignore** - metrics created by some plugins, such as value_to_num by LabelAgent

# LabelAgent
LabelAgent allows to manipulate instance labels based on rules. You can define multiple rules, here is an example of what you could add to the yaml file of a collector:
LabelAgent are used to manipulate instance labels based on rules. You can define multiple rules, here is an example of what you could add to the yaml file of a collector:


```yaml
Expand Down Expand Up @@ -130,23 +129,23 @@ Rule syntax:
```yaml
split: LABEL `SEP` LABEL1,LABEL2,LABEL3
# source label - seperator - comma-seperated target labels
# source label - separator - comma-seperated target labels
```

Splits the value of a given label by seperator `SEP` and creates new labels if their number matches to the number of target labels defined in rule. To discard a subvalue, just add a redundant `,` in the names of the target labels.
Splits the value of a given label by separator `SEP` and creates new labels if their number matches to the number of target labels defined in rule. To discard a subvalue, just add a redundant `,` in the names of the target labels.

Example:

```yaml
split: node `/` ,aggr,plex,disk
# will split the value of "node" using seperator "/"
# will split the value of "node" using separator "/"
# will expect 4 values: first will be discarded, remaining
# three will be stored as labels "aggr", "plex" and "disk"
```

## split_regex

Does the same as `split_regex` but uses a regular expression instead of a seperator.
Does the same as `split_regex` but uses a regular expression instead of a separator.

Rule syntax:

Expand All @@ -170,7 +169,7 @@ Rule syntax:

```yaml
split_pairs: LABEL `SEP1` `SEP2`
# source label - pair seperator - key-value seperator
# source label - pair separator - key-value separator
```

Extracts key-value pairs from the value of source label `LABEL`. Note that you need to add these keys in the export options, otherwise they will not be exported.
Expand All @@ -186,13 +185,13 @@ split_pairs: comment ` ` `:`

## join

Join multiple label values using seperator `SEP` and create a new label.
Join multiple label values using separator `SEP` and create a new label.

Rule syntax:

```yaml
join: LABEL `SEP` LABEL1,LABEL2,LABEL3
# target label - seperator - comma-seperated source labels
# target label - separator - comma-seperated source labels
```

Example:
Expand Down Expand Up @@ -307,7 +306,7 @@ value_mapping was deprecated in 21.11 and removed in 22.02. Use [value_to_num ma

## value_to_num

Maps values of a given label to a numeric metric (of type `uint8`).
Map values of a given label to a numeric metric (of type `uint8`).
This rule maps values of a given label to a numeric metric (of type `unit8`). Healthy is mapped to 1 and all non-healthy values are mapped to 0.

This is handy to manipulate the data in the DB or Grafana (e.g. change color based on status or create alert).
Expand All @@ -318,8 +317,8 @@ Rule syntax:

```yaml
value_to_num: METRIC LABEL ZAPI_VALUE REST_VALUE `N`
# maps values of LABEL to 1 if it is ZAPI_VALUE or REST_VALUE
# otherwise value of METRIC is set to N
# map values of LABEL to 1 if it is ZAPI_VALUE or REST_VALUE
# otherwise, value of METRIC is set to N
```
The default value `N` is optional, if no default value is given and the label value does not match any of the given values, the metric value will not be set.

Expand All @@ -341,6 +340,6 @@ value_to_num: status state up online `4`
```yaml
value_to_num: status outage - - `0` #ok_value is empty value.
# metric value will be set to 1 if "outage" is empty, if it's any other value, it will be set to the default, 0
# '-' is a special symbol in this mapping and it will be converted to blank while processing.
# '-' is a special symbol in this mapping, and it will be converted to blank while processing.
```

4 changes: 3 additions & 1 deletion cmd/poller/plugin/label_agent/parse_rules.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,9 @@ func (me *LabelAgent) parseRules() int {
case "value_to_num":
me.parseValueToNumRule(rule)
default:
me.Logger.Warn().Msgf("unknown rule (%s)", name)
me.Logger.Warn().
Str("object", me.ParentParams.GetChildContentS("object")).
Str("name", name).Msg("Unknown rule name")
}
}
}
Expand Down
4 changes: 2 additions & 2 deletions cmd/tools/doctor/doctor.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -49,12 +49,12 @@ func doMergeCmd(_ *cobra.Command, _ []string) {
}

func doMerge(path1 string, path2 string) {
template, err := tree.Import("yaml", path1)
template, err := tree.ImportYaml(path1)
if err != nil {
fmt.Printf("error reading template file [%s]. err=%+v\n", path1, err)
return
}
subTemplate, err := tree.Import("yaml", path2)
subTemplate, err := tree.ImportYaml(path2)
if err != nil {
fmt.Printf("error reading template file [%s] err=%+v\n", path2, err)
return
Expand Down
2 changes: 1 addition & 1 deletion conf/rest/9.13.0/volume.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ plugins:
replace:
- style style `flexgroup_constituent` `flexgroup`
Aggregator:
volume<style=flexgroup>volume node,svm,aggr,style
- volume<style=flexgroup>volume node,svm,aggr,style

export_options:
instance_keys:
Expand Down
90 changes: 45 additions & 45 deletions conf/zapi/7mode/8.6.0/shelf.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,56 +26,56 @@ plugins:
Shelf:
objects:
- cooling-element-list => fan:
cooling-element-info:
- ^^cooling-element-number => fan_id
- ^location #fake
- rpm
# - ^cooling-element-is-error
# - ^cooling-element-is-not-installed
cooling-element-info:
- ^^cooling-element-number => fan_id
- ^location #fake
- rpm
# - ^cooling-element-is-error
# - ^cooling-element-is-not-installed
- current-sensor-list => sensor:
current-sensor-info:
- ^^current-sensor-no => sensor_id
- ^location #fake
- sensor-reading => reading
# - ^is-sensor-error
# - ^is-sensor-not-installed
# - ^sensor-condition
current-sensor-info:
- ^^current-sensor-no => sensor_id
- ^location #fake
- sensor-reading => reading
# - ^is-sensor-error
# - ^is-sensor-not-installed
# - ^sensor-condition
- temp-sensor-list => temperature:
temp-sensor-info:
- ^^temp-sensor-element-no => sensor_id
- temp-sensor-current-temperature => reading
- ^temp-sensor-hi-critical => high_critical
- ^temp-sensor-hi-warning => high_warning
- ^temp-sensor-is-ambient => temp_is_ambient
- ^temp-sensor-low-critical => low_critical
- ^temp-sensor-low-warning => low_warning
# - ^temp-sensor-is-error
# - ^temp-sensor-is-not-installed
# - ^temp-sensor-current-condition
temp-sensor-info:
- ^^temp-sensor-element-no => sensor_id
- temp-sensor-current-temperature => reading
- ^temp-sensor-hi-critical => high_critical
- ^temp-sensor-hi-warning => high_warning
- ^temp-sensor-is-ambient => temp_is_ambient
- ^temp-sensor-low-critical => low_critical
- ^temp-sensor-low-warning => low_warning
# - ^temp-sensor-is-error
# - ^temp-sensor-is-not-installed
# - ^temp-sensor-current-condition
- voltage-sensor-list => voltage:
voltage-sensor-info:
- ^^voltage-sensor-no => sensor_id
- sensor-reading => reading
- ^location #fake
# - ^is-sensor-error
# - ^is-sensor-not-installed
# - ^sensor-condition
voltage-sensor-info:
- ^^voltage-sensor-no => sensor_id
- sensor-reading => reading
- ^location #fake
# - ^is-sensor-error
# - ^is-sensor-not-installed
# - ^sensor-condition

- power-supply-list => psu:
power-supply-info:
- ^^power-supply-element-number => psu_id
- ^power-supply-part-no => part_number
- ^power-supply-serial-no => serial
# - power-supply-swap-count => psu_swap_count
- ^power-supply-type => type
# fake counters to match CDOT
- ^enabled
- ^location
# - ^is-auto-power-reset-enabled
# - ^is-power-reset-capable
# - ^power-supply-firmware-revision
# - ^power-supply-is-error
# - ^power-supply-is-not-installed
power-supply-info:
- ^^power-supply-element-number => psu_id
- ^power-supply-part-no => part_number
- ^power-supply-serial-no => serial
# - power-supply-swap-count => psu_swap_count
- ^power-supply-type => type
# fake counters to match CDOT
- ^enabled
- ^location
# - ^is-auto-power-reset-enabled
# - ^is-power-reset-capable
# - ^power-supply-firmware-revision
# - ^power-supply-is-error
# - ^power-supply-is-not-installed
LabelAgent:
# metric label zapi_value rest_value `default_value`
value_to_num:
Expand Down
74 changes: 37 additions & 37 deletions conf/zapi/cdot/9.8.0/shelf.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,47 +20,47 @@ plugins:
Shelf:
objects:
- cooling-fans => fan:
storage-shelf-cooling-fan-info:
- ^^fan-id
- ^fan-location => location
- ^fan-op-status => status
- fan-rpm => rpm
storage-shelf-cooling-fan-info:
- ^^fan-id
- ^fan-location => location
- ^fan-op-status => status
- fan-rpm => rpm
- current-sensors => sensor:
storage-shelf-current-sensor-info:
- ^^current-sensor-id => sensor_id
- ^current-sensor-location => location
- ^current-op-status => status
- current-sensor-reading => reading
storage-shelf-current-sensor-info:
- ^^current-sensor-id => sensor_id
- ^current-sensor-location => location
- ^current-op-status => status
- current-sensor-reading => reading
- power-supply-units => psu:
storage-shelf-power-supply-unit-info:
# - ^psu-crest-factor => crest_factor
# - ^psu-fw-version => fw_version
- ^^psu-id
- ^psu-is-enabled => enabled
- ^psu-location => location
- ^psu-part-number => part_number
- psu-power-drawn => power_drawn
- psu-power-rating => power_rating
# - ^psu-reset-capable => reset_capable
- ^psu-serial-number => serial
- ^psu-type => type
- ^psu-op-status => status
storage-shelf-power-supply-unit-info:
# - ^psu-crest-factor => crest_factor
# - ^psu-fw-version => fw_version
- ^^psu-id
- ^psu-is-enabled => enabled
- ^psu-location => location
- ^psu-part-number => part_number
- psu-power-drawn => power_drawn
- psu-power-rating => power_rating
# - ^psu-reset-capable => reset_capable
- ^psu-serial-number => serial
- ^psu-type => type
- ^psu-op-status => status
- temperature-sensors => temperature:
storage-shelf-temperature-sensor-info:
- ^high-critical-threshold => high_critical
- ^high-warning-threshold => high_warning
- ^temp-is-ambient
- ^temp-low-critical-threshold => low_critical
- ^temp-low-warning-threshold => low_warning
- ^^temp-sensor-id => sensor_id
- ^temp-sens-op-status => status
- temp-sensor-reading => reading
storage-shelf-temperature-sensor-info:
- ^high-critical-threshold => high_critical
- ^high-warning-threshold => high_warning
- ^temp-is-ambient
- ^temp-low-critical-threshold => low_critical
- ^temp-low-warning-threshold => low_warning
- ^^temp-sensor-id => sensor_id
- ^temp-sens-op-status => status
- temp-sensor-reading => reading
- voltage-sensors => voltage:
storage-shelf-voltage-sensor-info:
- ^^voltage-sensor-id => sensor_id
- ^voltage-sensor-location => location
- ^voltage-op-status => status
- voltage-sensor-reading => reading
storage-shelf-voltage-sensor-info:
- ^^voltage-sensor-id => sensor_id
- ^voltage-sensor-location => location
- ^voltage-op-status => status
- voltage-sensor-reading => reading

LabelAgent:
# metric label zapi_value rest_value `default_value`
Expand Down
Loading

0 comments on commit b09df58

Please sign in to comment.