25.02.0

[cgrinds]

25.02.0 / 2025-02-13 Release

📌 Highlights of this major release include:

⭐ New Features

• ⭐ The Volume dashboard was updated to clarify that volume latencies are missing some latencies from NAS protocols. Use the workload volume metrics in the QoS row for a more detailed breakdown. Thanks to MatthiasS for reporting.

• All Harvest dashboards default to Datacenter=All instead of the first datacenter in the list. Thanks to @roybatty2019 for reporting.

• Harvest provides a FIPS 140-3 compliant container image, available as a separate image at ghcr.io/netapp/harvest:25.02.0-1-fips.

• 🌾 Harvest bin/grafana import

  • Supports nested Grafana folders. Thanks to @IvanZenger for reporting.
  • Supports setting variables' default values during import. See #3384 for details. Thanks to @mamoep for reporting.

• Harvest collects shelf firmware versions and shows them in the Shelf dashboard, Module row. Thanks to @summertony15 for reporting.

• ⭐ Several of the existing dashboards include new panels in this release:

  • The Disk dashboard includes a Top Disk and Tape Drives Throughput by Host Adapter panel. Thanks to Amir for reporting.
  • The Datacenter and Data Protection dashboards were updated with data protection buckets and policy rows.

• The volumes templates exclude transient volumes by default. Thanks to Yann for reporting.

• Harvest collects rewind context (rwctx) metrics for ONTAP 9.16.0 and later. Thanks to @shawnahall71 for reporting.

• 📕 Documentation additions

  • Document Podman Quadlet as a deployment option. Thanks to ttlexceeded for reporting.
  • Describe how to use a Go binary as a credential script for Harvest. Thanks to AdiZ for reporting.

🚀 Performance Improvements

• RestPerf collector uses less memory by streaming results.

In case you missed the previous 24.11.1 dot release, here are the features included in it:

Performance Improvements in 24.11.1

• Significant memory footprint improvements for the REST collector. More details here. Thanks to @ryan for reporting it.
• Reduced memory footprint by using streaming in the REST collector.

New Features in 24.11.1

• Harvest supports Top files metrics collection. More details here.
• Volume and Cluster tags are supported via Volume and Cluster dashboards.
• Field Replaceable Unit (FRU) details have been added to the power dashboard.
• Track ONTAP image update progress for a cluster via the Cluster dashboard. Thanks to @knappmi for reporting it.
• prom_port is now supported within the poller. More details here.
• We've fixed an intermittent latency/operations spike issue in the plugin-generated Harvest performance metrics. Thanks to @wooyoungAhn for reporting it.

Announcements

‼️ IMPORTANT Harvest version 25.02.0 disables the out-of-the-box Qtree templates because of reported ONTAP slowdowns when collecting a large number of qtree objects. If you want to enable the Qtree templates, please see these instructions.

‼️ IMPORTANT Harvest version 25.02.0 removes the WorkloadDetail and WorkloadDetailVolume templates and all dashboard panels that use them. These templates are removed because they are expensive to collect and currently there is no way to collect them from ONTAP without introducing an unacceptable amount of skew in the results. See #3423 for details.

‼️ IMPORTANT If using Docker Compose and you want to keep your historical Prometheus data, please read how to migrate your Prometheus volume

💡 IMPORTANT After upgrade, don't forget to re-import your dashboards, so you get all the new enhancements and fixes. You can import them via the 'bin/harvest grafana import' CLI, from the Grafana UI, or from the 'Maintenance > Reset Harvest Dashboards' button in NAbox3. For NAbox4, this step is not needed.

Known Issues

💡 IMPORTANT FSx ZapiPerf workload collector fails to collect metrics, please use RestPerf instead.

Thanks to all the awesome contributors

🤘 Thanks to all the people who've opened issues, asked questions on Discord, and contributed code or dashboards this release:

@Falcon667, @IvanZenger, @cheese1, @embusalacchi, @mamoep, @roybatty2019, @summertony15, AdiZ, Amir, MatthiasS, Yann, ttlexceeded

🌱 This release includes 18 features, 22 bug fixes, 8 documentation, 1 performance, 1 refactoring, 6 miscellaneous, and 12 ci pull requests.

🚀 Features

• Hide Transient Volumes (#3337)
• Adding Ifgrp Api To Fetch Ifgrp Labels In Net_port (#3342)
• Adding Rwctx Template For Restperf In 9.16.0 (#3349)
• Add Disk And Tape Drives Throughput By Host Adapter (#3372)
• Adding The Bucket And Policy Rest Template (#3374)
• Include Lun And Namespace Templates In Keyperf (#3379)
• Dashboard Variables Set Default Value On Import (#3399)
• Optimize Workload_detail And Workload_detail_volume Through Delay Center Filter (#3406)
• Handled Empty Admin Svm In Plugin Call (#3410)
• Support Metric For Module Type In Frus (#3411)
• Harvest Grafana Import Should Support Nested Grafana Folders (#3417)
• Remove Workload Detail Templates (#3433)
• Add Streaming To Keyperf Collector (#3435)
• Negative Metrics Spike Handling (#3439)
• Disabled Qtree Perf Template And Update Docs (#3445)
• Harvest Dashboards Should Default To Datacenter=All (#3448)
• Update Qos Row In Volume Dashboard (#3453)
• Improve Cp Summary In Disk Dashboard (#3456)

🐛 Bug Fixes

• Include Instances Generated By Inbuilt Plugins In Plugininstances Log (#3343)
• Handled Duplicate Key In Securityauditdestination (#3348)
• Harvest Permissions Should Include Fru (#3354)
• No Instances Handling In Rest Collector (#3358)
• Rest No Instance Handling (#3360)
• Don't Clear Performance Volume Cache When There Is An Error (#3361)
• Failed To Find Scanner Instance In Cache Zapiperf (#3366)
• Installation Broken On Debian 11 Bullseye (#3368)
• Changed Var Label To Ne Null From Empty (#3385)
• Update Snapshot Policy Endpoint (#3391)
• Update Export Rule Endpoint (#3392)
• Upgrade Golang.org/X/Net Due To Dependabot Alert (#3395)
• Update Export Rule Endpoint (#3396)
• Remove Redundant Label From Node Template (#3404)
• Enable Request/Response Logging For Restperf (#3408)
• Disable Cache To Avoid Cache Poisoning Attack (#3409)
• Duplicate Time Series In Volume Dashboard (#3418)
• Update Tr Link In Security Dashboard (#3419)
• Typo (#3425)
• Handle Only Labels In Zapi Snapshotpolicy (#3444)
• "Top Ethernet Ports By Utilization %" Panel Legend Should Not In… (#3451)
• Handle Cp Labels In Dashboard (#3455)

📕 Documentation

• Fix Release Announcements (#3330)
• Keyperf Documentation (#3345)
• Updating Doc For Custom.yaml (#3352)
• Rest Endpoint Permissions (#3359)
• Add Go Binary Steps For Credential Script (#3381)
• Fix Alignment Of Template (#3421)
• Document Podman Quadlet As A Deployment Option (#3442)
• Add Description About Cp In Disk Dashboard (#3454)

⚡ Performance

• Restperf Should Stream Results While Parsing (#3356)

Refactoring

• Remove Openapi Dependency (#3436)

Miscellaneous

• Merge 24.11.1 To Main (#3328)
• Update Module github.com/Shirou/Gopsutil/V4 To V4.24.11 (#3347)
• Update All Dependencies (#3365)
• Update All Dependencies (#3407)
• Update All Dependencies (#3450)
• Update All Dependencies (#3457)

🔨 CI

• Fix Go Vet Errors (#3331)
• Add Misspell, Nakedret, And Predeclared Linter (#3350)
• Bump Go (#3351)
• Add Role Permissions Validation (#3375)
• Enable More Linters (#3413)
• Add Workflow Permissions At Codeql Recommendation (#3426)
• Add Workflow Permissions At Codeql Recommendation (#3427)
• Pin Actions To Sha (#3428)
• Cheese1 Has Signed The Ccla (#3429)
• Bump Go (#3430)
• Ci-Local Requires Passing Admin Argument (#3431)
• Bump Go (#3452)

---

This discussion was created from the release 25.02.0.

[WayneShen2]

After upgraded, Volume dashboard highlights no data if query 6 hours

But use 30 mins we can see data

[rahulguptajss]

If you click on these red icons in the corner in the first screenshot, does it show any error messages? From which version did you upgrade to 25.02?

[WayneShen2]

I upgrade from 24.11.1
6h error

2 days is normal

[rahulguptajss]

Thanks. There were 2 labels added on 24.11 for volume_labels: is_space_reporting_logical and is_space_enforcement_logical. Based on the error, the two time series with and without these labels are causing this many-to-many issue. Let's focus on the Volume Average Latency panel. If you update its panel query to the one below, does it fix the error for that panel?

avg(
  topk(
    $TopResources,
    (
      avg by (datacenter, cluster, svm, volume) (
        volume_avg_latency{datacenter=~"$Datacenter", cluster=~"$Cluster", svm=~"$SVM", volume=~"$Volume", style!="flexgroup_constituent"}
      )
    )
    * on (datacenter, cluster, svm, volume) group_left(node)
    (
      max by (datacenter, cluster, svm, volume, node) (
        volume_labels{datacenter=~"$Datacenter", cluster=~"$Cluster", svm=~"$SVM", volume=~"$Volume", tags=~".*$Tag.*"}
      )
    )
  )
)

[WayneShen2]

That is working for Volume Average Latency.

I could see performance also no data

[rahulguptajss]

Thanks. This will be fixed for all panels via issue #3480

[WayneShen2]

If we do not change after 1days later that will normal.

[rahulguptajss]

Yes, That is expected. We'll share fix for this asap.

[rahulguptajss]

@WayneShen2 could you please import the JSON file into grafana at: https://raw.githubusercontent.com/NetApp/harvest/185c8f681bff83f7ebff7b0469d54ecb9e462458/grafana/dashboards/cmode/volume.json and verify if it resolves the issue with the volume dashboard?

[WayneShen2]

Thank you. New dashboard is working.

[rahulguptajss]

Thank you. New dashboard is working.

Great! Fix is also available in nightly release.