21.05.4

Change Log

Releases

21.05.4 / 2021-07-22

This release introduces Qtree protocol collection, improved Docker and client authentication documentation, publishing to Docker Hub, and a new plugin that helps build richer dashboards, as well as a couple of important fixes for collector panics.

IMPORTANT RPM and Debian packages will be deprecated in the future, replaced with Docker and native binaries. See #330 for details and tell us what you think.

Known Issues

On RHEL and Debian, the example Unix collector does not work at the moment due to the harvest user lacking permissions to read the /proc filesystem. See #249 for details.

Enhancements

• Harvest collects Qtree protocol ops #298. Thanks to Martin Möbius for contributing

• Harvest Grafana tool (optionally) adds a user-specified prefix to all Dashboard metrics during import. See harvest grafana --help #87

• Harvest is taking its first steps to talk REST: query ONTAP, show Swagger API, model, and definitions #292

• Tagged releases of Harvest are published to Docker Hub

• Harvest honors Go's http(s) environment variable proxy information. See https://pkg.go.dev/net/http#ProxyFromEnvironment for details #252

• New plugin value_to_num helps map labels to numeric values for Grafana dashboards. Current dashboards updated to use this plugin #319

• harvest.yml supports YAML flow style. E.g. collectors: [Zapi] #260

• New Simple collector that runs on Macos and Unix #270

• Improve client certificate authentication documentation

• Improve Docker deployment documentation 4019308

Fixes

• Harvest collector should not panic when resources are deleted from ONTAP #174 and #302. Thanks to @hashi825 and @mamoep for providing steps to reproduce

• Shelf metrics should report on op-status for components. Thanks to @hashi825 for working with us on this fix and dashboard improvements #262

• Harvest should not panic when InfluxDB is the only exporter #286

• Volume dashboard space-used column should display with percentage filled. Thanks to @florianmulatz for reporting and suggesting a fix #303

• Certificate authentication should honor path in harvest.yml #318

• Harvest should not kill processes with poller in their arguments #328

• Harvest ZAPI command line tool should limit perf-object-get-iter to subset of counters when using --counter #299

---

21.05.3

Change Log

Releases

21.05.3 / 2021-06-28

This release introduces a significantly simplified way to connect Harvest and Prometheus, containerization enchantments, improved Harvest build times by 7x, reduced executable sizes by 3x, enabled cross compiling support, and several dashboard and other fixes.

🎉 Thanks especially to @hashi825, @mamoep, @matejzero, and @florianmulatz for opening issues and pitching in to help fix them this release.

Known Issues

On RHEL and Debian, the example Unix collector does not work at the moment due to the harvest user lacking permissions to read the /proc filesystem. See #249 for details.

Enhancements

• Create Prometheus port range exporter that allows you to connect multiple pollers to Prometheus without needing to specify a port-per-poller. This makes it much easier to connect Prometheus and Harvest; especially helpful when you're monitoring many clusters #172

• Improve Harvest build times by 7x and reduce executable sizes by 3x #100

• Improve containerization with the addition of a poller-per-container Dockerfile. Create a new subcommand harvest generate docker which generates a docker-compose.yml file for all pollers defined in your config

• Improve systemd integration by using instantiated units for each poller and a harvest target to tie them together. Create a new subcommand harvest generate systemd which generates a Harvest systemd target for all pollers defined in your config #systemd

• Harvest doctor checks that all Prometheus exporters specify a unique port #118

• Harvest doctor warns when an unknown exporter type is specified (likely a spelling error) #118

• Add Harvest CUE validation and type-checking #208

• bin/zapi uses the --config command line option to read the harvest config file. This brings this tool inline with other Harvest tools. This makes it easier to switch between multiple sets of harvest.yml files.

• Harvest no longer writes pidfiles; simplifying management code and install #159

Fixes

• Ensure that the Prometheus exporter does not create duplicate labels #132

• Ensure that the Prometheus exporter includes HELP and TYPE metatags when requested. Some tools require these #104

• Disk status should return zero for a failed disk and one for a healthy disk. Thanks to @hashi825 for reporting and fixing #182

• Lun info should be collected by Harvest. Thanks to @hashi825 for reporting and fixing #230

• Grafana dashboard units, typo, and filtering fixes. Thanks to @mamoep, @matejzero, and @florianmulatz for reporting these 🎉 #184 #186 #190 #192 #195 #202

• Unix collector should not panic when harvest.yml is changed #160

• Reduce log noise about poller lagging behind by few milliseconds. Thanks @hashi825 #214

• Don't assume debug when foregrounding the poller process. Thanks to @florianmulatz for reporting. #246

• Improve Docker all-in-one-container argument handling and simplify building in air gapped environments. Thanks to @optiz0r for reporting these issues and creating fixes. #166 #167 #168

21.05.2

Change Log

Releases

21.05.2 / 2021-06-14

This release adds support for user-defined URLs for InfluxDB exporter, a new command to validate your harvest.yml file, improved logging, panic handling, and collector documentation. We also enabled GitHub security code scanning for the Harvest repo to catch issues sooner. These scans happen on every push.

There are also several quality-of-life bug fixes listed below.

Fixes

• Handle special characters in cluster credentials #79
• TLS server verification works with basic auth #51
• Collect metrics from all disk shelves instead of one #75
• Disk serial number and is-failed are missing from cdot query #60
• Ensure collectors and pollers recover from panics #105
• Cluster status is initially reported, but then stops being reported #66
• Performance metrics don't display volume names #40
• Allow insecure Grafana TLS connections --insecure and honor requested transport. See harvest grafana --help for details #111
• Prometheus dashboards don't load when exemplar is true. Thanks to @sevenval-admins, @florianmulatz, and @unbreakabl3 for their help tracking this down and suggesting a fix. #96
• harvest stop does not stop pollers that have been renamed #20
• Harvest stops working after reboot on rpm/deb #50
• harvest start shall start as harvest user in rpm/deb #129
• harvest start detects stale pidfiles and makes start idempotent #123
• Don't include unknown metrics when talking with older versions of ONTAP #116

Enhancements

• InfluxDB exporter supports user-defined URLs
• Add workload counters to ZapiPerf #9
• Add new command to validate harvest.yml file and optionally redact sensitive information #16 e.g. harvest doctor --config ./harvest.yml
• Improve documentation for Unix, Zapi, and ZapiPerf collectors
• Add Zerolog framework for structured logging #61
• Vendor 3rd party code to increase reliability and make it easier to build in air-gapped environments #26
• Make contributing easier with a digital CCLA instead of 1970's era PDF :)
• Enable GitHub security code scanning
• InfluxDB exporter provides the option to pass the URL end-point unchanged. Thanks to @steverweber for their suggestion and validation. #63

21.05.1

Change Log

Releases

21.05.1 / 2021-05-20

Announcing the release of Harvest2. With this release the core of Harvest has been completely rewritten in Go. Harvest2 is a replacement for the older versions of Harvest 1.6 and below.

If you're using one of the Harvest 2.x release candidates, you can do a direct upgrade.

Going forward Harvest2 will follow a year.month.fix release naming convention with the first release being 21.05.0. See SUPPORT.md for details.

IMPORTANT v21.05 increased Harvest's out-of-the-box security posture - self-signed certificates are rejected by default. You have two options:

1. Setup client certificates for each cluster
2. Disable the TLS check in Harvest. To disable, you need to edit harvest.yml and add use_insecure_tls=true to each poller or add it to the Defaults section. Doing so tells Harvest to ignore invalid TLS certificates.

IMPORTANT RPM and Debian packages will be deprecated in the future, replaced with Docker and native packages.

IMPORTANT Harvest 1.6 is end of support. We recommend you upgrade to Harvest 21.05 to take advantage of the improvements.

Changes since rc2

Fixes

• Log mistyped exporter names and continue, instead of stopping
• harvest grafana should work with custom harvest.yml files passed via --config
• Harvest will try harder to stop pollers when they're stuck
• Add Grafana version check to ensure Harvest can talk to a supported version of Grafana
• Normalize rate counter calculations - improves latency values
• Workload latency calculations improved by using related objects operations
• Make cli flags consistent across programs and subcommands
• Reduce aggressive logging; if first object has fatal errors, abort to avoid repetitive errors
• Throw error when use_insecure_tls is false and there are no certificates setup for the cluster
• Harvest status fails to print port number after restart
• RPM install should create required directories
• Collector now warns if it falls behind schedule
• package.sh fails without internet connection
• Version flag is missing new line on some shells #4
• Poller should not ignore --config #28

Enhancements

• Add new exporter for InfluxDB
• Add native install package
• Add ARCHITECTURE.md and improve overall documentation
• Use systemd harvest.service on RPM and Debian installs to manage Harvest
• Add runtime profiling support - off by default, enabled with --profiling flag. See harvest start --help for details
• Document how to use ONTAP client certificates for password-less polling
• Add per-poller Prometheus end-point support with promPort
• The release, commit and build date information are baked into the release executables
• You can pick a subset of pollers to manage by passing the name of the poller to harvest. e.g. harvest start|stop|restart POLLERS

21.05.0

Change Log

Releases

21.05.0 / 2021-05-20

Announcing the release of Harvest2. With this release the core of Harvest has been completely rewritten in Go. Harvest2 is a replacement for the older versions of Harvest 1.6 and below.

If you're using one of the Harvest 2.x release candidates, you can do a direct upgrade.

Going forward Harvest2 will follow a year.month.fix release naming convention with the first release being 21.05.0. See SUPPORT.md for details.

IMPORTANT v21.05 increased Harvest's out-of-the-box security posture - self-signed certificates are rejected by default. You have two options:

1. Setup client certificates for each cluster
2. Disable the TLS check in Harvest. To disable, you need to edit harvest.yml and add use_insecure_tls=true to each poller or add it to the Defaults section. Doing so tells Harvest to ignore invalid TLS certificates.

IMPORTANT RPM and Debian packages will be deprecated in the future, replaced with Docker and native packages.

IMPORTANT Harvest 1.6 is end of support. We recommend you upgrade to Harvest 21.05 to take advantage of the improvements.

Changes since rc2

Fixes

• Log mistyped exporter names and continue, instead of stopping
• harvest grafana should work with custom harvest.yml files passed via --config
• Harvest will try harder to stop pollers when they're stuck
• Add Grafana version check to ensure Harvest can talk to a supported version of Grafana
• Normalize rate counter calculations - improves latency values
• Workload latency calculations improved by using related objects operations
• Make cli flags consistent across programs and subcommands
• Reduce aggressive logging; if first object has fatal errors, abort to avoid repetitive errors
• Throw error when use_insecure_tls is false and there are no certificates setup for the cluster
• Harvest status fails to print port number after restart
• RPM install should create required directories
• Collector now warns if it falls behind schedule
• package.sh fails without internet connection
• Version flag is missing new line on some shells #4

Enhancements

• Add new exporter for InfluxDB
• Add native install package
• Add ARCHITECTURE.md and improve overall documentation
• Use systemd harvest.service on RPM and Debian installs to manage Harvest
• Add runtime profiling support - off by default, enabled with --profiling flag. See harvest start --help for details
• Document how to use ONTAP client certificates for password-less polling
• Add per-poller Prometheus end-point support with promPort
• The release, commit and build date information are baked into the release executables
• You can pick a subset of pollers to manage by passing the name of the poller to harvest. e.g. harvest start|stop|restart POLLERS