Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Harvest should collect iwarp counters and include on Metrocluster dashboard #2068

Closed
cgrinds opened this issue May 10, 2023 · 3 comments · Fixed by #2096
Closed

Harvest should collect iwarp counters and include on Metrocluster dashboard #2068

cgrinds opened this issue May 10, 2023 · 3 comments · Fixed by #2096
Labels
23.08 customer feature New feature or request status/done

Comments

@cgrinds
Copy link
Collaborator

cgrinds commented May 10, 2023

Thanks to roller for raising on Discord.

bin/harvest zapi -p sar show counters --object iwarp | dasel -r xml -w json
{
  "root": {
    "counter-info": [
      {
        "desc": "Histogram for completion latency of RDMA read operations.",
        "is-deprecated": "false",
        "labels": {
          "label-info": "<2us,<6us,<10us,<14us,<20us,<40us,<60us,<80us,<100us,<200us,<400us,<600us,<800us,<1ms,<2ms,<4ms,<6ms,<8ms,<10ms,<12ms,<14ms,<16ms,<18ms,<20ms,<40ms,<60ms,<80ms,<100ms,<200ms,<400ms,<600ms,<800ms,<1s,<2s,<4s,<6s,<8s,<10s,<20s,>20s"
        },
        "name": "ic_compl_latency_histogram",
        "privilege-level": "advanced",
        "properties": "delta",
        "type": "array",
        "unit": "none"
      },
      {
        "desc": "Histogram for latency of RDMA I/O operations.",
        "is-deprecated": "false",
        "labels": {
          "label-info": "<2us,<6us,<10us,<14us,<20us,<40us,<60us,<80us,<100us,<200us,<400us,<600us,<800us,<1ms,<2ms,<4ms,<6ms,<8ms,<10ms,<12ms,<14ms,<16ms,<18ms,<20ms,<40ms,<60ms,<80ms,<100ms,<200ms,<400ms,<600ms,<800ms,<1s,<2s,<4s,<6s,<8s,<10s,<20s,>20s"
        },
        "name": "ic_latency_histogram",
        "privilege-level": "advanced",
        "properties": "delta",
        "type": "array",
        "unit": "none"
      },
      {
        "desc": "Histogram for size of scatter gather entries for RDMA I/O operations.",
        "is-deprecated": "false",
        "labels": {
          "label-info": "<=512B,<=1KB,<=2KB,<=4KB,<=8KB,<=16KB,<=32KB,<=64KB,<=128KB,<=256KB,<=512KB,<=1MB,<=2MB,<=4MB,<=8MB,>8MB"
        },
        "name": "ic_sge_size_histogram",
        "privilege-level": "advanced",
        "properties": "delta",
        "type": "array",
        "unit": "none"
      },
      {
        "desc": "Histogram for size of RDMA I/O operations.",
        "is-deprecated": "false",
        "labels": {
          "label-info": "<=512B,<=1KB,<=2KB,<=4KB,<=8KB,<=16KB,<=32KB,<=64KB,<=128KB,<=256KB,<=512KB,<=1MB,<=2MB,<=4MB,<=8MB,>8MB"
        },
        "name": "ic_size_histogram",
        "privilege-level": "advanced",
        "properties": "delta",
        "type": "array",
        "unit": "none"
      },
      {
        "desc": "Histogram for latency of RDMA write operations.",
        "is-deprecated": "false",
        "labels": {
          "label-info": "<2us,<6us,<10us,<14us,<20us,<40us,<60us,<80us,<100us,<200us,<400us,<600us,<800us,<1ms,<2ms,<4ms,<6ms,<8ms,<10ms,<12ms,<14ms,<16ms,<18ms,<20ms,<40ms,<60ms,<80ms,<100ms,<200ms,<400ms,<600ms,<800ms,<1s,<2s,<4s,<6s,<8s,<10s,<20s,>20s"
        },
        "name": "ic_wr_latency_histogram",
        "privilege-level": "advanced",
        "properties": "delta",
        "type": "array",
        "unit": "none"
      },
      {
        "desc": "Histogram for size of RDMA work requests.",
        "is-deprecated": "false",
        "labels": {
          "label-info": "<=512B,<=1KB,<=2KB,<=4KB,<=8KB,<=16KB,<=32KB,<=64KB,<=128KB,<=256KB,<=512KB,<=1MB,<=2MB,<=4MB,<=8MB,>8MB"
        },
        "name": "ic_wr_size_histogram",
        "privilege-level": "advanced",
        "properties": "delta",
        "type": "array",
        "unit": "none"
      },
      {
        "desc": "Instance Name",
        "is-deprecated": "false",
        "is-key": "true",
        "name": "instance_name",
        "privilege-level": "advanced",
        "properties": "string",
        "unit": "none"
      },
      {
        "desc": "Instance UUID",
        "is-deprecated": "false",
        "name": "instance_uuid",
        "privilege-level": "advanced",
        "properties": "string,no-display",
        "unit": "none"
      },
      {
        "base-counter": "iw_ops",
        "desc": "Average RDMA I/O latency.",
        "is-deprecated": "false",
        "name": "iw_avg_latency",
        "privilege-level": "advanced",
        "properties": "average",
        "unit": "microsec"
      },
      {
        "desc": "Number of RDMA I/Os issued.",
        "is-deprecated": "false",
        "name": "iw_ops",
        "privilege-level": "advanced",
        "properties": "rate",
        "unit": "none"
      },
      {
        "desc": "Number of RDMA read I/Os issued.",
        "is-deprecated": "false",
        "name": "iw_read_ops",
        "privilege-level": "advanced",
        "properties": "rate",
        "unit": "none"
      },
      {
        "desc": "Number of RDMA write I/Os issued.",
        "is-deprecated": "false",
        "name": "iw_write_ops",
        "privilege-level": "advanced",
        "properties": "rate",
        "unit": "none"
      },
      {
        "desc": "System node name",
        "is-deprecated": "false",
        "is-key": "true",
        "name": "node_name",
        "privilege-level": "advanced",
        "properties": "string",
        "unit": "none"
      },
      {
        "desc": "System node id",
        "is-deprecated": "false",
        "name": "node_uuid",
        "privilege-level": "advanced",
        "properties": "string,no-display",
        "unit": "none"
      },
      {
        "desc": "Ontap process that provided this instance",
        "is-deprecated": "false",
        "is-key": "true",
        "name": "process_name",
        "privilege-level": "diag",
        "properties": "string",
        "unit": "none"
      }
    ]
  }
}
@cgrinds cgrinds added feature New feature or request customer 23.08 labels May 10, 2023
@cgrinds
Copy link
Collaborator Author

cgrinds commented May 10, 2023 •
edited
Loading

Using template in pull request #2071

HARVEST_NO_COLLECTOR_UPGRADE=1 bin/poller --poller foople --promPort 12991 --config cbg/harvest.openlab.yml --collectors ZapiPerf --objects Iwarp

Output

2023-05-10T11:29:03-04:00 INF ./poller.go:205 > Init Poller=foople configPath=cbg/harvest.openlab.yml logLevel=info version="harvest version 23.05.0807-v23.05.0 (commit 7a247418) (build date 2023-05-08T07:54:53-0400) darwin/amd64\n"
2023-05-10T11:29:03-04:00 INF ./poller.go:242 > started in foreground Poller=foople pid=55810
2023-05-10T11:29:03-04:00 INF ./poller.go:1215 > Use ZAPIs HARVEST_NO_COLLECTOR_UPGRADE=1 Poller=foople collector=ZapiPerf preferZAPI=false v=9.12.1
2023-05-10T11:29:03-04:00 INF collector/helpers.go:139 > best-fit template Poller=foople collector=ZapiPerf:Iwarp path=conf/zapiperf/cdot/9.8.0/iwarp.yaml v=9.12.1
2023-05-10T11:29:03-04:00 INF ./poller.go:382 > Autosupport scheduled. Poller=foople asupSchedule=24h
2023-05-10T11:29:03-04:00 INF ./poller.go:391 > poller start-up complete Poller=foople
2023-05-10T11:29:03-04:00 INF prometheus/httpd.go:29 > http server listen Poller=foople addr=localhost exporter=prometheus1 port=12991
2023-05-10T11:29:03-04:00 INF ./poller.go:539 > updated status, up collectors: 1 (of 1), up exporters: 1 (of 1) Poller=foople
2023-05-10T11:29:04-04:00 INF collector/collector.go:484 > Collected Poller=foople apiMs=50 calcMs=0 collector=ZapiPerf:Iwarp instances=4 metrics=16 parseMs=1 pluginMs=0 skips=0
2023-05-10T11:30:04-04:00 INF collector/collector.go:484 > Collected Poller=foople apiMs=296 calcMs=0 collector=ZapiPerf:Iwarp instances=4 metrics=16 parseMs=0 pluginMs=0 skips=0

curl -s 'http://localhost:12991/metrics' | rg -v "#|metadata_" | sort

Results

iw_avg_latency{adapter="iWARP_0",cluster="a800-1",datacenter="openlab",node="a800-1-01"} 13.796523053665911
iw_avg_latency{adapter="iWARP_0",cluster="a800-1",datacenter="openlab",node="a800-1-02"} 13.945789364783778
iw_avg_latency{adapter="iWARP_1",cluster="a800-1",datacenter="openlab",node="a800-1-01"} 14.356561445304745
iw_avg_latency{adapter="iWARP_1",cluster="a800-1",datacenter="openlab",node="a800-1-02"} 14.123546312765084
iw_ops{adapter="iWARP_0",cluster="a800-1",datacenter="openlab",node="a800-1-01"} 109.79618510138128
iw_ops{adapter="iWARP_0",cluster="a800-1",datacenter="openlab",node="a800-1-02"} 80.21843727815204
iw_ops{adapter="iWARP_1",cluster="a800-1",datacenter="openlab",node="a800-1-01"} 167.20888415892895
iw_ops{adapter="iWARP_1",cluster="a800-1",datacenter="openlab",node="a800-1-02"} 121.31524065094419
iw_read_ops{adapter="iWARP_0",cluster="a800-1",datacenter="openlab",node="a800-1-01"} 13.709999832765071
iw_read_ops{adapter="iWARP_0",cluster="a800-1",datacenter="openlab",node="a800-1-02"} 6.124685155315147
iw_read_ops{adapter="iWARP_1",cluster="a800-1",datacenter="openlab",node="a800-1-01"} 4.000133123119107
iw_read_ops{adapter="iWARP_1",cluster="a800-1",datacenter="openlab",node="a800-1-02"} 4.000133123119107
iw_write_ops{adapter="iWARP_0",cluster="a800-1",datacenter="openlab",node="a800-1-01"} 114.79220198959229
iw_write_ops{adapter="iWARP_0",cluster="a800-1",datacenter="openlab",node="a800-1-02"} 85.23105222911458
iw_write_ops{adapter="iWARP_1",cluster="a800-1",datacenter="openlab",node="a800-1-01"} 172.20490104713997
iw_write_ops{adapter="iWARP_1",cluster="a800-1",datacenter="openlab",node="a800-1-02"} 126.32785560190672

@cgrinds
Copy link
Collaborator Author

cgrinds commented May 12, 2023

  • Add relevant panels to metrocluster dashboard in a new row

@Hardikl Hardikl linked a pull request May 22, 2023 that will close this issue
feat: collection of other counters for fcvi perf object #2096
Merged
@rahulguptajss
Copy link
Contributor

verified in 23.08. Openlab doesn't have fcvi counters. Validated via templates and dashboard panels.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
23.08 customer feature New feature or request status/done
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat: collection of other counters for fcvi perf object NetApp/harvest
3 participants
@cgrinds @rahulguptajss @Hardikl