Add counters for netstat object for packet loss

[MrObvious]

This is a consideration to be asked for. The netstat object is a counter manager version of the netstat -anCEWcT command in ONTAP 9.5 and higher, and tracks things like per-TCP connection out-of-order recieves and zero windows. https://kb.netapp.com/Advice_and_Troubleshooting/Data_Storage_Software/ONTAP_OS/How_to_use_netstat_to_troubleshoot_network_problems_in_ONTAP_9.5_or_newer has more info.

I could see a dashboard being useful for customers wanting to track packet loss on TCP sessions. Most of these counters wouldn't be useful but ooorcv and rexmit_pkts would be I think:
pstejska_vsim::*> statistics catalog counter show -object netstat

Object: netstat
Counter Description
--------------------------- ----------------------------------------------
bytes_recvd Number of bytes received by a TCP connection
bytes_sent Number of bytes sent by a TCP connection
cc_algo TCP congestion control name
cg_class CG traffic class of a connection
cg_id CG ID used for the connection
cgtc_enabled CG traffic classification enabled
col Column ID of CG table used for a TCP
Connection
cong_win Congestion window of a TCP connection
cong_win_th Congestion window threshold of a TCP
connection
ecn TCP Explicit Congestion Notification (ECN)
status
faddr Foreign IP address of a TCP connection
fport_hbo Foreign layer-4 port of a TCP connection
hash Hash corresponding to a TCP connection
instance_name Instance Name
instance_uuid Instance UUID
laddr Local IP address of a TCP connection
lport_hbo Local layer-4 port of a TCP connection
mss Maximum segment size of a TCP connection
node_name System node name
node_uuid System node id
ooorcv_pkts Number of out-of-order packets received by
this TCP connection
process_name Ontap process that provided this instance
recv_hiwa Hi-water mark of socket receive buffer
recv_q Receive queue size of TCP connections
established
recv_window Receive window size of a TCP connection
req_cg_id Requested CG ID used for the connection by
application
rexmit_pkts Number of packets retransmitted by this TCP
connection
row Row ID of CG table used for a TCP Connection
send_hiwa Hi-water mark for socket send buffer
send_q Send queue size of TCP connections
send_window Send window size of a TCP connection
tc_weight CG traffic class weight of a connection

[cgrinds]

Thanks for the suggestion and KB article @MrObvious

The following template includes the metrics from the article. Does this include what you need? Any thoughts on which labels should be exported with each metric. The template below creates metrics that look like this.

2022-11-01T09:18:02-04:00 INF collector/collector.go:482 > Collected Poller=aff-250 apiMs=1043 calcMs=0 collector=ZapiPerf:Netstat instances=49 metrics=392 parseMs=19 pluginMs=0 skips=0

netstat_bytes_sent{cluster="A250-15-28-29",datacenter="openlab",faddr="169.254.141.82",fport="59359",laddr="169.254.48.240",lport="7700",node="A250-15-28-29-01"} 658878424487
netstat_ooorcv_pkts{cluster="A250-15-28-29",datacenter="openlab",faddr="10.195.15.78",fport="861",laddr="10.195.15.32",lport="2049",node="A250-15-28-29-02"} 729
netstat_rexmit_pkts{cluster="A250-15-28-29",datacenter="openlab",faddr="169.254.48.240",fport="7700",laddr="169.254.141.82",lport="59355",node="A250-15-28-29-02"} 5

• Do you think including faddr, fport, laddr, lport individually is useful or would it be better to join faddr:fport into something like this?

netstat_send_window{cluster="A250-15-28-29",datacenter="openlab",faddr="169.254.48.240_7700",laddr="169.254.141.82_19614",node="A250-15-28-29-02"} 262144

• Are there other labels you want included?

Template

name:                     Netstat
query:                    netstat
object:                   netstat

instance_key:             uuid

counters:
  - bytes_recvd
  - bytes_sent
  - cong_win
  - cong_win_th
  - faddr
  - fport_hbo                => fport
  - instance_uuid
  - laddr
  - lport_hbo                => lport
  - node_name                => node
  - ooorcv_pkts
  - recv_window
  - rexmit_pkts
  - send_window

override:
  lport_hbo: string
  fport_hbo: string

export_options:
  instance_keys:
    - node
    - faddr
    - laddr
    - fport
    - lport

[MrObvious]

Do you think including faddr, fport, laddr, lport individually is useful or would it be better to join faddr:fport into something like this?

I think joining faddr:fport together works better. Thanks for putting that together Chris. It looks great!

[cgrinds]

Thanks! Fixed in #1423

My guess is this probably makes sense to be commented out by default and only enabled when needed. Do you agree?

I also noticed REST Perf does not include this object.

curl -k -n 'https://10.195.15.41/api/cluster/counter/tables/netstat?return_records=true&fields=*'
{
  "error": {
    "message": "Object \"netstat\" was not found.",
    "code": "8585320",
    "target": "name"
  }
}

Would you mind opening an ONTAP burt to have this object/metrics included?

[rahulguptajss]

verfied in 22.11