Skip to content

Commit

Permalink
Procstat: collect connections and listener for each proc
Browse files Browse the repository at this point in the history
Procstat modified to add a new metric "procstat_tcp" with to values
"conn" and "listen".
It also adds number of connections for each TCP state.

Those metrics will be added only if it they are activated specifically.

Those values will contain a comma separated value of the endpoints where
the proc is connecting to or listening (IPv4 and IPv6).
Only for linux.

Local, virtual and docker interfaces are ignored.

If some proc is listening on 0.0.0.0, an endpoint for each of the
"public" (those not ignored as internal) IPv4 IPs is created.
If it is listening on :: (IPv6) an endpoint is created for each IPv4 and
IPv6.

For programs with one parent and several children, all listening in some
endpoint, only the parent process is taken into account. Child endpoints
are ignored.

Connections made to this cost (local port is one of the listening ports)
are ignored. To avoid having servers with thousands of connections.
We prefer to collect that info in the clients.

It is also added connection info (number of connections in each of the
TCP states) for each proc. Improving PR #5402

To gather tcp connections netlink is used, to avoid the cost of parsing
/proc/net/tcp(6), but /proc should be readed to get the mapping between
inodes and pids.
  • Loading branch information
adrianlzt committed Jan 27, 2021
1 parent c237989 commit 4a0d64e
Show file tree
Hide file tree
Showing 13 changed files with 1,245 additions and 17 deletions.
1 change: 1 addition & 0 deletions docs/LICENSE_OF_DEPENDENCIES.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ following works:
- github.com/eapache/go-xerial-snappy [MIT License](https://github.com/eapache/go-xerial-snappy/blob/master/LICENSE)
- github.com/eapache/queue [MIT License](https://github.com/eapache/queue/blob/master/LICENSE)
- github.com/eclipse/paho.mqtt.golang [Eclipse Public License - v 1.0](https://github.com/eclipse/paho.mqtt.golang/blob/master/LICENSE)
- github.com/elastic/gosigar [Apache License 2.0](https://github.com/elastic/gosigar/blob/master/LICENSE)
- github.com/ericchiang/k8s [Apache License 2.0](https://github.com/ericchiang/k8s/blob/master/LICENSE)
- github.com/ghodss/yaml [MIT License](https://github.com/ghodss/yaml/blob/master/LICENSE)
- github.com/go-logfmt/logfmt [MIT License](https://github.com/go-logfmt/logfmt/blob/master/LICENSE)
Expand Down
1 change: 1 addition & 0 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ require (
github.com/docker/go-units v0.3.3 // indirect
github.com/docker/libnetwork v0.8.0-dev.2.0.20181012153825-d7b61745d166
github.com/eclipse/paho.mqtt.golang v1.2.0
github.com/elastic/gosigar v0.13.0
github.com/ericchiang/k8s v1.2.0
github.com/ghodss/yaml v1.0.1-0.20190212211648-25d852aebe32
github.com/go-logfmt/logfmt v0.4.0
Expand Down
3 changes: 3 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -188,6 +188,8 @@ github.com/eapache/queue v1.1.0 h1:YOEu7KNc61ntiQlcEeUIoDTJ2o8mQznoNvUhiigpIqc=
github.com/eapache/queue v1.1.0/go.mod h1:6eCeP0CKFpHLu8blIFXhExK/dRa7WDZfr6jVFPTqq+I=
github.com/eclipse/paho.mqtt.golang v1.2.0 h1:1F8mhG9+aO5/xpdtFkW4SxOJB67ukuDC3t2y2qayIX0=
github.com/eclipse/paho.mqtt.golang v1.2.0/go.mod h1:H9keYFcgq3Qr5OUJm/JZI/i6U7joQ8SYLhZwfeOo6Ts=
github.com/elastic/gosigar v0.13.0 h1:EIeuQcLPKia759s6mlVztlxUyKiKYHo6y6kOODOLO7A=
github.com/elastic/gosigar v0.13.0/go.mod h1:iXRIGg2tLnu7LBdpqzyQfGDEidKCfWcCMS0WKyPWoMs=
github.com/elazarl/goproxy v0.0.0-20170405201442-c4fc26588b6e/go.mod h1:/Zj4wYkgs4iZTTu3o/KG3Itv/qCCa8VVMlb3i9OVuzc=
github.com/emicklei/go-restful v0.0.0-20170410110728-ff4f55a20633/go.mod h1:otzb+WCGbkyDHkqmQmT5YD2WR4BBwUdeQoFo8l/7tVs=
github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
Expand Down Expand Up @@ -729,6 +731,7 @@ golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJ
golang.org/x/sync v0.0.0-20200317015054-43a5402ce75a h1:WXEvlFVvvGxCJLG6REjsT03iWnKLEWinaScsxF2Vm2o=
golang.org/x/sync v0.0.0-20200317015054-43a5402ce75a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sys v0.0.0-20170830134202-bb24a47a89ea/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20180810173357-98c5dad5d1a0/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
Expand Down
44 changes: 44 additions & 0 deletions plugins/inputs/procstat/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,12 @@ Processes can be selected for monitoring using one of several methods:
## the native finder performs the search directly in a manor dependent on the
## platform. Default is 'pgrep'
# pid_finder = "pgrep"

## Select wich extra metrics should be added:
## - "connections_stats": tcp_* and upd_socket metrics
## - "connections_endpoints": new metric procstat_tcp with connections and listeners endpoints
## Default is empty list.
# metrics_include = ["connections_stats", "connections_endpoints"]
```

#### Windows support
Expand Down Expand Up @@ -157,11 +163,49 @@ the `win_perf_counters` input plugin as a more mature alternative.
- running (int)
- result_code (int, success = 0, lookup_error = 1)

If ``connections_stats`` enabled, added fields:
- procstat
- fields:
- tcp_close (int)
- tcp_close_wait (int)
- tcp_closing (int)
- tcp_established (int)
- tcp_fin_wait1 (int)
- tcp_fin_wait2 (int)
- tcp_last_ack (int)
- tcp_listen (int)
- tcp_none (int)
- tcp_syn_recv (int)
- tcp_syn_sent (int)

If ``connections_endpoints`` enabled, added fields:
- procstat_tcp
- tags:
- pid (when `pid_tag` is true)
- cmdline (when 'cmdline_tag' is true)
- process_name
- pidfile (when defined)
- exe (when defined)
- pattern (when defined)
- user (when selected)
- systemd_unit (when defined)
- cgroup (when defined)
- fields:
- conn (string)
- listen (string)

To gather connection info, if Telegraf is not run as root, it needs the following capabilities
```
sudo setcap "CAP_DAC_READ_SEARCH,CAP_SYS_PTRACE+ep" telegraf
```

*NOTE: Resource limit > 2147483647 will be reported as 2147483647.*

### Example Output:

```
procstat_lookup,host=prash-laptop,pattern=influxd,pid_finder=pgrep,result=success pid_count=1i,running=1i,result_code=0i 1582089700000000000
procstat,host=prash-laptop,pattern=influxd,process_name=influxd,user=root involuntary_context_switches=151496i,child_minor_faults=1061i,child_major_faults=8i,cpu_time_user=2564.81,cpu_time_idle=0,cpu_time_irq=0,cpu_time_guest=0,pid=32025i,major_faults=8609i,created_at=1580107536000000000i,voluntary_context_switches=1058996i,cpu_time_system=616.98,cpu_time_steal=0,cpu_time_guest_nice=0,memory_swap=0i,memory_locked=0i,memory_usage=1.7797634601593018,num_threads=18i,cpu_time_nice=0,cpu_time_iowait=0,cpu_time_soft_irq=0,memory_rss=148643840i,memory_vms=1435688960i,memory_data=0i,memory_stack=0i,minor_faults=1856550i 1582089700000000000
procstat,host=laptop,pattern=httpd,process_name=httpd,user=root child_major_faults=0i,child_minor_faults=70i,cpu_time=0i,cpu_time_guest=0,cpu_time_guest_nice=0,cpu_time_idle=0,cpu_time_iowait=0,cpu_time_irq=0,cpu_time_nice=0,cpu_time_soft_irq=0,cpu_time_steal=0,cpu_time_system=0.01,cpu_time_user=0.02,cpu_usage=0,created_at=1611738400000000000i,involuntary_context_switches=15i,listen=1i,major_faults=0i,memory_data=999424i,memory_locked=0i,memory_rss=4677632i,memory_stack=135168i,memory_swap=0i,memory_usage=0.013990458101034164,memory_vms=6078464i,minor_faults=1636i,nice_priority=20i,num_fds=8i,num_threads=1i,pid=1738811i,read_bytes=0i,read_count=4397i,realtime_priority=0i,rlimit_cpu_time_hard=2147483647i,rlimit_cpu_time_soft=2147483647i,rlimit_file_locks_hard=2147483647i,rlimit_file_locks_soft=2147483647i,rlimit_memory_data_hard=2147483647i,rlimit_memory_data_soft=2147483647i,rlimit_memory_locked_hard=65536i,rlimit_memory_locked_soft=65536i,rlimit_memory_rss_hard=2147483647i,rlimit_memory_rss_soft=2147483647i,rlimit_memory_stack_hard=2147483647i,rlimit_memory_stack_soft=8388608i,rlimit_memory_vms_hard=2147483647i,rlimit_memory_vms_soft=2147483647i,rlimit_nice_priority_hard=0i,rlimit_nice_priority_soft=0i,rlimit_num_fds_hard=1048576i,rlimit_num_fds_soft=1048576i,rlimit_realtime_priority_hard=0i,rlimit_realtime_priority_soft=0i,rlimit_signals_pending_hard=127473i,rlimit_signals_pending_soft=127473i,signals_pending=0i,tcp_close=0i,tcp_close_wait=0i,tcp_closing=0i,tcp_established=0i,tcp_fin_wait1=0i,tcp_fin_wait2=0i,tcp_last_ack=0i,tcp_listen=1i,tcp_syn_recv=0i,tcp_syn_sent=0i,voluntary_context_switches=169i,write_bytes=53248i,write_count=10i 1611738522000000000
procstat_tcp,host=laptop,pattern=httpd,process_name=httpd,user=root conn="",listen="192.168.1.35:80,192.168.1.48:80,[da01:beef:234:3830:aeda:f001:a00c:0091]:80,[aa01:beef:234:3830:e8e:0000:000a:6b0f]:80" 1611738522000000000
```
38 changes: 38 additions & 0 deletions plugins/inputs/procstat/connections.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
package procstat

import (
"fmt"
"net"
)

const (
// DockerMACPrefix https://macaddress.io/faq/how-to-recognise-a-docker-container-by-its-mac-address
DockerMACPrefix = "02:42"
// VirtualBoxMACPrefix https://github.com/mdaniel/virtualbox-org-svn-vbox-trunk/blob/2d259f948bc352ee400f9fd41c4a08710cd9138a/src/VBox/HostDrivers/VBoxNetAdp/VBoxNetAdp.c#L93
VirtualBoxMACPrefix = "0a:00:27"
// HardwareAddrLength is the number of bytes of a MAC address
HardwareAddrLength = 6
)

var (
// ErrorPIDNotFound is the error generated when the pid does not have network info
ErrorPIDNotFound = fmt.Errorf("pid not found")
)

// InodeInfo represents information of a proc associated with an inode
type InodeInfo struct {
pid uint32
ppid uint32
}

// NetworkInfo implements NetworkInfo using the netlink calls and parsing /proc to map sockets to PIDs
type NetworkInfo struct {
// tcp contains the connection info for each pid
tcp map[uint32][]ConnInfo
// listenPorts is a map with the listen ports in the host, used to ignore inbound connections
listenPorts map[uint32]interface{}
// publicIPs list of IPs considered "public" (used to connect to other hosts)
publicIPs []net.IP
// privateIPs list of IPs considered "private" (loopback, virtual interfaces, point2point, etc)
privateIPs []net.IP
}
35 changes: 35 additions & 0 deletions plugins/inputs/procstat/connections_fallback.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
// +build !linux

package procstat

import (
"fmt"
"net"
)

type ConnInfo struct {
}

func (n *NetworkInfo) IsAListenPort(port uint32) bool {
return false
}

func (n *NetworkInfo) Fetch() error {
return fmt.Errorf("platform not supported")
}

func (n *NetworkInfo) GetConnectionsByPid(pid uint32) (conn []ConnInfo, err error) {
return conn, fmt.Errorf("platform not supported")
}

func (n *NetworkInfo) GetPublicIPs() []net.IP {
return []net.IP{}
}

func (n *NetworkInfo) GetPrivateIPs() []net.IP {
return []net.IP{}
}

func (n *NetworkInfo) IsPidListeningInAddr(pid uint32, ip net.IP, port uint32) bool {
return false
}
Loading

0 comments on commit 4a0d64e

Please sign in to comment.