Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rafthttp: add Raft send latency metric for writes #10023

Closed
wants to merge 2 commits into from
Closed

rafthttp: add Raft send latency metric for writes #10023

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
dcb4d76
etcdserver/api/rafthttp: replace "Sub" call with "time.Since"
gyuho Aug 18, 2018
e07f1a2
etcdserver/api/rafthttp: add "etcd_network_raft_send_total_duration_s…
gyuho Aug 18, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
etcdserver/api/rafthttp: add "etcd_network_raft_send_total_duration_s… 
…econds" metric

Currently, only v2 metrics ("stats.FollowerStats") tracks Raft message
send latencies. Add Prometheus histogram to track Raft messages for
writes, since heartbeats are probed (see #10022)
and snapshots are already being tracked via #9997.

```
etcd_network_raft_send_total_duration_seconds_bucket{To="7339c4e5e833c029",Type="MsgProp",le="0.0001"} 1
etcd_network_raft_send_total_duration_seconds_bucket{To="7339c4e5e833c029",Type="MsgProp",le="0.0002"} 1
etcd_network_raft_send_total_duration_seconds_bucket{To="729934363faa4a24",Type="MsgApp",le="0.0001"} 9
etcd_network_raft_send_total_duration_seconds_bucket{To="729934363faa4a24",Type="MsgApp",le="0.0002"} 9
etcd_network_raft_send_total_duration_seconds_bucket{To="7339c4e5e833c029",Type="MsgAppResp",le="0.0001"} 8
etcd_network_raft_send_total_duration_seconds_bucket{To="7339c4e5e833c029",Type="MsgAppResp",le="0.0002"} 8
```

Signed-off-by: Gyuho Lee <[email protected]>
  • Loading branch information
@gyuho
gyuho committed Aug 29, 2018
commit e07f1a2daab01c51e452a8cd1abf1f62e5dfcd59
14 changes: 14 additions & 0 deletions etcdserver/api/rafthttp/metrics.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -133,6 +133,19 @@ var (
[]string{"From"},
)

raftSendSeconds = prometheus.NewHistogramVec(prometheus.HistogramOpts{
Namespace: "etcd",
Subsystem: "network",
Name: "raft_send_total_duration_seconds",
Help: "Total latency distributions of Raft message sends",

// lowest bucket start of upper bound 0.0001 sec (0.1 ms) with factor 2
// highest bucket start of 0.0001 sec * 2^15 == 3.2768 sec
Buckets: prometheus.ExponentialBuckets(0.0001, 2, 16),
},
[]string{"Type", "To"},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean []string{"To"}?

)

rttSec = prometheus.NewHistogramVec(prometheus.HistogramOpts{
Namespace: "etcd",
Subsystem: "network",
Expand Down Expand Up @@ -162,5 +175,6 @@ func init() {
prometheus.MustRegister(snapshotReceiveFailures)
prometheus.MustRegister(snapshotReceiveSeconds)

prometheus.MustRegister(raftSendSeconds)
prometheus.MustRegister(rttSec)
}
10 changes: 10 additions & 0 deletions etcdserver/api/rafthttp/stream.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -201,8 +201,10 @@ func (cw *streamWriter) run() {
heartbeatc, msgc = nil, nil

case m := <-msgc:
start := time.Now()
err := enc.encode(&m)
if err == nil {
took := time.Since(start)
unflushed += m.Size()

if len(msgc) == 0 || batched > streamBufSize/2 {
Expand All @@ -214,6 +216,14 @@ func (cw *streamWriter) run() {
batched++
}

// snapshot sends are tracked via separate metrics https://github.com/etcd-io/etcd/pull/9997
// heartbeats are tracked via prober https://github.com/etcd-io/etcd/pull/10022
// TODO: track other messages?
if m.Type == raftpb.MsgProp ||
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is not the sending time (flushing the buffer to the network card). we actually send the messages at line 211 when flush is called explicitly.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i would rather just remove this metrics. it is not easy to measure after all with buffers all over the places.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@xiang90 Good point. For now, tracking snapshots and heartbeats is good enough. Will come back to this later.

m.Type == raftpb.MsgApp ||
m.Type == raftpb.MsgAppResp {
raftSendSeconds.WithLabelValues(m.Type.String(), types.ID(m.To).String()).Observe(took.Seconds())
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason that MsgProp, MsgApp and MsgAppResp are chosen here? I was reading the doc, and you explained snapshot and heartbeats, how about other message types?

continue
}

Expand Down