Track closed connections and reason for closing #692
Conversation
Signed-off-by: Derek Collison <[email protected]>
| return ConnOpen, nil | ||
| } | ||
| switch strings.ToLower(str) { | ||
| case "open": |
There was a problem hiding this comment.
should this be opened since you have closed (or add both open, opened and close, closed.
There was a problem hiding this comment.
I think open is correct since it means perpetual vs closed.
| func (reason ClosedState) String() string { | ||
| switch reason { | ||
| case ClientClosed: | ||
| return "Client" |
There was a problem hiding this comment.
Should this be Closed instead or Client Closed?
There was a problem hiding this comment.
I had it as Client Closed, but thought it read better looking at the JSON as reason for closed is Client.
|
|
||
| // Fixed sized ringbuffer for closed connections. | ||
| type closedRingBuffer struct { | ||
| total uint64 |
There was a problem hiding this comment.
Would that make more sense to add a mutex in that object and so we don't need server lock for that? When I create a new object like that, I tend to favor that object having its own lock so we can reduce lock contention. Especially this is a singleton, so adding a mutex should not be costly mem wise.
There was a problem hiding this comment.
I thought about that too, but decided its very fast and will only hold for time to copy. I was thinking about preallocating the structs too. I could put a lock around it.
| @@ -834,6 +838,35 @@ func (s *Server) createClient(conn net.Conn) *client { | |||
| return c | |||
| } | |||
|
|
|||
| // This will save off a closed client in a ring buffer such that | |||
| // /connz can inspect. Useful for debugging, etc. | |||
| func (s *Server) saveClosedClient(c *client, nc net.Conn, reason ClosedState) { | |||
There was a problem hiding this comment.
This could be a receiving of the ring buffer object (especially if we use a dedicated lock there)
There was a problem hiding this comment.
Yes it could. Feels like a server verb though that just uses the ring buffer. We do the time stamp etc.
| ci.InMsgs = atomic.LoadInt64(&client.inMsgs) | ||
| ci.InBytes = atomic.LoadInt64(&client.inBytes) | ||
|
|
||
| // If the connection is gone, too bad, we won't set TLSVersion and TLSCipher. |
There was a problem hiding this comment.
I believe you should use nc down below, not client.nc...
server/server.go
Outdated
| @@ -786,7 +790,7 @@ func (s *Server) createClient(conn net.Conn) *client { | |||
| if err := conn.Handshake(); err != nil { | |||
| c.Errorf("TLS handshake error: %v", err) | |||
| c.sendErr("Secure Connection - TLS Required") | |||
| c.closeConnection() | |||
| c.closeConnection(TLSHandshakeError) | |||
There was a problem hiding this comment.
Humm... I tried to connect a non TLS client to a TLS server and I get Write Error, not TLS Handshake Error. I believe that since we call sendErr(), it is possible that flushOutbound() gets an error because the client will also close the connection (after receiving the error), so the actual close of the connection occurs there, with WriteError. That may then not prove that useful.
There was a problem hiding this comment.
Non TLS client won't try to handshake will it? I will look into it though, agree we want the correct reason.
There was a problem hiding this comment.
No they won't.
Actually, what happened is that I was connecting without TLS, so the client when getting the INFO fails the connect and says that secure connection is required. The server when getting into the Handshake will fail and send the error, but since the connection is closed it will get a write error, so that kind of makes sense.
But, I just tried with a client connection with TLS but without the CA root cert so it won't be able to verify the server certificate, and I still get the Write Error as a reason.
There was a problem hiding this comment.
ok I will fix and add a test, thanks.
|
I also just found a bug with closed connections and optional Subs (and AuthorizedUser). Once set it stays set, will fix in the am. |
Signed-off-by: Derek Collison <[email protected]>
Signed-off-by: Derek Collison <[email protected]>
Signed-off-by: Derek Collison <[email protected]>
Signed-off-by: Derek Collison <[email protected]>
Signed-off-by: Derek Collison <[email protected]>
Signed-off-by: Derek Collison <[email protected]>
Signed-off-by: Derek Collison <[email protected]>
server/closed_conns_test.go
Outdated
| @@ -267,7 +267,7 @@ func TestClosedMaxPayload(t *testing.T) { | |||
|
|
|||
| func TestClosedSlowConsumerWriteDeadline(t *testing.T) { | |||
| opts := DefaultOptions() | |||
| opts.WriteDeadline = 10 * time.Microsecond // Make very small to trip. | |||
| opts.WriteDeadline = 10 * time.Millisecond // Make very small to trip. | |||
There was a problem hiding this comment.
Probably not enough. Bumping it to 30ms in TestWriteDeadline for very same reason.
There was a problem hiding this comment.
(it was originally at 1 ms though), so that may be ok...
There was a problem hiding this comment.
Seems to pass on travis..
There was a problem hiding this comment.
But we can change it..
|
LGTM |
This PR allows the server to maintain state on closed connections. Uses a fixed size ring buffer that is configurable. Monitoring endpoints now can return closed connections via /connz?state=closed and will report when they closed and why.
Adresses #520 and #663
Signed-off-by: Derek Collison [email protected]
/cc @nats-io/core