Relay client not resilient against lost connection

[alichaddad]

Describe the bug

Currently we create an instance of the relay client in grid proxy and it is used to fetch node statistics. The client doesn't a logic to attempt a reconnect and handle lost connections, meaning that once a disconnect happens, all further requests to the statistics endpoint would fail.

To Reproduce

Steps to reproduce the behavior:

This can only happens when the relay connection is broken. When that happens going to that endpoint wouldn't work: https://gridproxy.dev.grid.tf/nodes/12/statistics

Expected behavior

There are two approaches here. Make the client more resistant to broken connection or ensure that client connection is valid in the proxy server.

Logs

From the proxy pod:

2023/04/26 09:28:09 http: panic serving 10.244.2.213:40192: send on closed channel
goroutine 824444 [running]:
net/http.(*conn).serve.func1()
	/usr/local/go/src/net/http/server.go:1854 +0xbf
panic({0xd45680, 0x107d620})
	/usr/local/go/src/runtime/panic.go:890 +0x263
github.com/threefoldtech/rmb-sdk-go/direct.(*DirectClient).request(0xc000ebed20, {0x1086688, 0xc001033980}, 0xc0000d89a0)
	/go/pkg/mod/github.com/threefoldtech/[email protected]/direct/direct.go:304 +0x1b9
github.com/threefoldtech/rmb-sdk-go/direct.(*DirectClient).Call(0xc000ebed20, {0x1086688, 0xc001033980}, 0x1414198?, {0xe63115, 0x12}, {0x0, 0x0}, {0xcd50c0, 0xc00016a2d0})
	/go/pkg/mod/github.com/threefoldtech/[email protected]/direct/direct.go:342 +0x1c9
github.com/threefoldtech/grid_proxy_server/internal/explorer.(*App).getNodeStatistics(0xc001716140, 0xc001068200)
	/src/internal/explorer/server.go:409 +0x1a5
github.com/threefoldtech/grid_proxy_server/internal/explorer/mw.AsHandlerFunc.func1({0x1085cd0, 0xc000d528c0}, 0xc001068200)
	/src/internal/explorer/mw/action.go:113 +0xc7
net/http.HandlerFunc.ServeHTTP(0xc001068100?, {0x1085cd0?, 0xc000d528c0?}, 0xc0010509e8?)
	/usr/local/go/src/net/http/server.go:2122 +0x2f
github.com/gorilla/mux.(*Router).ServeHTTP(0xc0014a4240, {0x1085cd0, 0xc000d528c0}, 0xc001068000)
	/go/pkg/mod/github.com/gorilla/[email protected]/mux.go:210 +0x1cf
net/http.serverHandler.ServeHTTP({0xc001033650?}, {0x1085cd0, 0xc000d528c0}, 0xc001068000)
	/usr/local/go/src/net/http/server.go:2936 +0x316
net/http.(*conn).serve(0xc00086e7e0, {0x1086688, 0xc0008a37d0})
	/usr/local/go/src/net/http/server.go:1995 +0x612
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:3089 +0x5ed

[xmonader]

@muhamadazmy

[mohamedamer453]

Verified on devnet.

i was able to access and check the statistics endpoint with multiple nodes and it worked without any issues and over the last couple of days i was able to access this endpoint and it worked with me every time without any issues.