eth_subscribe in WS is not consistent between two nodes

[eduadiez]

System information

zkEVM Node version: v0.3.0
OS & Version: Linux
Network: Mainnet

RPC returns the latest block on both nodes but on WS it doesn't return the latest block. Checkout the screen shot below.

This issue was reported by quicknode

[sjoshi10]

thank you 🙏

[sjoshi10]

As soon as we restart zkemv-rpc, it catches up to the tip on WS subscription.

[NikitosnikN]

related logs in RPC component:

{"level":"error","ts":1696403623.5247061,"caller":"jsonrpc/server.go:356","msg":"Unable to read WS message, websocket: close 4000: No new messages within 60 seconds","pid":10,"version":"v0.3.1","stacktrace":"github.com/0xPolygonHermez/zkevm-node/jsonrpc.(*Server).handleWs\n\t/src/jsonrpc/server.go:356\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2109\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2487\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2947\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:1991"}

[tclemos]

Just to give it a heads-up, I'm fully focused and working on this at the moment.

I have made some experiments in order to identify the source of the problem, but unfortunately I wasn't able to reproduce it yet in a controlled environment. I'm continuing to investigate it and find different ways to reproduce the issue.

I've identified some points in the code that could be causing the issue and I'll be improving them.

[tclemos]

@sjoshi10

• Are both nodes running the same resources(CPU, RAM, Storage and Network)?
• Could you share with us the resource specification for these nodes?

[sjoshi10]

@tclemos
Both nodes are running same resources.

192 cores
1.5TB RAM
1x 3.5TB NVMe

[tclemos]

@tclemos Both nodes are running same resources.

192 cores
1.5TB RAM
1x 3.5TB NVMe

Holy moly! 😱

[lyh169]

latest block

Hi, Could you please tell me the reason that will lead to 'RPC returns the latest block on both nodes but on WS it doesn't return the latest block' in the old version?

[tclemos]

In the previous versions, the messages were sent sequentially considering all the connections, in case one connection was stuck for any reason it would delay all the other messages to be sent, creating a snowball effect accumulating messages. While the node was stuck waiting one of the connections to be able to write the message, the network was generating more blocks due to new transactions, the blocks were updated in the state due to the synchronizer, but the WS writing sequence was stuck.

In the new versions we changed this and now one connection shouldn't affect each other.