Add metrics for external RPC calls (success / error)

[CryptoFewka]

Currently zetaclientd seems to panic when the external RPC returns an error response. Exposing success/failure counts by network as a metric would be very useful for monitoring.

Panic details:

Jan 03 21:26:39 validator0-eu-west-1-testnet zetaclientd[678456]: {"level":"info","chain":"ZetaChain","time":"2024-01-03T21:26:39Z","message":"starting core observer"}
Jan 03 21:26:39 validator0-eu-west-1-testnet zetaclientd[678456]: {"level":"info","chain":"bsc_testnet","module":"ExternalChainWatcher","time":"2024-01-03T21:26:39Z","message":"ExternalChainWatcher for inboundTrackerSuggestions started"}
Jan 03 21:26:39 validator0-eu-west-1-testnet zetaclientd[678456]: {"level":"info","chain":"bsc_testnet","module":"ObserveOutTx","time":"2024-01-03T21:26:39Z","message":"observeOutTx using timeoutNonce 300 seconds, rpcRestTime 20 ms"}
Jan 03 21:26:39 validator0-eu-west-1-testnet zetaclientd[678456]: {"level":"info","chain":"bsc_testnet","module":"ExternalChainWatcher","time":"2024-01-03T21:26:39Z","message":"ExternalChainWatcher started"}
Jan 03 21:26:39 validator0-eu-west-1-testnet zetaclientd[678456]: {"level":"info","chain":"mumbai_testnet","module":"ExternalChainWatcher","time":"2024-01-03T21:26:39Z","message":"ExternalChainWatcher for inboundTrackerSuggestions started"}
Jan 03 21:26:39 validator0-eu-west-1-testnet zetaclientd[678456]: {"level":"info","chain":"mumbai_testnet","module":"ExternalChainWatcher","time":"2024-01-03T21:26:39Z","message":"ExternalChainWatcher started"}
Jan 03 21:26:39 validator0-eu-west-1-testnet zetaclientd[678456]: {"level":"info","chain":"goerli_testnet","module":"ExternalChainWatcher","time":"2024-01-03T21:26:39Z","message":"ExternalChainWatcher started"}
Jan 03 21:26:39 validator0-eu-west-1-testnet zetaclientd[678456]: {"level":"info","chain":"mumbai_testnet","module":"ObserveOutTx","time":"2024-01-03T21:26:39Z","message":"observeOutTx using timeoutNonce 300 seconds, rpcRestTime 20 ms"}
Jan 03 21:26:39 validator0-eu-west-1-testnet zetaclientd[678456]: {"level":"info","chain":"goerli_testnet","module":"ExternalChainWatcher","time":"2024-01-03T21:26:39Z","message":"ExternalChainWatcher for inboundTrackerSuggestions started"}
Jan 03 21:26:39 validator0-eu-west-1-testnet zetaclientd[678456]: {"level":"info","chain":"goerli_testnet","module":"ObserveOutTx","time":"2024-01-03T21:26:39Z","message":"observeOutTx using timeoutNonce 300 seconds, rpcRestTime 20 ms"}
Jan 03 21:26:39 validator0-eu-west-1-testnet zetaclientd[678456]: {"level":"info","chain":"ZetaChain","module":"ZetaChainWatcher","time":"2024-01-03T21:26:39Z","message":"Starting Send Scheduler for zeta1euvlmpffq7gl77vlwlv97kn004zq6rdw0skdd3"}
Jan 03 21:26:39 validator0-eu-west-1-testnet zetaclientd[678456]: {"level":"info","module":"startup","time":"2024-01-03T21:26:39Z","message":"awaiting the os.Interrupt, syscall.SIGTERM signals..."}
Jan 03 21:26:39 validator0-eu-west-1-testnet zetaclientd[678456]: {"level":"error","chain":"goerli_testnet","module":"WatchGasPrice","error":"400 Bad Request: {\"error\": \"request has an invalid host info\",\"id\":1}","time":"2024-01-03T21:26:39Z","message":"Err SuggestGasPrice:"}
Jan 03 21:26:39 validator0-eu-west-1-testnet zetaclientd[678456]: block height: 3059406
Jan 03 21:26:39 validator0-eu-west-1-testnet zetaclientd[678456]: {"level":"error","chain":"goerli_testnet","module":"WatchGasPrice","time":"2024-01-03T21:26:39Z","message":"PostGasPrice error at zeta block : 3059406  "}
Jan 03 21:26:41 validator0-eu-west-1-testnet zetaclientd[678456]: {"level":"info","chain":"mumbai_testnet","module":"ExternalChainWatcher","time":"2024-01-03T21:26:41Z","message":"checking tracker with hash :0xc378b09ea33bfa34b19ca198b45a51596d8d50efddc883887ed41714cfdde9fd and coin-type :Gas "}
Jan 03 21:26:41 validator0-eu-west-1-testnet zetaclientd[678456]: {"level":"info","chain":"mumbai_testnet","module":"ExternalChainWatcher","time":"2024-01-03T21:26:41Z","message":"Checking for all inTX : startBlock 44358778, toBlock 44358877"}
Jan 03 21:26:41 validator0-eu-west-1-testnet zetaclientd[678456]: {"level":"info","chain":"mumbai_testnet","module":"ExternalChainWatcher","time":"2024-01-03T21:26:41Z","message":"TSS inTx detected: 0xc378b09ea33bfa34b19ca198b45a51596d8d50efddc883887ed41714cfdde9fd, blocknum 44303774"}
Jan 03 21:26:41 validator0-eu-west-1-testnet zetaclientd[678456]: {"level":"info","chain":"mumbai_testnet","module":"ExternalChainWatcher","time":"2024-01-03T21:26:41Z","message":"TSS inTx value: 4000000000000000000"}
Jan 03 21:26:41 validator0-eu-west-1-testnet zetaclientd[678456]: {"level":"info","chain":"mumbai_testnet","module":"ExternalChainWatcher","time":"2024-01-03T21:26:41Z","message":"TSS inTx from: 0xe855E2dab793Dec569Dfad9edfc9f2EaBf9Acf6c"}
Jan 03 21:26:41 validator0-eu-west-1-testnet zetaclientd[678456]: {"level":"info","chain":"mumbai_testnet","module":"ExternalChainWatcher","time":"2024-01-03T21:26:41Z","message":"Gas Deposit detected and reported: PostSend zeta tx: 6BBA4A83124CE5F2317BD78461761B7BFCB490495DF06D895C26BA8F15C619DB"}
Jan 03 21:26:41 validator0-eu-west-1-testnet zetaclientd[678456]: {"level":"info","chain":"mumbai_testnet","module":"ExternalChainWatcher","time":"2024-01-03T21:26:41Z","message":"Vote submitted for inbound Tracker,Chain : mumbai_testnet,Ballot Identifier : 0xc9c563f1f49ab42550339492bc96b20a70dcb959fea008f74fe0de01ceff0c6f, coin-type Gas"}
Jan 03 21:26:42 validator0-eu-west-1-testnet zetaclientd[678456]: {"level":"info","chain":"ZetaChain","module":"OutTxProcessorManager","time":"2024-01-03T21:26:42Z","message":"StartTryProcess 0x9c915eba4d041aeb78b930c909e4fb1f576ed358132dabc0921c587b96f29437-5-215827, numActiveProcessor 1"}
Jan 03 21:26:42 validator0-eu-west-1-testnet zetaclientd[678456]: panic: runtime error: invalid memory address or nil pointer dereference
Jan 03 21:26:42 validator0-eu-west-1-testnet zetaclientd[678456]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x2027c70]
Jan 03 21:26:42 validator0-eu-west-1-testnet zetaclientd[678456]: goroutine 820 [running]:
Jan 03 21:26:42 validator0-eu-west-1-testnet zetaclientd[678456]: github.com/zeta-chain/zetacore/zetaclient.(*CoreObserver).startSendScheduler(0x4001b60200)
Jan 03 21:26:42 validator0-eu-west-1-testnet zetaclientd[678456]:         /go/src/github.com/zeta-chain/node/zetaclient/zetacore_observer.go:245 +0xdf0
Jan 03 21:26:42 validator0-eu-west-1-testnet zetaclientd[678456]: created by github.com/zeta-chain/zetacore/zetaclient.(*CoreObserver).MonitorCore
Jan 03 21:26:42 validator0-eu-west-1-testnet zetaclientd[678456]:         /go/src/github.com/zeta-chain/node/zetaclient/zetacore_observer.go:91 +0xe0
Jan 03 21:26:42 validator0-eu-west-1-testnet systemd[1]: zetaclientd.service: Main process exited, code=exited, status=2/INVALIDARGUMENT

[brewmaster012]

This only happens upon startup when an incorrect RPC URL is set correct?

[CharlieMc0]

The panic only happens at start if the RPC is not working.

It would helpful to see how many are successful or failing at all times and could identify issues with RPC connectivity.