Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WebSocketProvider handle ws close and reconnect #1053

Open
PierreJeanjacquot opened this issue Sep 18, 2020 · 77 comments
Open

WebSocketProvider handle ws close and reconnect #1053

PierreJeanjacquot opened this issue Sep 18, 2020 · 77 comments
Labels
enhancement New feature or improvement.

Comments

@PierreJeanjacquot
Copy link

PierreJeanjacquot commented Sep 18, 2020

Hi @ricmoo,

I'm using WebSocketProvider server-side to listen to blockchain events and performing calls to smart contracts.
Sometimes the websocket pipe got broken and I need to reconnect it.

I use this code to detect ws close and reconnect but it would be nice to not have to rely on _websocket to do it:

let wsProvider;

init = async () => {
  wsProvider = new ethers.providers.WebSocketProvider(wsHost);
  wsProvider._websocket.on('close', async (code) => {
    console.log('ws closed', code);
    wsProvider._websocket.terminate();
    await sleep(3000); // wait before reconnect
    init();
  });
  wsProvider.on('block', doStuff);
};

I also noticed when the websocket is broken Promise call don't reject wich is not super intuitive.

@ricmoo
Copy link
Member

ricmoo commented Oct 23, 2020

This is a very large feature... When I first (begrudgingly) added WebSocketProvider mentioned this would be something I would eventually get to, but that it won't be high priority any time soon. :)

But I want to! :)

It is still on the backlog, and I'll use this issue to track it, but there are other things I need to work on first.

Keep in mind when you reconnect, you may have been disconnected for a long time, in which case you should find and trigger events that were missed; you may have also been down fo a short period of time, in which case you must dedup events you've already emitted. Also, earlier events should be emitted before later ones. Unless there was a re-org, exactly-once semantics should be adhered to. All subscriptions will need some custom logic, depending on the type of subscription to handle this.

Also ethers providers guarantee consistent read-after-events. So, if a block number X has been emitted, a call to getBlock(X) must return a block. In many cases, due to the distributed nature of the Blockchain, especially with a FallbackProvider, one backend may have seen a block before others, so calling getBlock might occur on a node before it has actually seen the block, so the call must stall and (with exponential back-off) poll for the block and resolve it when it comes in. Similarly, this is true for events which include the transactionHash; a call to getTransaction must succeed, stalling until the data becomes available.

Also keep special note of block, debug, poll and network events which need themselves some coordination and may recall some changes in their super class to handle properly...

Basically, it's a feature I really want too, but I know it's going to take considerable time to complete and properly test. I just wanted to give some background on the complexity.

@ricmoo ricmoo added the enhancement New feature or improvement. label Oct 23, 2020
@mikevercoelen
Copy link

mikevercoelen commented Mar 27, 2021

I think this is probably the best solution:

const EXPECTED_PONG_BACK = 15000
const KEEP_ALIVE_CHECK_INTERVAL = 7500

export const startConnection = () => {
  provider = new ethers.providers.WebSocketProvider(config.ETH_NODE_WSS)

  let pingTimeout = null
  let keepAliveInterval = null

  provider._websocket.on('open', () => {
    keepAliveInterval = setInterval(() => {
      logger.debug('Checking if the connection is alive, sending a ping')

      provider._websocket.ping()

      // Use `WebSocket#terminate()`, which immediately destroys the connection,
      // instead of `WebSocket#close()`, which waits for the close timer.
      // Delay should be equal to the interval at which your server
      // sends out pings plus a conservative assumption of the latency.
      pingTimeout = setTimeout(() => {
        provider._websocket.terminate()
      }, EXPECTED_PONG_BACK)
    }, KEEP_ALIVE_CHECK_INTERVAL)

    // TODO: handle contract listeners setup + indexing
  })

  provider._websocket.on('close', () => {
    logger.error('The websocket connection was closed')
    clearInterval(keepAliveInterval)
    clearTimeout(pingTimeout)
    startConnection()
  })

  provider._websocket.on('pong', () => {
    logger.debug('Received pong, so connection is alive, clearing the timeout')
    clearInterval(pingTimeout)
  })
}

This send a ping every 15 seconds, when it sends a ping, it expects a pong back within 7.5 seconds otherwise it closes the connection and calls the main startConnection function to start everything over.

Where it says // TODO: handle contract listeners setup + indexing that's where you should do any indexing or listening for contract events etc.

Fine tune these timing vars to taste, depending on who your Node provider is, this are the settings I use for QuikNode

const EXPECTED_PONG_BACK = 15000
const KEEP_ALIVE_CHECK_INTERVAL = 7500

@gwendall
Copy link

gwendall commented May 17, 2021

To elaborate on @mikevercoelen's answer, I extracted the logic to a function

type KeepAliveParams = {
  provider: ethers.providers.WebSocketProvider;
  onDisconnect: (err: any) => void;
  expectedPongBack?: number;
  checkInterval?: number;
};

const keepAlive = ({
  provider,
  onDisconnect,
  expectedPongBack = 15000,
  checkInterval = 7500,
}: KeepAliveParams) => {
  let pingTimeout: NodeJS.Timeout | null = null;
  let keepAliveInterval: NodeJS.Timeout | null = null;

  provider._websocket.on('open', () => {
    keepAliveInterval = setInterval(() => {
      provider._websocket.ping();

      // Use `WebSocket#terminate()`, which immediately destroys the connection,
      // instead of `WebSocket#close()`, which waits for the close timer.
      // Delay should be equal to the interval at which your server
      // sends out pings plus a conservative assumption of the latency.
      pingTimeout = setTimeout(() => {
        provider._websocket.terminate();
      }, expectedPongBack);
    }, checkInterval);
  });

  // eslint-disable-next-line @typescript-eslint/no-explicit-any
  provider._websocket.on('close', (err: any) => {
    if (keepAliveInterval) clearInterval(keepAliveInterval);
    if (pingTimeout) clearTimeout(pingTimeout);
    onDisconnect(err);
  });

  provider._websocket.on('pong', () => {
    if (pingTimeout) clearInterval(pingTimeout);
  });
};

Then in my code, i get:

const startBot = () => {
  const provider = new ethers.providers.WebSocketProvider(wsUrl);
  keepAlive({
      provider,
      onDisconnect: (err) => {
        startBot();
        console.error('The ws connection was closed', JSON.stringify(err, null, 2));
      },
    });
};

@mikevercoelen
Copy link

We're two months in and the code mentioned before, has been running steadily on our node :) 0 downtime.

@gwendall
Copy link

Really cool ! Thanks again for sharing :)

@sentilesdal
Copy link

@mikevercoelen I'm using ethers 5.0.32 and the websocket provider doesn't have the 'on' method which really hampers implementing your solution ;). What version of ethers are you using?

@ricmoo
Copy link
Member

ricmoo commented May 19, 2021

There should definitely be an .on method. There is no version of WebSocketProvider that didn’t have it, since it inherits from JsonRpcProvider.

@sentilesdal
Copy link

Ok well I'm not sure what's going on. Its definitely not there, I'm seeing an interface for provider._websocket that looks just like a regular websocket interface: https://developer.mozilla.org/en-US/docs/Web/API/WebSocket/onopen.

Is there typo in the code above? Perhaps instead of
provider._websocket.on('open', () => {})
I should be calling these directly on the provider? I tried this too but the provider doesn't recognize the 'open', 'close', and 'pong' event types. websocket-provider.ts from ethers only shows these event types: 'block', 'pending', 'filter', 'tx', 'debug', 'poll', 'willPoll', 'didPoll', 'error'.

@ricmoo
Copy link
Member

ricmoo commented May 20, 2021

Oh! Sorry, yes. In general you should use provider.on. The _websocket is a semi-private member and should not generally be touched , unless direct access too it is needed. But only ethers-supported events are supported by provider.on.

It depends on your environment what your ._webprovider is. Some platforms may use .addEventListener instead of .on, maybe?

If your goal is to enable automatic reconnect, this is not something that is simple to do in a safe way, so make sure you test it thoroughly. :)

@sentilesdal
Copy link

We are actually using alchemy so was able to just use their web3 websocket provider and plugged it into our ethers ecosystem with ethers.provider.Web3Provider. they handle all the reconnects and even dropped calls very gracefully.

@rrecuero
Copy link

rrecuero commented Jun 9, 2021

One question @ricmoo , @gwendall when trying to use the code snippet above I get that the websocket object doesn't have on method.

I am using the latest ethers 5.3 from the dapp

@sentilesdal
Copy link

@rrecuero I ran into the same problem and I'm still not sure how that code above works :P

@tarik0
Copy link

tarik0 commented Jun 14, 2021

The solution of the @mikevercoelen didn't worked on me maybe because I'm using the browser version of WebSocket so for me the workaround was writing a custom class that reconnect's everytime the connection closes.

const ethers = require("ethers");

class ReconnectableEthers {
    /** 
     * Constructs the class
    */
    constructor() {
        this.provider = undefined;
        this.wallet   = undefined;
        this.account  = undefined;
        this.config   = undefined;
        
        this.KEEP_ALIVE_CHECK_INTERVAL = 1000;

        this.keepAliveInterval = undefined;
        this.pingTimeout       = undefined;
    }

    /**
     * Load assets.
     * @param {Object} config Config object.
     */
    load(config) {
        this.config = config;
        this.provider = new ethers.providers.WebSocketProvider(this.config["BSC_PROVIDER_ADDRESS"])
        this.wallet   = new ethers.Wallet(this.config["PRIVATE_KEY"]);
        this.account  = this.wallet.connect(this.provider);
        
        this.defWsOpen    = this.provider._websocket.onopen;
        this.defWsClose   = this.provider._websocket.onclose;

        this.provider._websocket.onopen    = (event) => this.onWsOpen(event);
        this.provider._websocket.onclose   = (event) => this.onWsClose(event);
    }

    /**
     * Check class is loaded.
     * @returns Bool
     */
    isLoaded() {
        if (!this.provider) return false;
        return true;
    }

    /**
     * Triggered when provider's websocket is open.
     */
    onWsOpen(event) {
        console.log("Connected to the WS!");
        this.keepAliveInterval = setInterval(() => { 
            if (
                this.provider._websocket.readyState === WebSocket.OPEN ||
                this.provider._websocket.readyState === WebSocket.OPENING
            ) return;

            this.provider._websocket.close();
        }, this.KEEP_ALIVE_CHECK_INTERVAL)

        if (this.defWsOpen) this.defWsOpen(event);
    }

    /**
     * Triggered on websocket termination.
     * Tries to reconnect again.
     */
    onWsClose(event) {
        console.log("WS connection lost! Reconnecting...");
        clearInterval(this.keepAliveInterval)
        this.load(this.config);

        if (this.defWsClose) this.defWsClose(event);
    }
}

module.exports = ReconnectableEthers;

@mhughdo
Copy link

mhughdo commented Jul 2, 2021

@tarik0 i'm running WebSocket on browser too, that's exactly what I need, thank you

@EvilJordan
Copy link

I implemented @mikevercoelen's method and it works perfectly. Thank you!

Question, though: Where is the ping() function in provider._websocket.ping() defined? I'm not seeing it in the ethers source and confused as to where/how it's doing the sending.

@pulasthibandara
Copy link

@Dylan-Kerler thanks, it is definitely cleaner. I agree with your point for most use cases. But if you need certain consistency guarantees, have a read of @ricmoo's concerns on #1053 (comment). Which the web3 client won't handle.

@phbrgnomo
Copy link

Hello all. Is there plans to implement his on a near future?
Our project is using ethers and at first we considered building this function on our code, but if there is any plans for implementing this on ethers, it would save us some time.

Thanks!

@fubhy
Copy link

fubhy commented Nov 4, 2021

This is harder to implement then it looks on the surface if you want to include:

  • re-establish subscriptions after reconnecting
  • queue and re-send queued requests after the connection is re-established (like it's done before the connection is established for the first time)
  • differentiate between the different reasons for a disconnect (allow client code to define which action to take based on disconnect status code, e.g. if the server is going down, reconnecting is pointless... this must be configurable because it depends on the server implementation)
  • retry & exponential back-off logic
  • keep-alive

None of the solutions (including the web3-provider, which also as a lot of other downsides) posted in here cover this properly so far.

There are resilient, generic websocket implementations out there already (not json-rpc specific) that this could be built around. The only thing that would have to be custom-built in ethers would be tracking of active subscriptions to re-establish those on reconnect.

There's https://github.com/pladaria/reconnecting-websocket which could use some cleanup and modernization but is otherwise rather robust. Could make sense to fork / integrate that into ethers.js with the json-rpc specifics simply built around it .

@cryptofomo1
Copy link

How can this be the accepted solution? It's like 50 lines of custom code that people have to copy and paste for what should be a boilerplate problem.

@sentilesdal solution seems much better - wrapping the input provider in web3Ws provider:

import Web3WsProvider from 'web3-providers-ws'

this.provider = new ethers.providers.Web3Provider(
  new Web3WsProvider(process.env.PROVIDER_URL, {
    clientConfig: {
        keepalive: true,
        keepaliveInterval: 60000, // ms
     }
     // Enable auto reconnection
     reconnect: {
        auto: true,
        delay: 5000, // ms
        maxAttempts: 5,
        onTimeout: false
     }
  }),
)

I don't see why this is so hard to add to ethersjs though. If web3Ws provider can do it, why can't ethersjs do it?

Thanks a lot, def. the answer i was looking for ! works like a charm.

@Vladkryvoruchko
Copy link

How can this be the accepted solution? It's like 50 lines of custom code that people have to copy and paste for what should be a boilerplate problem.
@sentilesdal solution seems much better - wrapping the input provider in web3Ws provider:

import Web3WsProvider from 'web3-providers-ws'

this.provider = new ethers.providers.Web3Provider(
  new Web3WsProvider(process.env.PROVIDER_URL, {
    clientConfig: {
        keepalive: true,
        keepaliveInterval: 60000, // ms
     }
     // Enable auto reconnection
     reconnect: {
        auto: true,
        delay: 5000, // ms
        maxAttempts: 5,
        onTimeout: false
     }
  }),
)

I don't see why this is so hard to add to ethersjs though. If web3Ws provider can do it, why can't ethersjs do it?

Thanks a lot, def. the answer i was looking for ! works like a charm.

So I've tried this solution to mitigate connection hang using websocket provider. It indeed have worked out, BUT
after switching to this provider implementation, my request count has increased 10x

Im listening on 'block' method, and then calling eth_getBlockByNumber using other HTTP provider.

Over 24 hours using InfuraWebsocket provider that may hang, my daily request count was 10k. What a surprise I had when my daily limit of 100k has been reached, after using this provider implementation.

Havent dig why that happened yet

@bennyvenassi
Copy link

bennyvenassi commented Jan 11, 2022

How can this be the accepted solution? It's like 50 lines of custom code that people have to copy and paste for what should be a boilerplate problem.

@sentilesdal solution seems much better - wrapping the input provider in web3Ws provider:

import Web3WsProvider from 'web3-providers-ws'

this.provider = new ethers.providers.Web3Provider(
  new Web3WsProvider(process.env.PROVIDER_URL, {
    clientConfig: {
        keepalive: true,
        keepaliveInterval: 60000, // ms
     }
     // Enable auto reconnection
     reconnect: {
        auto: true,
        delay: 5000, // ms
        maxAttempts: 5,
        onTimeout: false
     }
  }),
)

I don't see why this is so hard to add to ethersjs though. If web3Ws provider can do it, why can't ethersjs do it?

Hey, did you manage to get this to work in node.js?
With integrating your code I get the following message: Web3WsProvider is not a constructor

Maybe this is because you used or code in React with TypeScript or something like that?

@mightymatth
Copy link

I made a workaround like this: created a custom WebSocketProvider that uses ReconnectingWebSocket('url') instead of ordinary Websocket('url') that you use in ethers' WebSocketProvider.
For me, it works fine, without any issue for now.
It could be nice if we could initialize ethers' WebSocketProvider with a custom WebSocket object rather than just a WebSocket URL. What are your thoughts?

@nischitpra
Copy link

Hi, I'm facing a similar issue, but not sure if its entirely related so I've created a SO post.

Basically I have dynamic contract addresses and need to listen to their Transfer events. I'm doing

  contract = new ethers.Contract(tokenAddress, ERC721Abi, _wsprovider);
  contract.on("Transfer", cb);

The txn is logged when the app starts but after 5mins, it stops calling the callback. I cannot do as said by @58bits because my contract addresses are dynamic and I'm create websocket connection at server start, so cannot do it in the open event of websocket.

https://ethereum.stackexchange.com/questions/149750/cannot-listen-to-contract-events-with-ethers-js

@IlyaSinyugin
Copy link

It's pretty easy to use SturdyWebSocket.

import SturdyWebSocket from "sturdy-websocket"

...
new ethers.providers.WebSocketProvider(
  new SturdyWebSocket(this.appEnv$.wsRPCUrl),
  {
    name: this.appEnv$.chainName,
    chainId: this.appEnv$.chainId
  }
)
...

and we have the reconnecting websocket

Thanks, this solution works indeed. If anyone's having trouble with setting up SturdyWebSocket on Node.js refer to this -> dphilipson/sturdy-websocket#25 (comment)

@ubuntutest
Copy link

how to update it with ethers v6? currently this code is deprecated

import { ethers } from 'ethers';

const WEBSOCKET_PING_INTERVAL = 10000;
const WEBSOCKET_PONG_TIMEOUT = 5000;
const WEBSOCKET_RECONNECT_DELAY = 100;

const WebSocketProviderClass = (): new () => ethers.providers.WebSocketProvider => (class {} as never);

export class WebSocketProvider extends WebSocketProviderClass() {
  private provider?: ethers.providers.WebSocketProvider;
  private events: ethers.providers.WebSocketProvider['_events'] = [];
  private requests: ethers.providers.WebSocketProvider['_requests'] = {};

  private handler = {
    get(target: WebSocketProvider, prop: string, receiver: unknown) {
      const value = target.provider && Reflect.get(target.provider, prop, receiver);

      return value instanceof Function ? value.bind(target.provider) : value;
    },
  };

  constructor(private providerUrl) {
    super();
    this.create();

    return new Proxy(this, this.handler);
  }

  private create() {
    if (this.provider) {
      this.events = [...this.events, ...this.provider._events];
      this.requests = { ...this.requests, ...this.provider._requests };
    }

    const provider = new ethers.providers.WebSocketProvider(this.providerUrl, this.provider?.network?.chainId);
    let pingInterval: NodeJS.Timer | undefined;
    let pongTimeout: NodeJS.Timeout | undefined;

    provider._websocket.on('open', () => {
      pingInterval = setInterval(() => {
        provider._websocket.ping();

        pongTimeout = setTimeout(() => { provider._websocket.terminate(); }, WEBSOCKET_PONG_TIMEOUT);
      }, WEBSOCKET_PING_INTERVAL);

      let event;
      while ((event = this.events.pop())) {
        provider._events.push(event);
        provider._startEvent(event);
      }

      for (const key in this.requests) {
        provider._requests[key] = this.requests[key];
        provider._websocket.send(this.requests[key].payload);
        delete this.requests[key];
      }
    });

    provider._websocket.on('pong', () => {
      if (pongTimeout) clearTimeout(pongTimeout);
    });

    provider._websocket.on('close', (code: number) => {
      provider._wsReady = false;

      if (pingInterval) clearInterval(pingInterval);
      if (pongTimeout) clearTimeout(pongTimeout);

      if (code !== 1000) {
        setTimeout(() => this.create(), WEBSOCKET_RECONNECT_DELAY);
      }
    });

    this.provider = provider;
  }
}

@ricmoo
Copy link
Member

ricmoo commented Jul 12, 2023

It looks like the main goal is to configure the WebSocket (including some extra WebSocket events). In v6, you can pass in a function like:

const configWebSocket = () => {
  const ws = new WebSocket(url);
  // Do config…
  ws.on();
  return ws;
};
const provider = new WebSocketProvider(configWebSocket);

That callback will be used whenever it need to reconnect, and in the next minor bump that will be used on disconnect to resubscribe to events.

@ChrisLahaye
Copy link

ricmoo does that mean that the new websocket provider will automatically reconnect and resubscribe to subscriptions?

@makarid
Copy link

makarid commented Jul 12, 2023

Hello, can you please add the reconnection functionality on the WebSocketProvider class to work by default or to have some option to enable/disable it. It is a must feature for every websocket connection. Thank you

@hubchub
Copy link

hubchub commented Jul 20, 2023

@ricmoo I think you may have missed the main goal of this thread. Above was a fix for ethers v5 to make WebsocketProvider reconnect on disconnect. This issue has reappeared in v6 (See code below. Onclose is commented out in main 5a56fc3)

Do you have time to implement this or would it help to have it lifted off your plate?

export class WebSocketProvider extends SocketProvider {
    #connect: null | WebSocketCreator;

    #websocket: null | WebSocketLike;
    get websocket(): WebSocketLike {
        if (this.#websocket == null) { throw new Error("websocket closed"); }
        return this.#websocket;
    }

    constructor(url: string | WebSocketLike | WebSocketCreator, network?: Networkish) {
        super(network);
        if (typeof(url) === "string") {
            this.#connect = () => { return new _WebSocket(url); };
            this.#websocket = this.#connect();
        } else if (typeof(url) === "function") {
            this.#connect = url;
            this.#websocket = url();
        } else {
            this.#connect = null;
            this.#websocket = url;
        }

        this.websocket.onopen = async () => {
            try {
                await this._start()
                this.resume();
            } catch (error) {
                console.log("failed to start WebsocketProvider", error);
                // @TODO: now what? Attempt reconnect?
            }
        };

        this.websocket.onmessage = (message: { data: string }) => {
            this._processMessage(message.data);
        };
/*
        this.websocket.onclose = (event) => {
            // @TODO: What event.code should we reconnect on?
            const reconnect = false;
            if (reconnect) {
                this.pause(true);
                if (this.#connect) {
                    this.#websocket = this.#connect();
                    this.#websocket.onopen = ...
                    // @TODO: this requires the super class to rebroadcast; move it there
                }
                this._reconnect();
            }
        };
*/
    }

    async _write(message: string): Promise<void> {
        this.websocket.send(message);
    }

    async destroy(): Promise<void> {
        if (this.#websocket != null) {
            this.#websocket.close();
            this.#websocket = null;
        }
        super.destroy();
    }
}

@fallonp
Copy link

fallonp commented Sep 21, 2023

It's not pretty. but I used
yarn add ethers5@npm:[email protected]
to get 2 different ethers.js in my node app:

    "ethers": "^6.7.1",
    "ethers5": "npm:[email protected]",

Then used ethers5 with the websocket.ts code @ubuntutest mentioned for my Websocket provider, and ethers 6.7.1 for everything else.

@ghost
Copy link

ghost commented Oct 2, 2023

For V6 with reconnection to WebSocket every 100ms ( Will throw when the failure reaches 5 times for a single reconnection attempt )

isomorphic-ws is used to initiate new WebSocket object since sharing socket listeners cause issue on Node.js with my previous experience

( The code works from the browser-side or react-native as well )

const WebSocket = require('isomorphic-ws');
const { WebSocketProvider, SocketBlockSubscriber } = require('ethers');

const sleep = ms => new Promise(r => setTimeout(r, ms));

// Testing WebSocket on connection / reconnection before initiating new provider to prevent deadlock
const testWS = async (url, reconnectDelay = 100) => {
  const MAX_RETRY = 5;
  let retry = 0;
  let errorObject;
  
  while (retry < MAX_RETRY + 1) {
    try {
      return await new Promise((resolve, reject) => {
        const socket = new WebSocket(url);
    
        socket.onopen = () => {
          socket.send(JSON.stringify([
            {
              jsonrpc: '2.0',
              method: 'eth_chainId',
              params: [],
              id: 1
            },
            {
              jsonrpc: '2.0',
              method: 'eth_getBlockByNumber',
              params: ['latest', false],
              id: 2
            }
          ]));
        };
    
        socket.onmessage = (event) => {
          const data = JSON.parse(event.data);
    
          resolve({
            chainId: Number(data[0]?.result),
            block: data[1]?.result,
          });
        };
    
        socket.onerror = (e) => {
          reject(e);
        };
      });
    } catch (e) {
      console.log(`Connection to ${url} failed, attempt: (${retry} of ${MAX_RETRY})`);
      await sleep(reconnectDelay);
      errorObject = e;
      retry++;
    }
  }

  throw errorObject;
}

const connectWS = async (url, reconnectDelay = 100) => {
  // Test websocket connection to prevent WebSocketProvider deadlock caused by await this._start();
  const { chainId, block } = await testWS(url, reconnectDelay);
  console.log(`WebSocket ${url} connected: Chain ${chainId} Block ${Number(block?.number)}`);

  const provider = new WebSocketProvider(url);
  const blockSub = new SocketBlockSubscriber(provider);

  provider.websocket.onclose = (e) => {
    console.log(`Socket ${url} is closed, reconnecting in ${reconnectDelay} ms`);
    setTimeout(() => connectWS(url, reconnectDelay), reconnectDelay);
  }

  provider.websocket.onerror = (e) => {
    console.error(`Socket ${url} encountered error, reconnecting it:\n${e.stack || e.message}`);
    blockSub.stop();
    provider.destroy();
  }

  blockSub._handleMessage = (result) => {
    console.log(provider._wrapBlock({...result, transactions: []}));
  };
  blockSub.start();

  provider.on('pending', (tx) => {
    console.log(`New pending tx: ${tx}`)
  });
}

connectWS('wss://ethereum.publicnode.com');

@ghost
Copy link

ghost commented Oct 2, 2023

cc @makarid @hubchub @fallonp

@iquirino
Copy link

iquirino commented Oct 3, 2023

+1 v6

@iquirino
Copy link

iquirino commented Oct 3, 2023

I've found a way to implement that old solution on v6:

import { Networkish, WebSocketProvider } from "ethers";
import WebSocket from "ws";

const EXPECTED_PONG_BACK = 15000;
const KEEP_ALIVE_CHECK_INTERVAL = 60 * 1000; //7500;

const debug = (message: string) => {
  console.debug(new Date().toISOString(), message);
};

export const ResilientWebsocket = (
  url: string,
  network: Networkish,
  task: (provider: WebSocketProvider) => void
) => {
  let terminate = false;
  let pingTimeout: NodeJS.Timeout | null = null;
  let keepAliveInterval: NodeJS.Timeout | null = null;
  let ws: WebSocket | null;

  const sleep = (ms: number) =>
    new Promise((resolve) => setTimeout(resolve, ms));

  const startConnection = () => {
    ws = new WebSocket(url);
    ws.on("open", async () => {
      keepAliveInterval = setInterval(() => {
        if (!ws) {
          debug("No websocket, exiting keep alive interval");
          return;
        }
        debug("Checking if the connection is alive, sending a ping");

        ws.ping();

        // Use `WebSocket#terminate()`, which immediately destroys the connection,
        // instead of `WebSocket#close()`, which waits for the close timer.
        // Delay should be equal to the interval at which your server
        // sends out pings plus a conservative assumption of the latency.
        pingTimeout = setTimeout(() => {
          if (ws) ws.terminate();
        }, EXPECTED_PONG_BACK);
      }, KEEP_ALIVE_CHECK_INTERVAL);

      const wsp = new WebSocketProvider(() => ws!, network);

      while (ws?.readyState !== WebSocket.OPEN) {
        debug("Waiting for websocket to be open");
        await sleep(1000);
      }

      wsp._start();

      while (!wsp.ready) {
        debug("Waiting for websocket provider to be ready");
        await sleep(1000);
      }

      task(wsp);
    });

    ws.on("close", () => {
      console.error("The websocket connection was closed");
      if (keepAliveInterval) clearInterval(keepAliveInterval);
      if (pingTimeout) clearTimeout(pingTimeout);
      if (!terminate) startConnection();
    });

    ws.on("pong", () => {
      debug("Received pong, so connection is alive, clearing the timeout");
      if (pingTimeout) clearInterval(pingTimeout);
    });

    return ws;
  };

  startConnection();

  return () => {
    terminate = true;
    if (keepAliveInterval) clearInterval(keepAliveInterval);
    if (pingTimeout) clearTimeout(pingTimeout);
    if (ws) {
      ws.removeAllListeners();
      ws.terminate();
    }
  };
};

Usage:

terminate = ResilientWebsocket(
    WEBSOCKET_URL,
    Number(CHAIN_ID),
    async (provider) => {
      console.log("connected");
    }
  );

So, you can terminate your process anytime using terminate();

@valeriivanchev
Copy link

For V6 with reconnection to WebSocket every 100ms ( Will throw when the failure reaches 5 times for a single reconnection attempt )

isomorphic-ws is used to initiate new WebSocket object since sharing socket listeners cause issue on Node.js with my previous experience

( The code works from the browser-side or react-native as well )

const WebSocket = require('isomorphic-ws');
const { WebSocketProvider, SocketBlockSubscriber } = require('ethers');

const sleep = ms => new Promise(r => setTimeout(r, ms));

// Testing WebSocket on connection / reconnection before initiating new provider to prevent deadlock
const testWS = async (url, reconnectDelay = 100) => {
  const MAX_RETRY = 5;
  let retry = 0;
  let errorObject;
  
  while (retry < MAX_RETRY + 1) {
    try {
      return await new Promise((resolve, reject) => {
        const socket = new WebSocket(url);
    
        socket.onopen = () => {
          socket.send(JSON.stringify([
            {
              jsonrpc: '2.0',
              method: 'eth_chainId',
              params: [],
              id: 1
            },
            {
              jsonrpc: '2.0',
              method: 'eth_getBlockByNumber',
              params: ['latest', false],
              id: 2
            }
          ]));
        };
    
        socket.onmessage = (event) => {
          const data = JSON.parse(event.data);
    
          resolve({
            chainId: Number(data[0]?.result),
            block: data[1]?.result,
          });
        };
    
        socket.onerror = (e) => {
          reject(e);
        };
      });
    } catch (e) {
      console.log(`Connection to ${url} failed, attempt: (${retry} of ${MAX_RETRY})`);
      await sleep(reconnectDelay);
      errorObject = e;
      retry++;
    }
  }

  throw errorObject;
}

const connectWS = async (url, reconnectDelay = 100) => {
  // Test websocket connection to prevent WebSocketProvider deadlock caused by await this._start();
  const { chainId, block } = await testWS(url, reconnectDelay);
  console.log(`WebSocket ${url} connected: Chain ${chainId} Block ${Number(block?.number)}`);

  const provider = new WebSocketProvider(url);
  const blockSub = new SocketBlockSubscriber(provider);

  provider.websocket.onclose = (e) => {
    console.log(`Socket ${url} is closed, reconnecting in ${reconnectDelay} ms`);
    setTimeout(() => connectWS(url, reconnectDelay), reconnectDelay);
  }

  provider.websocket.onerror = (e) => {
    console.error(`Socket ${url} encountered error, reconnecting it:\n${e.stack || e.message}`);
    blockSub.stop();
    provider.destroy();
  }

  blockSub._handleMessage = (result) => {
    console.log(provider._wrapBlock({...result, transactions: []}));
  };
  blockSub.start();

  provider.on('pending', (tx) => {
    console.log(`New pending tx: ${tx}`)
  });
}

connectWS('wss://ethereum.publicnode.com');

The solution is not working anymore as the onclose func of the websocket provider is commented out in the package

@iquirino
Copy link

iquirino commented Jan 22, 2024

@valeriivanchev

Solution: #1053 (comment)

@RastogiAbhijeet
Copy link

RastogiAbhijeet commented Feb 20, 2024

Has there been further progress on this thread?
Any cleaner solution?

@iquirino
Copy link

Has there been further progress on this thread? Any cleaner solution?

As I can see, the ethers.js owner preferred to let users to implement himself the websocket reconnection strategy,

@Amirjab21
Copy link

It's pretty easy to use SturdyWebSocket.

import SturdyWebSocket from "sturdy-websocket"

...
new ethers.providers.WebSocketProvider(
  new SturdyWebSocket(this.appEnv$.wsRPCUrl),
  {
    name: this.appEnv$.chainName,
    chainId: this.appEnv$.chainId
  }
)
...

and we have the reconnecting websocket

This didn't work for me on NodeJS, the websocket still dies

const SturdyWebSocket = require('sturdy-websocket');
const WebSocket = require('ws');
const { WebSocketProvider } = require('ethers');

new WebSocketProvider(new SturdyWebSocket.default(WS_URL, { wsConstructor: WebSocket}), { chainId: 8453}),

@ChrisLahaye
Copy link

Maintaining the websocket connection is only half of the equation. You should also take care of requests triggering when the connection is down as well as re-subscribing to pending subscriptions to avoid that your code gets stuck.

Since I still see comments frequently being posted in this issue I would like to share our solution. It is an improvement on what was previously shared with exponential back off and better closing code. It is not fancy but it works reliably.

import { ethers } from 'ethers';
import { WebSocket } from 'ws';

const WEBSOCKET_BACKOFF_BASE = 100;
const WEBSOCKET_BACKOFF_CAP = 30000;
const WEBSOCKET_PING_INTERVAL = 10000;
const WEBSOCKET_PONG_TIMEOUT = 5000;

function getRandomInt(min: number, max: number) {
  return Math.floor(Math.random() * (max - min + 1)) + min;
}

const WebSocketProviderClass = (): new () => ethers.providers.WebSocketProvider => (class {} as never);

class WebSocketProvider extends WebSocketProviderClass() {
  private attempts = 0;
  private destroyed = false;
  private timeout?: NodeJS.Timeout;

  private events: ethers.providers.WebSocketProvider['_events'] = [];
  private requests: ethers.providers.WebSocketProvider['_requests'] = {};
  private provider?: ethers.providers.WebSocketProvider;

  private handler = {
    get(target: WebSocketProvider, prop: keyof WebSocketProvider, receiver: unknown) {
      if (target[prop]) return target[prop];

      const value = target.provider && Reflect.get(target.provider, prop, receiver);

      return value instanceof Function ? value.bind(target.provider) : value;
    },
  };

  constructor(private providerUrl: string) {
    super();
    this.create();

    return new Proxy(this, this.handler);
  }

  private create() {
    if (this.provider) {
      this.events = [...this.events, ...this.provider._events];
      this.requests = { ...this.requests, ...this.provider._requests };
    }

    const webSocket = new WebSocket(this.providerUrl);
    const provider = new ethers.providers.WebSocketProvider(webSocket as never, this.provider?.network?.chainId);
    let pingInterval: NodeJS.Timer | undefined;
    let pongTimeout: NodeJS.Timeout | undefined;

    webSocket.on('open', () => {
      this.attempts = 0;

      pingInterval = setInterval(() => {
        webSocket.ping();

        pongTimeout = setTimeout(() => {
          webSocket.terminate();
        }, WEBSOCKET_PONG_TIMEOUT);
      }, WEBSOCKET_PING_INTERVAL);

      let event;
      while ((event = this.events.pop())) {
        provider._events.push(event);
        provider._startEvent(event);
      }

      for (const key in this.requests) {
        provider._requests[key] = this.requests[key];
        webSocket.send(this.requests[key].payload);
        delete this.requests[key];
      }
    });

    webSocket.on('error', (err) => {
      console.info('WebSocket error: %s', err.message);
    });

    webSocket.on('pong', () => {
      if (pongTimeout) clearTimeout(pongTimeout);
    });

    webSocket.on('close', () => {
      provider._wsReady = false;

      if (pingInterval) clearInterval(pingInterval);
      if (pongTimeout) clearTimeout(pongTimeout);

      if (!this.destroyed) {
        const sleep = getRandomInt(0, Math.min(WEBSOCKET_BACKOFF_CAP, WEBSOCKET_BACKOFF_BASE * 2 ** this.attempts++));

        this.timeout = setTimeout(() => this.create(), sleep);
      }
    });

    this.provider = provider;
  }

  public async destroy() {
    this.destroyed = true;

    if (this.timeout) {
      clearTimeout(this.timeout);
    }

    if (this.provider) {
      await this.provider.destroy();
    }
  }
}
const provider = new WebSocketProvider('wss://xxx');
await provider.getBlockNumber();

This code has only been tested and used with ethers 5. It (mis-)uses ethers internals so I doubt it would work with ethers 6.

@inapeace0
Copy link

inapeace0 commented May 14, 2024

Hi @ricmoo,

I'm using WebSocketProvider server-side to listen to blockchain events and performing calls to smart contracts. Sometimes the websocket pipe got broken and I need to reconnect it.

I use this code to detect ws close and reconnect but it would be nice to not have to rely on _websocket to do it:

let wsProvider;

init = async () => {
  wsProvider = new ethers.providers.WebSocketProvider(wsHost);
  wsProvider._websocket.on('close', async (code) => {
    console.log('ws closed', code);
    wsProvider._websocket.terminate();
    await sleep(3000); // wait before reconnect
    init();
  });
  wsProvider.on('block', doStuff);
};

I also noticed when the websocket is broken Promise call don't reject wich is not super intuitive.

Hello, I used this way with quicknode, but the disconnection is sometimes occured.
"WebSocketProvider" solves this issue with the higher ethers version?

I'm using v5.5.4, and there was no issue with local full nodes for 15 days.

Would like to figure this asap. 🙏
Thanks in advance.

@iamnotstatic
Copy link

Maintaining the websocket connection is only half of the equation. You should also take care of requests triggering when the connection is down as well as re-subscribing to pending subscriptions to avoid that your code gets stuck.

Since I still see comments frequently being posted in this issue I would like to share our solution. It is an improvement on what was previously shared with exponential back off and better closing code. It is not fancy but it works reliably.

import { ethers } from 'ethers';
import { WebSocket } from 'ws';

const WEBSOCKET_BACKOFF_BASE = 100;
const WEBSOCKET_BACKOFF_CAP = 30000;
const WEBSOCKET_PING_INTERVAL = 10000;
const WEBSOCKET_PONG_TIMEOUT = 5000;

function getRandomInt(min: number, max: number) {
  return Math.floor(Math.random() * (max - min + 1)) + min;
}

const WebSocketProviderClass = (): new () => ethers.providers.WebSocketProvider => (class {} as never);

class WebSocketProvider extends WebSocketProviderClass() {
  private attempts = 0;
  private destroyed = false;
  private timeout?: NodeJS.Timeout;

  private events: ethers.providers.WebSocketProvider['_events'] = [];
  private requests: ethers.providers.WebSocketProvider['_requests'] = {};
  private provider?: ethers.providers.WebSocketProvider;

  private handler = {
    get(target: WebSocketProvider, prop: keyof WebSocketProvider, receiver: unknown) {
      if (target[prop]) return target[prop];

      const value = target.provider && Reflect.get(target.provider, prop, receiver);

      return value instanceof Function ? value.bind(target.provider) : value;
    },
  };

  constructor(private providerUrl: string) {
    super();
    this.create();

    return new Proxy(this, this.handler);
  }

  private create() {
    if (this.provider) {
      this.events = [...this.events, ...this.provider._events];
      this.requests = { ...this.requests, ...this.provider._requests };
    }

    const webSocket = new WebSocket(this.providerUrl);
    const provider = new ethers.providers.WebSocketProvider(webSocket as never, this.provider?.network?.chainId);
    let pingInterval: NodeJS.Timer | undefined;
    let pongTimeout: NodeJS.Timeout | undefined;

    webSocket.on('open', () => {
      this.attempts = 0;

      pingInterval = setInterval(() => {
        webSocket.ping();

        pongTimeout = setTimeout(() => {
          webSocket.terminate();
        }, WEBSOCKET_PONG_TIMEOUT);
      }, WEBSOCKET_PING_INTERVAL);

      let event;
      while ((event = this.events.pop())) {
        provider._events.push(event);
        provider._startEvent(event);
      }

      for (const key in this.requests) {
        provider._requests[key] = this.requests[key];
        webSocket.send(this.requests[key].payload);
        delete this.requests[key];
      }
    });

    webSocket.on('error', (err) => {
      console.info('WebSocket error: %s', err.message);
    });

    webSocket.on('pong', () => {
      if (pongTimeout) clearTimeout(pongTimeout);
    });

    webSocket.on('close', () => {
      provider._wsReady = false;

      if (pingInterval) clearInterval(pingInterval);
      if (pongTimeout) clearTimeout(pongTimeout);

      if (!this.destroyed) {
        const sleep = getRandomInt(0, Math.min(WEBSOCKET_BACKOFF_CAP, WEBSOCKET_BACKOFF_BASE * 2 ** this.attempts++));

        this.timeout = setTimeout(() => this.create(), sleep);
      }
    });

    this.provider = provider;
  }

  public async destroy() {
    this.destroyed = true;

    if (this.timeout) {
      clearTimeout(this.timeout);
    }

    if (this.provider) {
      await this.provider.destroy();
    }
  }
}
const provider = new WebSocketProvider('wss://xxx');
await provider.getBlockNumber();

This code has only been tested and used with ethers 5. It (mis-)uses ethers internals so I doubt it would work with ethers 6.

Thanks man, I've been using your initial solution, which has been working fine for us. Let's incorporate this update

@iquirino
Copy link

I've found a way to implement that old solution on v6:

import { Networkish, WebSocketProvider } from "ethers";
import WebSocket from "ws";

const EXPECTED_PONG_BACK = 15000;
const KEEP_ALIVE_CHECK_INTERVAL = 60 * 1000; //7500;

const debug = (message: string) => {
  console.debug(new Date().toISOString(), message);
};

export const ResilientWebsocket = (
  url: string,
  network: Networkish,
  task: (provider: WebSocketProvider) => void
) => {
  let terminate = false;
  let pingTimeout: NodeJS.Timeout | null = null;
  let keepAliveInterval: NodeJS.Timeout | null = null;
  let ws: WebSocket | null;

  const sleep = (ms: number) =>
    new Promise((resolve) => setTimeout(resolve, ms));

  const startConnection = () => {
    ws = new WebSocket(url);
    ws.on("open", async () => {
      keepAliveInterval = setInterval(() => {
        if (!ws) {
          debug("No websocket, exiting keep alive interval");
          return;
        }
        debug("Checking if the connection is alive, sending a ping");

        ws.ping();

        // Use `WebSocket#terminate()`, which immediately destroys the connection,
        // instead of `WebSocket#close()`, which waits for the close timer.
        // Delay should be equal to the interval at which your server
        // sends out pings plus a conservative assumption of the latency.
        pingTimeout = setTimeout(() => {
          if (ws) ws.terminate();
        }, EXPECTED_PONG_BACK);
      }, KEEP_ALIVE_CHECK_INTERVAL);

      const wsp = new WebSocketProvider(() => ws!, network);

      while (ws?.readyState !== WebSocket.OPEN) {
        debug("Waiting for websocket to be open");
        await sleep(1000);
      }

      wsp._start();

      while (!wsp.ready) {
        debug("Waiting for websocket provider to be ready");
        await sleep(1000);
      }

      task(wsp);
    });

    ws.on("close", () => {
      console.error("The websocket connection was closed");
      if (keepAliveInterval) clearInterval(keepAliveInterval);
      if (pingTimeout) clearTimeout(pingTimeout);
      if (!terminate) startConnection();
    });

    ws.on("pong", () => {
      debug("Received pong, so connection is alive, clearing the timeout");
      if (pingTimeout) clearInterval(pingTimeout);
    });

    return ws;
  };

  startConnection();

  return () => {
    terminate = true;
    if (keepAliveInterval) clearInterval(keepAliveInterval);
    if (pingTimeout) clearTimeout(pingTimeout);
    if (ws) {
      ws.removeAllListeners();
      ws.terminate();
    }
  };
};

Usage:

terminate = ResilientWebsocket(
    WEBSOCKET_URL,
    Number(CHAIN_ID),
    async (provider) => {
      console.log("connected");
    }
  );

So, you can terminate your process anytime using terminate();

Guys, V6 Solution... Why do you continue posting old solutions? You just need to read the thread.....
We have V5 and V6 solution here... Stop posting it again and again...

@blacklistholder
Copy link

blacklistholder commented Aug 16, 2024

Just in case any of you need, here is sophisticated solution which will get you in the end same interface as WebSocketProvider. Reconnecting with preserving topics, ping/pong, ethers v6. I believe it solves 100% of described here problems.

Example of use:

const { createResilientProviders } = require('./ResilientWebsocketProvider');

async initializeProviders() {
        this.providers = await createResilientProviders(config.wssProviders, config.networkID);

        this.providers.map(provider => provider.on('block', (blockNumber) => console.log("New Block", blockNumber)));
        // as you see, you can simply use provider.on(...) as usually, but it will have functionality of resilient WS
}
    
initializeProviders();

Here is implementation:

const { WebSocketProvider } = require("ethers");
const WebSocket = require("ws");

const EXPECTED_PONG_BACK = 15000;
const KEEP_ALIVE_CHECK_INTERVAL = 60 * 1000;
const MAX_RECONNECTION_ATTEMPTS = 5;
const RECONNECTION_DELAY = 5000; // 5 seconds

const debug = (message) => {
    console.debug(new Date().toISOString(), message);
};

class ResilientWebsocketProvider {
    constructor(url, network) {
        this.url = url;
        this.network = network;
        this.terminate = false;
        this.pingTimeout = null;
        this.keepAliveInterval = null;
        this.ws = null;
        this.provider = null;
        this.subscriptions = new Set();
        this.reconnectionAttempts = 0;
        this.isConnected = false;
    }

    async connect() {
        return new Promise((resolve) => {
            const startConnection = () => {
                if (this.reconnectionAttempts >= MAX_RECONNECTION_ATTEMPTS) {
                    console.error(`Max reconnection attempts (${MAX_RECONNECTION_ATTEMPTS}) reached for ${this.url}. Stopping reconnection.`);
                    this.terminate = true;
                    resolve(null);
                    return;
                }

                this.ws = new WebSocket(this.url);

                this.ws.on("open", async () => {
                    this.reconnectionAttempts = 0;
                    this.isConnected = true;
                    this.setupKeepAlive();

                    try {
                        const wsp = new WebSocketProvider(() => this.ws, this.network);

                        while (this.ws?.readyState !== WebSocket.OPEN) {
                            debug("Waiting for websocket to be open");
                            await this.sleep(1000);
                        }

                        wsp._start();

                        while (!wsp.ready) {
                            debug("Waiting for websocket provider to be ready");
                            await this.sleep(1000);
                        }

                        this.provider = wsp;
                        await this.resubscribe();
                        resolve(this.provider);
                    } catch (error) {
                        console.error(`Error initializing WebSocketProvider for ${this.url}:`, error);
                        this.cleanupConnection();
                        this.reconnectionAttempts++;
                        setTimeout(startConnection, RECONNECTION_DELAY);
                    }
                });

                this.ws.on("close", () => {
                    console.error(`The websocket connection was closed for ${this.url}`);
                    this.isConnected = false;
                    this.cleanupConnection();
                    if (!this.terminate) {
                        this.reconnectionAttempts++;
                        debug(`Attempting to reconnect... (Attempt ${this.reconnectionAttempts})`);
                        setTimeout(startConnection, RECONNECTION_DELAY);
                    }
                });

                this.ws.on("error", (error) => {
                    console.error(`WebSocket error for ${this.url}:`, error);
                });

                this.ws.on("pong", () => {
                    debug("Received pong, so connection is alive, clearing the timeout");
                    if (this.pingTimeout) clearTimeout(this.pingTimeout);
                });
            };

            startConnection();
        });
    }

    setupKeepAlive() {
        this.keepAliveInterval = setInterval(() => {
            if (!this.ws) {
                debug("No websocket, exiting keep alive interval");
                return;
            }
            debug("Checking if the connection is alive, sending a ping");

            this.ws.ping();

            this.pingTimeout = setTimeout(() => {
                if (this.ws) this.ws.terminate();
            }, EXPECTED_PONG_BACK);
        }, KEEP_ALIVE_CHECK_INTERVAL);
    }

    cleanupConnection() {
        if (this.keepAliveInterval) clearInterval(this.keepAliveInterval);
        if (this.pingTimeout) clearTimeout(this.pingTimeout);
    }

    async resubscribe() {
        debug("Resubscribing to topics...");
        for (const subscription of this.subscriptions) {
            try {
                await this.provider.subscribe(subscription.type, subscription.filter, subscription.listener);
                debug(`Resubscribed to ${subscription.type}`);
            } catch (error) {
                console.error(`Failed to resubscribe to ${subscription.type}:`, error);
            }
        }
    }

    sleep(ms) {
        return new Promise((resolve) => setTimeout(resolve, ms));
    }
}

async function createResilientProviders(urls, network) {
    const providers = await Promise.all(
        urls.map(async (url) => {
            try {
                const resilientProvider = new ResilientWebsocketProvider(url, network);
                const provider = await resilientProvider.connect();
                if (provider) {
                    // Wrap the provider's 'on' method to track subscriptions
                    const originalOn = provider.on.bind(provider);
                    provider.on = (eventName, listener) => {
                        resilientProvider.subscriptions.add({ type: eventName, listener });
                        return originalOn(eventName, listener);
                    };
                }
                return provider;
            } catch (error) {
                console.error(`Failed to create ResilientWebsocketProvider for ${url}:`, error);
                return null;
            }
        })
    );

    // Filter out any null providers (failed connections)
    return providers.filter(provider => provider !== null);
}

module.exports = { createResilientProviders, ResilientWebsocketProvider };

@trixobird
Copy link

trixobird commented Oct 9, 2024

here is the typescript version of the great work of @blacklistholder

import { Listener, ProviderEvent, WebSocketProvider } from 'ethers';
import { WebSocket } from 'ws';

const EXPECTED_PONG_BACK = 15000;
const KEEP_ALIVE_CHECK_INTERVAL = 60 * 1000;
const MAX_RECONNECTION_ATTEMPTS = 10;
const RECONNECTION_DELAY = 5000; // 5 seconds

const debug = (message: string) => {
  console.debug(message);
};

interface Subscription {
  type: ProviderEvent;
  listener: Listener;
}

class ResilientWebsocketProvider {
  private readonly url: string;
  private readonly network: number;
  private terminate: boolean;
  private pingTimeout: NodeJS.Timeout | null;
  private keepAliveInterval: NodeJS.Timeout | null;
  private ws: WebSocket | null;
  private provider: WebSocketProvider | null;
  readonly subscriptions: Set<Subscription>;
  private reconnectionAttempts: number;
  private isConnected: boolean;

  constructor(url: string, network: number) {
    this.url = url;
    this.network = network;
    this.terminate = false;
    this.pingTimeout = null;
    this.keepAliveInterval = null;
    this.ws = null;
    this.provider = null;
    this.subscriptions = new Set();
    this.reconnectionAttempts = 0;
    this.isConnected = false;
  }

  async connect(): Promise<WebSocketProvider | null> {
    return new Promise((resolve) => {
      const startConnection = () => {
        if (this.reconnectionAttempts >= MAX_RECONNECTION_ATTEMPTS) {
          console.error(
            `Max reconnection attempts (${MAX_RECONNECTION_ATTEMPTS}) reached for ${this.url}. Stopping reconnection.`,
          );
          this.terminate = true;
          resolve(null);
          return;
        }

        this.ws = new WebSocket(this.url);

        this.ws.on('open', async () => {
          this.reconnectionAttempts = 0;
          this.isConnected = true;
          this.setupKeepAlive();

          try {
            const wsp = new WebSocketProvider(() => this.ws, this.network);

            while (this.ws?.readyState !== WebSocket.OPEN) {
              debug('Waiting for websocket to be open');
              await this.sleep(1000);
            }

            wsp._start();

            while (!wsp.ready) {
              debug('Waiting for websocket provider to be ready');
              await this.sleep(1000);
            }

            this.provider = wsp;
            await this.resubscribe();
            resolve(this.provider);
          } catch (error) {
            console.error(`Error initializing WebSocketProvider for ${this.url}:`, error);
            this.cleanupConnection();
            this.reconnectionAttempts++;
            setTimeout(startConnection, RECONNECTION_DELAY);
          }
        });

        this.ws.on('close', () => {
          console.error(`The websocket connection was closed for ${this.url}`);
          this.isConnected = false;
          this.cleanupConnection();
          if (!this.terminate) {
            this.reconnectionAttempts++;
            debug(`Attempting to reconnect... (Attempt ${this.reconnectionAttempts})`);
            setTimeout(startConnection, RECONNECTION_DELAY);
          }
        });

        this.ws.on('error', (error) => {
          console.error(`WebSocket error for ${this.url}:`, error);
        });

        this.ws.on('pong', () => {
          debug('Received pong, so connection is alive, clearing the timeout');
          if (this.pingTimeout) clearTimeout(this.pingTimeout);
        });
      };

      startConnection();
    });
  }

  private setupKeepAlive() {
    this.keepAliveInterval = setInterval(() => {
      if (!this.ws) {
        debug('No websocket, exiting keep alive interval');
        return;
      }
      debug('Checking if the connection is alive, sending a ping');

      this.ws.ping();

      this.pingTimeout = setTimeout(() => {
        if (this.ws) this.ws.terminate();
      }, EXPECTED_PONG_BACK);
    }, KEEP_ALIVE_CHECK_INTERVAL);
  }

  private cleanupConnection() {
    if (this.keepAliveInterval) clearInterval(this.keepAliveInterval);
    if (this.pingTimeout) clearTimeout(this.pingTimeout);
  }

  private async resubscribe() {
    debug('Resubscribing to topics...');
    for (const subscription of this.subscriptions) {
      try {
        await this.provider?.on(subscription.type, subscription.listener);
        debug(`Resubscribed to ${JSON.stringify(subscription.type)}`);
      } catch (error) {
        console.error(error, `Failed to resubscribe to ${subscription.type}:`);
      }
    }
  }

  private sleep(ms: number) {
    return new Promise((resolve) => setTimeout(resolve, ms));
  }
}

async function createResilientProviders(urls: string[], network: number): Promise<WebSocketProvider[]> {
  const providers = await Promise.all(
    urls.map(async (url) => {
      try {
        const resilientProvider = new ResilientWebsocketProvider(url, network);
        const provider = await resilientProvider.connect();
        if (provider) {
          // Wrap the provider's 'on' method to track subscriptions
          const originalOn = provider.on.bind(provider);
          provider.on = (eventName: ProviderEvent, listener: Listener) => {
            resilientProvider.subscriptions.add({ type: eventName, listener });
            return originalOn(eventName, listener);
          };
        }
        return provider;
      } catch (error) {
        console.error(`Failed to create ResilientWebsocketProvider for ${url}:`, error);
        return null;
      }
    }),
  );

  // Filter out any null providers (failed connections)
  return providers.filter((provider) => provider !== null) as WebSocketProvider[];
}

export { createResilientProviders, ResilientWebsocketProvider };

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or improvement.
Projects
None yet
Development

No branches or pull requests