fix: reduce Connection keep-alive timeout to 1 second fewer than the Solana RPC's keep-alive timeout

[steveluscher]

Problem

When contacting an RPC that's behind a load balancer, clients can often send an RPC request down a free socket, only to discover that socket has since been disposed of. These requests will fail.

Based on this excellent article on tuning keep-alive, this is what I think I've learned.

• The underlying HTTP library that the Solana RPC uses (hyper) has a default keep-alive timeout of 20s.
• Typical Node.js servers have a default keep-alive timeout of 5s.
• When the RPC is behind a load balancer, a higher ‘free socket timeout’ in the load balancer can result in the RPC closing the socket, but the load balancer (ergo, the client) thinking that it's still open. The next request will fail.

I believe the solutions to be as follows:

1. Let people supply their own agents or disable agents altogether should they like to do some tuning feat: you can now supply your own HTTP agent to a web3.js Connection #29125.
2. Reduce the timeout of our default agent to the Solana RPC's timeout, minus one second (20s - 1s = 19s) fix: reduce Connection keep-alive timeout to 1 second fewer than the Solana RPC's keep-alive timeout #29130.
3. RPC providers should maybe do the same – setting their load balancer timeouts to 1 second less than the Solana RPC's timeout (20s - 1s = 19s).

Summary of Changes

• Replaced our custom agent implementation with the agentkeepalive module.
• Set the free socket timeout to the Solana RPC's timeout minus one second (20s - 1s = 19s)

Fixes #27859, hopefully.

[codecov]

Codecov Report

Merging #29130 (9904db5) into master (f1427dd) will decrease coverage by 0.1%.
The diff coverage is 33.3%.

@@            Coverage Diff            @@
##           master   #29130     +/-   ##
=========================================
- Coverage    76.7%    76.5%   -0.2%     
=========================================
  Files          55       54      -1     
  Lines        3140     3119     -21     
  Branches      472      468      -4     
=========================================
- Hits         2410     2388     -22     
- Misses        565      567      +2     
+ Partials      165      164      -1     

[steveluscher]

Thoughts, @dancamarg0, @linuskendall, @brianlong?

[steveluscher]

@0xCactus, @y2kappa, would you like a strategy to test this out before I land it? You can either patch in the changes yourself, or I can publish a release candidate package just for you, that you can add to your package.json.

[0xCactus]

@steveluscher yup i can help test it out. Would prefer a release candidate package

[steveluscher]

It's all yours, @0xCactus. You can yarn add @solana/web3.js@pr-29130.

[0xCactus]

getting the following error @steveluscher

{"level":50,"time":1670443877071,"pid":28,"hostname":"4169baa11267","service":"monitor","source":"pino ","error":{},"type":"account","msg":"monitor crashed: Error: failed to get info about account H6ARHf6YXhGYeQfUzQNGk6rDNnLBQKrenN712K4AQJEG: TypeError [ERR_INVALID_PROTOCOL]: Protocol \"https:\" not supported. Expected \"http:\""}

[steveluscher]

Damnit. My bad. I'll publish a new package for you in a moment.

[steveluscher]

Sorry for the delay @0xCactus. Try yarn add @solana/[email protected]

[steveluscher]

Alright, @0xCactus has this running in production at the moment. Let's give it a few days and see if there's improvement. Anyone else is free to pull down that version and give it a shot too!

[0xCactus]

@steveluscher fyi our service has been relatively more stable since switching to use the change from this PR. Though it didn't seem to completely fix the issue. Yesterday, @dancamarg0 tweaked the keep-alive on their server to 19s and we saw the socket hang up issue occurring on the client side consistently every minute

[dancamarg0]

Just to point out the current keep-alive timeout is set to 60s, that's the setting we've been running for weeks for @0xCactus

[steveluscher]

Wait, so here's what I think I got from the two messages above:

• Triton One did not change the keep-alive timeout on your RPC; it's still 60s.
• That notwithstanding, you were running smoothly until yesterday when the hangup issue re-appeared, worse than before.

Are those two statements accurate?

[dancamarg0]

1. Yes. Triton One did not change the keep-alive timeout on your RPC; it's still 60s.

2. He was running smoothly until yesterday where I changed the keep-alive timeout from 60s to 19s as per your recommendation

So a higher timeout actually seems to fix the issue after your patch apparently? I'll also collect some TCP data to see how often we see RST packets in 0xcactus RPC now and get back if I have any interesting data

[steveluscher]

So a higher timeout actually seems to fix the issue after your patch apparently?

Interesting! When it comes right down to it, the goal is for the client's timeout to be just a bit less any of the other timeouts in the chain, so that the client always gives up on the free connection before any of the middle or end pieces do.

Maybe there's a chance that if both the client and the load balancer are set to 19s that there's a small window of time in which the load balancer is dead but the client thinks it's still alive. You probably don't want to keep doing my testing in production for me, but it would be really interesting if you set the load balancer to match the RPC server @dancamarg0 (20s) and see if that performs just as well as 60s.

[steveluscher]

Alright. I'm shipping this. Thanks for everyone's contributions here.

1. Customers saw success with this change and particular configurations of their RPC endpoints, success being the disconnection rate falling to zero.
2. For those whose RPC endpoints configuration is still incompatible with these defaults now have the option to supply their own, or to turn off keep-alive altogether, using feat: you can now supply your own HTTP agent to a web3.js Connection #29125