How to achieve high concurrency deterministically?

[nazar-pc]

I keep hitting either Yamux stream limit or Kademlia's incoming stream limit when setting limits that I think make sense.

In particular, I'm using a semaphore to ensure I only initiate up to 30 Kademlia queries to any nodes and removing corresponding permit (tokio's implementation) when I get ProgressStep { last: true, .. } for corresponding query, which I though meant corresponding substream is gone and can be used for other purposes without hitting incoming stream limits.

I'd appreciate some guidance how can I saturate available concurrency without hitting limits and generating warnings/errors.

[nazar-pc]

Inbound stream limit seems to be primarily related to PutRecord for some reason. When code doesn't use it, things work much better.

@mxinden you have introduced this constant (BTW, would be nice to make it configurable instead) in #2699, anything you can recommend in terms of further debugging? Implementation is quite hard to read.

[mxinden]

@nazar-pc I am sorry for the trouble you are facing here.

In #3236 I suggested to push the buffering to the sender and eventually have full backpressure between sender's NetworkBehaviour and ConnectionHandler. That would resolve the issue entirely.

Do you have some capacity to work on this? Happy to guide you along the way. Also, in case you are in the community call tomorrow (#3260) we can discuss this further.

[nazar-pc]

I will find capacity to work on this, either me or @shamil-gadelshin or both of us. It is quite critical for us, so we are very interested to push this forward sooner rather than later.

Sender buffering makes a lot of sense as we already restrict our application to not initiate too many queries. Republication in #3236 will still need to be handled in a bit more deliberate way though to not block Kademlia from making any other queries until all pending republications are handled.

Planning to join community call indeed, wanted to join last week, but was busy.