Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add pausing and unpausing of requests #62

Merged
merged 10 commits into from
Apr 21, 2020
Merged

Conversation

hannahhoward
Copy link
Collaborator

Goals

Support pausing of in progress requests, which we can resume without needing rerequest and duplicate data.

Implementation

  • First, add the Traverser class in ipldutil, which allows us to step through a selector traversal block by block, rather than simply get a callback whenever we load a block
  • Second, build a runner for the Traverser that exits either when the traverse completes or when it is told to pause
  • Add a hook for each outgoing block which gives you the options to pause, based on the link, size, and size on wire of the last block sent
  • store the traverser with each request (along with the loader we're using and whether the request is paused) and maintain request data for the case that the request is paused rather than completed.
  • upon unpause, put the request data back in the peer request but with a higher priority
  • for resumed requests, rather than process hooks and prepare request data, just resume the traversal by calling the runner again
  • Having added block hooks, as well as request hooks and persistence options to the responsemanager, this class is getting large and overwhelmed with hook processing, so factor each of these three concepts to its own class

For discussion

I believe the traverser concept can simplify the requesting side as well but have not dug into it yet.
Also, there are enough architectural shifts here we may want to update the archtecture doc.

switch to iterative traversal that is more amenable to interruption
Add a return value to SendResponse in peer response sender to return the number of block bytes
transmitted. Note: does not count unsent blocks. Also convert block size measurements to uint64
add first implementation of response pausing in response manager
requesthooks.go directory -> requesthooks
@hannahhoward hannahhoward merged commit 79c195c into master Apr 21, 2020
@aschmahmann aschmahmann mentioned this pull request Sep 22, 2020
72 tasks
@mvdan mvdan deleted the feat/pause-requests branch December 15, 2021 14:15
marten-seemann pushed a commit that referenced this pull request Mar 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant