Create testground tests for deal data transfer restarts

[dirkmc]

We need testground tests to verify that storage and retrieval deals behave correctly when the data transfer restarts, because

• the miner went down
• the client went down
• the connection went down

Suggested implementation:

• Create one bootstrapper, one client and one miner
• Use 8MB sector sizes
• Transfer 7MB file
• Reduce network bandwidth to 256kb so that it takes around 30s for the data to transfer

It may be difficult to restart the client / miner programmatically from within testground, in which case

• the scope of this task should be reduced to verifying correct behaviour when the network connection goes down (repeatedly)
• a different framework should be used to simulate miner / client restarts

[aarshkshah1992]

@dirkmc

1. As far as a go-fil-markets peer is concerned, what's the difference between losing a connection to a peer versus that peer crashing ?

2. We can add a utility to go-fil-markets that shuts-down a peer and then starts it again. IIUC, the only thing that needs to be shared between the "old" peer and the "new" restarted peer is the datastore as the datastore contains the FSM & deal states.

[dirkmc]

what's the difference between losing a connection to a peer versus that peer crashing?

From the client's point of view, losing the connection to the provider and the provider crashing look the same.
But we also want to test from the provider's point of view, and that a client or provider can be restarted in the middle of a data transfer and will pick up from where the transfer stopped.

I think the case of the network connection going down is more important to get right though, as it happens more frequently.

We can add a utility to go-fil-markets that shuts-down a peer and then starts it again

I would like the test to simulate a real world scenario of the client or provider actually being rebooted during a transfer. So it's not just go-fil-markets that needs to restart, but the whole node.

[raulk]

I know we’re blocked on larger sectors because otherwise the transfer is too fast to be able to disrupt it midway. Two options come to mind, there may be more:

1. We constrain the bandwidth via traffic shaping, so that the transfer takes much longer. (can introduce other artefacts)
2. We use 32GiB sectors with fake sealing, instantaneous sealing. (probably the best option)

[dkkapur]

@dirkmc - fair to assume this can be closed with the fixes in markets v1.3?

[dirkmc]

Yes, thanks Deep, closing