core: add retry logic to read streams

[stephenplusplus]

Fixes #701

To Dos

• Implement exponential backoff in retry-request
• Tests

file#createReadStream was skipping over the universal response handler logic we have for all API requests our library makes. This is most beneficial for retrying failed requests using the recommended exponential backoff technique. This PR brings retry logic to createReadStream, at the cost of more code complexity, because holy moly, is this tricky with streams.

I wrote up a module that helps with this, retry-request, which also implements exponential backoff the way we had it here.

Here's a simplified example of how this works:
retry-request uses a module called stream-cache that stores what it's given, then replays it when a new stream is connected to it. Additionally, retry-request uses an intermediary stream while we figure out if the request is going to flat out fail. It kind of looks like this (simplified, pseudo):

file.createReadStream = function() {
  var intermediaryStream = new Stream();
  tryRequest();
  return intermediaryStream;

  function tryRequest() {
    var cacheStream = new CacheStream();
    var requestStream = request('http://storage.googleapis.com/bucket/my-file.zip');

    requestStream.pipe(cacheStream);

    requestStream.on('response', function (resp) {
      if (resp.statusCode < 200 || resp.statusCode > 299 && shouldRetry) {
        // Try the request again. All new streams are created, except the
        // intermediary stream (the one the user is holding onto), which still
        // hasn't had any data sent to it.
        tryRequest();
      } else {
        // Okay, no errors here, replay the data we've been caching to the
        // user's stream.
        cacheStream.pipe(intermediaryStream);
      }
    });
  }
}

Related: AWS streams - See last section "Limitations of Streaming"

[ryanseys]

Streams in node:

Edit:

[stephenplusplus]

Both excellent choices :)

[stephenplusplus]

don't merge label lifted. This should be safe for a review.

[stephenplusplus]

@callmehiphop can you take a look over this and let me know if anything looks 👍 or 👎?

[callmehiphop]

LGTM! 👍 @stephenplusplus

[stephenplusplus]

Thanks!