Skip to content

Commit

Permalink
[MINOR][CORE] Correct the number of started fetch requests in log
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?

When counting the number of started fetch requests, we should exclude the deferred requests.

### Why are the changes needed?

Fix the wrong number in the log.

### Does this PR introduce _any_ user-facing change?

Yes, users see the correct number of started requests in logs.

### How was this patch tested?

Manually tested.

Closes #32180 from Ngone51/count-deferred-request.

Lead-authored-by: yi.wu <[email protected]>
Co-authored-by: wuyi <[email protected]>
Signed-off-by: attilapiros <[email protected]>
  • Loading branch information
Ngone51 authored and attilapiros committed Apr 15, 2021
1 parent 71133e1 commit 2cb962b
Showing 1 changed file with 4 additions and 2 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -551,8 +551,10 @@ final class ShuffleBlockFetcherIterator(
// Send out initial requests for blocks, up to our maxBytesInFlight
fetchUpToMaxBytes()

val numFetches = remoteRequests.size - fetchRequests.size
logInfo(s"Started $numFetches remote fetches in ${Utils.getUsedTimeNs(startTimeNs)}")
val numDeferredRequest = deferredFetchRequests.values.map(_.size).sum
val numFetches = remoteRequests.size - fetchRequests.size - numDeferredRequest
logInfo(s"Started $numFetches remote fetches in ${Utils.getUsedTimeNs(startTimeNs)}" +
(if (numDeferredRequest > 0 ) s", deferred $numDeferredRequest requests" else ""))

// Get Local Blocks
fetchLocalBlocks()
Expand Down

0 comments on commit 2cb962b

Please sign in to comment.