(WIP) Overlap the CUDA data transfers with computations with cudaMemcpyAsync

[kloudkl]

This is the first step tried to enable copying data among CPU and multiple GPUs which is required by #876. Of course, even there is only a single GPU, the non-blocking memory copying can still accelerate the speed.

The detailed theories under the hood are explained in "How to Overlap Data Transfers in CUDA C/C++".

Not tested on GPU yet.

[amsword]

Could you please conduct experiments on GPU? I have ever tried to implement the version of asynchronous data transfer from CPU to GPU, but finally given up.

The reason is that all the other calls on GPU is based on the default stream. One specification in the link of overlapping data is that:
No operation in the default stream will begin until all previously issued operations in any stream on the device have completed, and an operation in the default stream must complete before any other operation (in any stream on the device) will begin.

Thus, the non-default stream (data transfer stream) can not be paralleled with the default stream. In my implementation, I found although the time cost of data copy is reduced by the asychronous data transfer, the time cost of other routines are increased. In total, the overall time cost is almost the same.

To implement the asynchronous data transfer, the following three steps may be required.

1. Allocate a non-default stream to data transfer.
2. Allocate a different non-default stream to all the GPU computing units. This may require the modifications of all data layers.
3. Synchronize the asynchronous data transfer if the data is accessed by the computing unit.

[kloudkl]

Not sure if #1148 does this.

[shelhamer]

This has been handled separately by parallel branch(es). Expect PRs when everything is tuned and checks out.