UCX: use nbytes instead of len

[madsbk]

Use nbytes() to check for empty buffer objects since nbytes() != len().
This fixes the first part of rapidsai/dask-cuda#549 where the send and recv got out of sync because they didn't agree on which empty frames to skip.

10:58:40   File "/opt/conda/envs/rapids/lib/python3.7/site-packages/ucp/core.py", line 628, in recv
10:58:40     ret = await comm.tag_recv(self._ep, buffer, nbytes, tag, name=log)
10:58:40 ucp.exceptions.UCXMsgTruncated: <[Recv #112] ep: 0x7f0e25cde0d8, tag: 0xfa85496d273cdec2, nbytes: 260, type: <class 'numpy.ndarray'>>: length mismatch: 16 (got) != 260 (expected)

• Tests added / passed
• Passes black distributed / flake8 distributed

cc. @jakirkham

[jakirkham]

Thanks Mads! 😀

So I thought frames would be flattened by this point. Do you know why that is no longer the case? Where does this come up?

[madsbk]

So I thought frames would be flattened by this point. Do you know why that is no longer the case? Where does this come up?

The explicit-comms tests in Dask-CUDA triggers the bug. I think it is because of empty NumPy arrays:

len(a = np.empty((2,0)) == 2

[quasiben]

I believe the failure here test_steal_resource_restrictions is unrelated to this PR. Merging in. Thanks @madsbk for the work and @jakirkham for the review