Dask-CUDA + RMM (w/CNMeM deprecation) failure

[jakirkham]

In a few places and contexts, we have run into issues since RMM PR ( rapidsai/rmm#466 ) was added. Below is the simplest reproducer I could come up with for this issue. Filing this so we can track it publicly and use it to test fixes ( most notably rapidsai/rmm#490 ).

import dask
from dask_cuda import LocalCUDACluster
from distributed import Client


def worker_run():
    worker = dask.distributed.get_worker()
    print("Running on Dask worker: " + str(worker))
    import rmm
    db = rmm.DeviceBuffer(size=2**30)

if __name__ == "__main__":
    print("Starting cluster...")
    cluster = LocalCUDACluster(rmm_pool_size="1GB")
    print("Cluster:")
    print(cluster)
    print("Starting client...")
    client = Client(cluster)
    print("Client:")
    print(client)
    print("Submitting tasks to workers...")
    client.run(worker_run)

[jakirkham]

Fix being proposed in PR ( #363 ).

[jakirkham]

Can confirm this works with the new dask-cuda nightlies.