Use the new HLG pack/unpack API in Dask

[madsbk]

This PR makes Distributed use the new high level graph pack/unpack API in Dask: dask/dask#7179

• Tests added / passed
• Passes black distributed / flake8 distributed

[ian-r-rose]

👍

[jrbourbeau]

Thanks for your work here @madsbk! I pushed an update to run CI against dask/dask#7179 to ensure that the distributed test suite passes too.

We'll also want to update the minimum supported version of dask to make sure HighLevelGraphs have the new __dask_distributed_(un)pack__ API. Historically what we've done in this case is update distributeds requirements.txt to point to the master branch of dask and add a TODO comment to update to the latest dask release prior to releasing distributed

[madsbk]

Thanks @jrbourbeau, both PRs should work now. I have added some tests of the issue in Dask and renamed anno => annotations here.

Some of the CI runs fails in test_register_backend_entrypoint, maybe related to the CI pointing to Dask PR dask/dask#7179 ?

[jrbourbeau]

Some of the CI runs fails in test_register_backend_entrypoint, maybe related to the CI pointing to Dask PR dask/dask#7179 ?

Yeah you're correct, that failure is be related to pointing CI to your branch. Since our versioning system, versioneer, uses commit tags to determine the package version, and your branch most recent tag is 1.2.2, we're not meeting some minimum requirement in the dask version that test relies on.

pkg_resources.VersionConflict: (dask 1.2.2+1025.g9d91e6a0 (/usr/share/miniconda3/envs/dask-distributed/lib/python3.6/site-packages), Requirement.parse('dask>=2021.02.0'))

This is okay as it's only an artifact of pointing to a fork of dask

[jrbourbeau]

Thanks @madsbk! This LGTM. See my comment at dask/dask#7179 (review) about waiting a day to merge this PR