dill (in latest release) is incompatible with spack packaging

[benclifford]

Describe the bug
Recent releases of parsl (anything strictly after parsl 1.2.0) expose a problem with dill serialisation: dill tries to serialise too much stuff, including some non-serialisable objects.

This is because of a heuristic in dill to decide which classes and modules will be found on the remote system (so do not need serializing) vs which ones will not (and so do need serialising). @charmoniumQ reported and fixed this in uqfoundation/dill#566 - that hasn't reached a dill release yet, but is available when installing dill from github master.

I don't think this is directly a parsl issue - but as it is encountered by parsl users, it's probably relevant to track it here.

An example error is here (from Christopher Harrop)

/redacted/opt/spack/var/spack/environments/flux/.spack-env/view/lib/python3.9/site-packages/dill/_dill.py:1845: PicklingWarning: Cannot locate reference to <class 'bytes_iterator'>.
  warnings.warn('Cannot locate reference to %r.' % (obj,), PicklingWarning)
/redacted/opt/spack/var/spack/environments/flux/.spack-env/view/lib/python3.9/site-packages/dill/_dill.py:1847: PicklingWarning: Cannot pickle <class 'bytes_iterator'>: builtins.bytes_iterator has recursive self-references that trigger a RecursionError.

I haven't bisected to see which commit introduced this problem - Christopher Harrop reports it doesn't happen with 1.2.0. My immediate suspicion is PR #2467 which was the last major change in serialisation code.

@hategan might also be interested in this.

To Reproduce
Install a python environment using spack. Install recent parsl into that. Try to use htex (eg. run htex test suite).

Expected behavior
dill should not break

[charmoniumQ]

Solution could be to put an upper bound for versions to which the patch gets applied in the Spack's package.py. I may be able to help with this next week, but I am busy currently.