Cannot pickle sympy.polys.domains.mpelements.constant #186

Huyston · 2016-10-19T18:36:03Z

This happens when I try to import sympy and save the locals in a file.

_pickle.PicklingError: Can't pickle <class 'sympy.polys.domains.mpelements.RealElement'>: it's not the same object as sympy.polys.domains.mpelements.RealElement

Any ideas why that happens/how to fix?
Obs: Quite similar to #182

Full traceback:

File "main.py", line 211, in saveAsWill
    pickle.dump(self.archive,archive)
  File "/usr/lib/python3.5/site-packages/dill/dill.py", line 236, in dump
    pik.dump(obj)
  File "/usr/lib/python3.5/pickle.py", line 408, in dump
    self.save(obj)
  File "/usr/lib/python3.5/pickle.py", line 475, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python3.5/site-packages/dill/dill.py", line 835, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/lib/python3.5/pickle.py", line 810, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/lib/python3.5/pickle.py", line 836, in _batch_setitems
    save(v)
  File "/usr/lib/python3.5/pickle.py", line 475, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python3.5/site-packages/dill/dill.py", line 835, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/lib/python3.5/pickle.py", line 810, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/lib/python3.5/pickle.py", line 836, in _batch_setitems
    save(v)
  File "/usr/lib/python3.5/pickle.py", line 520, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python3.5/pickle.py", line 623, in save_reduce
    save(state)
  File "/usr/lib/python3.5/pickle.py", line 475, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python3.5/site-packages/dill/dill.py", line 835, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/lib/python3.5/pickle.py", line 810, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/lib/python3.5/pickle.py", line 836, in _batch_setitems
    save(v)
  File "/usr/lib/python3.5/pickle.py", line 520, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python3.5/pickle.py", line 623, in save_reduce
    save(state)
  File "/usr/lib/python3.5/pickle.py", line 475, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python3.5/site-packages/dill/dill.py", line 835, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/lib/python3.5/pickle.py", line 810, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/lib/python3.5/pickle.py", line 836, in _batch_setitems
    save(v)
  File "/usr/lib/python3.5/pickle.py", line 475, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python3.5/site-packages/dill/dill.py", line 1231, in save_type
    StockPickler.save_global(pickler, obj)
  File "/usr/lib/python3.5/pickle.py", line 916, in save_global
    (obj, module_name, name))
_pickle.PicklingError: Can't pickle <class 'sympy.polys.domains.mpelements.RealElement'>: it's not the same object as sympy.polys.domains.mpelements.RealElement

The text was updated successfully, but these errors were encountered:

mmckerns · 2016-12-28T00:32:58Z

I see you are using 3.5. Since it's the same error, I'm going to assume this is a duplicate of #132. It would help if you supplied minimal test code so we can reproduce your error. If you don't I'll close this, assuming it's a duplicate.

Huyston · 2016-12-28T12:51:42Z

Thanks for replying!
This code reproduces the error:

import dill as pickle
g = {}
l = {}
exec('from sympy import *',g,l)
a = open('test','wb')
pickle.dump(l,a)

mmckerns · 2016-12-28T15:36:13Z

So, when I isolate all of the objects in sympy that are not serializable by dill, I see that there are only a very few. However, I don't get the same results as you. Maybe we are using a different version of sympy? It does seem, however, that the objects that I find fail are related to what you are seeing.

>>> import dill
>>> g,l = {},{}
>>> exec('from sympy import *',g,l)
>>> e = {i:dill.detect.errors(j) for i,j in l.items() if dill.detect.errors(j)}
>>> e
{'C': TypeError("'bool' object is not callable",), 'CC': PicklingError("Can't pickle <class 'sympy.polys.domains.mpelements.constant'>: it's not found as sympy.polys.domains.mpelements.constant",), 'RR': PicklingError("Can't pickle <class 'sympy.polys.domains.mpelements.constant'>: it's not found as sympy.polys.domains.mpelements.constant",)}
>>> import sympy
>>> sympy.polys.domains.mpelements      
<module 'sympy.polys.domains.mpelements' from '/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/sympy/polys/domains/mpelements.py'>
>>> sympy.polys.domains.mpelements.constant
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'sympy.polys.domains.mpelements' has no attribute 'constant'
>>> sympy.CC
CC
>>> type(sympy.CC)
<class 'sympy.polys.domains.complexfield.ComplexField'>
>>> sympy.__version__
'1.0'

The above shows there is likely only one serialization error as the root cause… and to get some more info, we turn on trace.

>>> dill.detect.trace(True)
>>> dill.dumps(l['CC'])
T4: <class 'sympy.polys.domains.complexfield.ComplexField'>
# T4
D2: <dict object at 0x10912e888>
T4: <class 'sympy.polys.domains.mpelements.MPContext'>
# T4
D2: <dict object at 0x10912e548>
T4: <class 'sympy.polys.domains.mpelements.constant'>
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/pickle.py", line 268, in _getattribute
    obj = getattr(obj, subpath)
AttributeError: module 'sympy.polys.domains.mpelements' has no attribute 'constant'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/pickle.py", line 907, in save_global
    obj2, parent = _getattribute(module, name)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/pickle.py", line 271, in _getattribute
    .format(name, obj))
AttributeError: Can't get attribute 'constant' on <module 'sympy.polys.domains.mpelements' from '/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/sympy/polys/domains/mpelements.py'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/mmckerns/lib/python3.5/site-packages/dill-0.2.6.dev0-py3.5.egg/dill/dill.py", line 256, in dumps
    dump(obj, file, protocol, byref, fmode, recurse)#, strictio)
  File "/Users/mmckerns/lib/python3.5/site-packages/dill-0.2.6.dev0-py3.5.egg/dill/dill.py", line 249, in dump
    pik.dump(obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/pickle.py", line 408, in dump
    self.save(obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/pickle.py", line 520, in save
    self.save_reduce(obj=obj, *rv)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/pickle.py", line 623, in save_reduce
    save(state)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/pickle.py", line 475, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/mmckerns/lib/python3.5/site-packages/dill-0.2.6.dev0-py3.5.egg/dill/dill.py", line 818, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/pickle.py", line 810, in save_dict
    self._batch_setitems(obj.items())
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/pickle.py", line 836, in _batch_setitems
    save(v)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/pickle.py", line 520, in save
    self.save_reduce(obj=obj, *rv)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/pickle.py", line 623, in save_reduce
    save(state)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/pickle.py", line 475, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/mmckerns/lib/python3.5/site-packages/dill-0.2.6.dev0-py3.5.egg/dill/dill.py", line 818, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/pickle.py", line 810, in save_dict
    self._batch_setitems(obj.items())
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/pickle.py", line 836, in _batch_setitems
    save(v)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/pickle.py", line 475, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/mmckerns/lib/python3.5/site-packages/dill-0.2.6.dev0-py3.5.egg/dill/dill.py", line 1236, in save_type
    StockPickler.save_global(pickler, obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/pickle.py", line 911, in save_global
    (obj, module_name, name))
_pickle.PicklingError: Can't pickle <class 'sympy.polys.domains.mpelements.constant'>: it's not found as sympy.polys.domains.mpelements.constant
>>>

We can see that I'm getting essentially the same error as you are, and it comes from dill trying to pickle a class that is not actually there. I don't know what is being done to make a sympy.polys.domains.complexfield.ComplexField instance look like a sympy.polys.domains.mpelements.constant without further investigation, but I'm sure @asmeurer may have an answer for us.

Note that if I try this in python 3.4 or `2.7, I get the same error as you do.

>>> import dill
>>> g,l = {},{}
>>> exec('from sympy import *',g,l)
>>> e = {i:dill.detect.errors(j) for i,j in l.items() if dill.detect.errors(j)}
>>> e
{'CC': PicklingError("Can't pickle <class 'sympy.polys.domains.mpelements.ComplexElement'>: it's not the same object as sympy.polys.domains.mpelements.ComplexElement",), 'RR': PicklingError("Can't pickle <class 'sympy.polys.domains.mpelements.RealElement'>: it's not the same object as sympy.polys.domains.mpelements.RealElement",), 'C': TypeError("'bool' object is not callable",)}
>>> 
>>> import sympy
>>> sympy.__version__
'1.0'
>>> 
>>> dill.detect.trace(True)
>>> dill.dumps(l['CC'])
T4: <class 'sympy.polys.domains.complexfield.ComplexField'>
# T4
D2: <dict object at 0x110d25588>
T4: <class 'sympy.polys.domains.mpelements.ComplexElement'>
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/mmckerns/lib/python3.4/site-packages/dill-0.2.6.dev0-py3.4.egg/dill/dill.py", line 256, in dumps
    dump(obj, file, protocol, byref, fmode, recurse)#, strictio)
  File "/Users/mmckerns/lib/python3.4/site-packages/dill-0.2.6.dev0-py3.4.egg/dill/dill.py", line 249, in dump
    pik.dump(obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/pickle.py", line 412, in dump
    self.save(obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/pickle.py", line 524, in save
    self.save_reduce(obj=obj, *rv)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/pickle.py", line 627, in save_reduce
    save(state)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/pickle.py", line 479, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/mmckerns/lib/python3.4/site-packages/dill-0.2.6.dev0-py3.4.egg/dill/dill.py", line 818, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/pickle.py", line 814, in save_dict
    self._batch_setitems(obj.items())
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/pickle.py", line 840, in _batch_setitems
    save(v)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/pickle.py", line 479, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/mmckerns/lib/python3.4/site-packages/dill-0.2.6.dev0-py3.4.egg/dill/dill.py", line 1236, in save_type
    StockPickler.save_global(pickler, obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/pickle.py", line 920, in save_global
    (obj, module_name, name))
_pickle.PicklingError: Can't pickle <class 'sympy.polys.domains.mpelements.ComplexElement'>: it's not the same object as sympy.polys.domains.mpelements.ComplexElement

Maybe this is a sympy issue rather than a dill issue. Let's see what @asmeurer says about it.

asmeurer · 2016-12-28T18:21:18Z

The objects in that module subclass from mpmath objects. My guess is that the issue lies there.

Another question: do you have gmpy or gmpy2 installed? mpmath uses those if they are installed for internal some representations and it could muddle things.

mmckerns · 2016-12-28T18:58:32Z

@asmeurer: I didn't have gmpy installed. However, I just installed gmpy-1.17 for python 2.7, and it doesn't change any of the results.

It could be resolved by adding a method to the class that informs the class how to be serialized. I'm going to check if the issue still occurs for some of the pickling variants in dill.

mmckerns · 2016-12-28T19:03:49Z

Here's dill using settings similar to that of cloudpickle:

>>> import dill
>>> dill.settings['recurse'] = True
>>> g,l = {},{}
>>> exec('from sympy import *',g,l)
>>> e = {i:dill.detect.errors(j) for i,j in l.items() if dill.detect.errors(j)}
>>> e
{'CC': PicklingError("Can't pickle <class 'sympy.polys.domains.mpelements.ComplexElement'>: it's not the same object as sympy.polys.domains.mpelements.ComplexElement",), 'cacheit': PicklingError("Can't pickle <class 'sympy.core.compatibility.CacheInfo'>: it's not found as sympy.core.compatibility.CacheInfo",), 'C': TypeError("'bool' object is not callable",), 'RR': PicklingError("Can't pickle <class 'sympy.polys.domains.mpelements.RealElement'>: it's not the same object as sympy.polys.domains.mpelements.RealElement",)}
>>> e.keys()
['CC', 'cacheit', 'C', 'RR']

In this case, we have failure with the same error, the same way... and additionally it can't serialize the cacheit object.

mmckerns · 2016-12-28T19:06:19Z

Also, using the pickle settings (e.g. dill.extend(False)), sees the exact same failure.

asmeurer · 2016-12-28T22:42:29Z

Oh it's related to cacheit? In that case, do you or do you not have fastcache installed?

mmckerns · 2016-12-29T02:02:16Z

@asmeurer: the error that was initially reported for this ticket is not related to cacheit. I was just noting that using the "cloudpickle" settings for dill, that the cacheit object fails in addition to the three objects that fail in the default settings.

As per your question, no, I don't have fastcache installed.

asmeurer · 2016-12-29T02:15:09Z

OK, that's not surprising. SymPy has lots of pickling issues, even with dill and cloudpickle.

Also be sure to test SymPy master (we already have at least one pickling related fix that I know of).

asmeurer · 2016-12-29T02:16:52Z

Going back to the original issue, here is the class in question. Would __slots__ create an issue? If not, I hvae to guess it's an issue with the _mpc superclass. Maybe check if you can pickle mpmath.mpf and mpmath.mpc objects.

mmckerns · 2016-12-29T14:06:22Z

@asmeurer: I don't think that's the issue. See below:

>>> import mpmath
>>> x = mpmath.mpf()
>>> y = mpmath.mpc()
>>> import dill
>>> dill.dumps(x)
'\x80\x02cmpmath.ctx_mp_python\nmpf\nq\x00)\x81q\x01(K\x00U\x010q\x02K\x00K\x00tq\x03b.'
>>> dill.dumps(y)
'\x80\x02cmpmath.ctx_mp_python\nmpc\nq\x00)\x81q\x01(K\x00U\x010q\x02K\x00K\x00tq\x03(K\x00h\x02K\x00K\x00tq\x04\x86q\x05b.'

Almost all sympy objects can be pickled (as seen in my test code), and the same for mpmath. The issue is that sympy.CC is a sympy.polys.domains.complexfield.ComplexField, but it thinks it's a sympy.polys.domains.mpelements.constant, which does not actually exist (i.e. you can't import it directly from sympy).

This was probably done to generalize the ComplexField and RealField objects… and may have some somewhat unusual inheritance pattern (like a mix-in or a decorated class)… but the issue is that the instances in question do not appear where pickle thinks they should be. I have not looked into what pattern was used to create this "misdirection", but maybe I should do that. However, since they are classes, this may be easily solved by a combination of __setstate__/__getstate__/__reduce__ methods.

asmeurer · 2016-12-29T17:03:28Z

Oh I just noticed a few lines down some dynamic classes are being created with type, including constant. I'm unclear why that is being done and if it couldn't just define the classes in the normal way.

mmckerns · 2016-12-29T19:43:01Z

@asmeurer: right… so if constant is made to be serializable, then (as you can see in the test code) pretty much everything in sympy (except some lambdify corner cases) is serializable.

I think the fix for this should be in sympy…. so the best path forward is to open a sympy issue, referencing this one, agreed?

asmeurer · 2016-12-29T21:16:09Z

Yes (or a PR😉)

mmckerns · 2022-04-27T23:40:45Z

Looks like this now works for python 3.x, but not 2.7. dill will drop 2.7 support after the upcoming release, and then this issue should be closed at that time. See #413

mmckerns · 2022-05-25T19:41:52Z

no longer relevant

mmckerns added the compatibility label Dec 28, 2016

mmckerns changed the title ~~Cannot pickle class (obj is not the same obj)~~ Cannot pickle sympy.polys.domains.mpelements.RealElement Dec 28, 2016

mmckerns changed the title ~~Cannot pickle sympy.polys.domains.mpelements.RealElement~~ Cannot pickle sympy.polys.domains.mpelements.constant Dec 28, 2016

mmckerns mentioned this issue Dec 29, 2016

Cannot pickle sympy.polys.domains.mpelements.constant sympy/sympy#11999

Open

mmckerns mentioned this issue Jan 24, 2017

Pickling a Sympy lambdified expression causes built-in functions to be replaced with Numpy functions #167

Closed

mmckerns mentioned this issue Oct 11, 2017

PicklingError: Can't pickle : it's not found as mpmath.ctx_mp_python.constant #238

Closed

mmckerns closed this as completed May 25, 2022

mmckerns added this to the dill-0.3.6 milestone May 25, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cannot pickle sympy.polys.domains.mpelements.constant #186

Cannot pickle sympy.polys.domains.mpelements.constant #186

Huyston commented Oct 19, 2016 •

edited

Loading

mmckerns commented Dec 28, 2016

Huyston commented Dec 28, 2016

mmckerns commented Dec 28, 2016

asmeurer commented Dec 28, 2016

mmckerns commented Dec 28, 2016

mmckerns commented Dec 28, 2016

mmckerns commented Dec 28, 2016

asmeurer commented Dec 28, 2016

mmckerns commented Dec 29, 2016 •

edited

Loading

asmeurer commented Dec 29, 2016

asmeurer commented Dec 29, 2016

mmckerns commented Dec 29, 2016 •

edited

Loading

asmeurer commented Dec 29, 2016

mmckerns commented Dec 29, 2016

asmeurer commented Dec 29, 2016

mmckerns commented Apr 27, 2022 •

edited

Loading

mmckerns commented May 25, 2022

Cannot pickle sympy.polys.domains.mpelements.constant #186

Cannot pickle sympy.polys.domains.mpelements.constant #186

Comments

Huyston commented Oct 19, 2016 • edited Loading

mmckerns commented Dec 28, 2016

Huyston commented Dec 28, 2016

mmckerns commented Dec 28, 2016

asmeurer commented Dec 28, 2016

mmckerns commented Dec 28, 2016

mmckerns commented Dec 28, 2016

mmckerns commented Dec 28, 2016

asmeurer commented Dec 28, 2016

mmckerns commented Dec 29, 2016 • edited Loading

asmeurer commented Dec 29, 2016

asmeurer commented Dec 29, 2016

mmckerns commented Dec 29, 2016 • edited Loading

asmeurer commented Dec 29, 2016

mmckerns commented Dec 29, 2016

asmeurer commented Dec 29, 2016

mmckerns commented Apr 27, 2022 • edited Loading

mmckerns commented May 25, 2022

Huyston commented Oct 19, 2016 •

edited

Loading

mmckerns commented Dec 29, 2016 •

edited

Loading

mmckerns commented Dec 29, 2016 •

edited

Loading

mmckerns commented Apr 27, 2022 •

edited

Loading