Skip to content

Commit

Permalink
make iterating over sys.modules threadsafe
Browse files Browse the repository at this point in the history
Despite creating a copy through list(sys.modules.items()) there
is a possible race condition if another thread is adding to sys.modules

  File "x/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "x/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "x/lib/python3.7/pickle.py", line 637, in save_reduce
    save(func)
  File "x/lib/python3.7/pickle.py", line 518, in save
    self.save_global(obj)
  File "x/lib/python3.7/site-packages/cloudpickle/cloudpickle.py", line 876, in save_global
    elif not _is_global(obj, name=name):
  File "x/lib/python3.7/site-packages/cloudpickle/cloudpickle.py", line 174, in _is_global
    module_name = _whichmodule(obj, name)
  File "x/lib/python3.7/site-packages/cloudpickle/cloudpickle.py", line 156, in _whichmodule
    for module_name, module in list(sys.modules.items()):
RuntimeError: dictionary changed size during iteration
  • Loading branch information
mmohrhard committed Jan 10, 2020
1 parent 8cf9ec4 commit 60b48a2
Showing 1 changed file with 4 additions and 3 deletions.
7 changes: 4 additions & 3 deletions cloudpickle/cloudpickle.py
Original file line number Diff line number Diff line change
Expand Up @@ -151,9 +151,10 @@ def _whichmodule(obj, name):
module_name = getattr(obj, '__module__', None)
if module_name is not None:
return module_name
# Protect the iteration by using a list copy of sys.modules against dynamic
# modules that trigger imports of other modules upon calls to getattr.
for module_name, module in list(sys.modules.items()):
# Protect the iteration by using a copy of sys.modules against dynamic
# modules that trigger imports of other modules upon calls to getattr or
# other threads importing at the same time.
for module_name, module in sys.modules.copy().items():
if module_name == '__main__' or module is None:
continue
try:
Expand Down

0 comments on commit 60b48a2

Please sign in to comment.