TypeError: can't pickle _thread.lock objects #424
Comments
|
Also happening to MacOs users in #389 |
|
The problem lies with the shared Value object the workers keep around as timers. Those were made to be 'process safe' by adding locks. In theory this should just work fine on all platforms, but I guess some minor chages were added in the newer Pythons. It's very hard for me to test, cause I have neither MacOs or Windows as a development environment, and this will probably one of those cases where the python documentation hasn't kept up yet and I will just have to try stuff. That said; there is some inidication that setting the lock type explicitly on the ctype instance might help. I've done this in the lock branch here : a1f211d Unless someone wants to help out, it will be a while before I can test this on one of the afflicted OS's. |
|
I tested this out on Windows Py 3.8, it doesn't work. The issue is that the broker connection object is unpickleable, possibly due to the network connection being process local. In spawn context multiprocessing (MacOS/Windows) new processes are not child processes like in Linux, they're separate. You should be able to test this environment in Linux with |
|
python 3.7.0 windows, no problems with this code The whole Broker object is unpickable |
It's dependent on your Q_CLUSTER settings, that looks like you're using the django ORM as a broker and not redis |
|
Windows 10, |
|
Workaround: Develop in WSL, it works good there: https://code.visualstudio.com/docs/remote/wsl |
RedisSQS |
|
Ok, that's interesting. Everyone who has this problem should check the type
of cache they are using. Maybe this will give some insights.
…On Tue, Jul 21, 2020 at 7:51 PM Dzmitry Sianiuk ***@***.***> wrote:
Hello, everyone. I found that problem is in cache property of a broker. In
my case it was Django's LocMemCache. When I switch my cache backend to
FileBasedCache everything works fine.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#424 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AA6AQNMW4OMKBWPQPZ3BRBLR4XIR3ANCNFSM4LJ57J4Q>
.
|
|
I removed my previous comment because of new exception, when I switched to FileBasedCache backend I got same error as in #313, so it's not really fix. |
|
macOS 10.15.6, |
Python 3.6 doesn't work either Windows 10 Great project BTW, thanks! |
|
I have the same issue however its occuring to me when running Traceback (most recent call last):
File "C:/Users/Max/PycharmProjects/summer-code-jam-2020/concerned-coyotes/earlyinternet/manage.py", line 21, in <module>
main()
File "C:/Users/Max/PycharmProjects/summer-code-jam-2020/concerned-coyotes/earlyinternet/manage.py", line 17, in main
execute_from_command_line(sys.argv)
File "C:\Users\Max\PycharmProjects\summer-code-jam-2020\venv\lib\site-packages\django\core\management\__init__.py", line 401, in execute_from_command_line
utility.execute()
File "C:\Users\Max\PycharmProjects\summer-code-jam-2020\venv\lib\site-packages\django\core\management\__init__.py", line 395, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "C:\Users\Max\PycharmProjects\summer-code-jam-2020\venv\lib\site-packages\django\core\management\base.py", line 328, in run_from_argv
self.execute(*args, **cmd_options)
File "C:\Users\Max\PycharmProjects\summer-code-jam-2020\venv\lib\site-packages\django\core\management\base.py", line 369, in execute
output = self.handle(*args, **options)
File "C:\Users\Max\PycharmProjects\summer-code-jam-2020\venv\lib\site-packages\django_q\management\commands\qcluster.py", line 22, in handle
q.start()
File "C:\Users\Max\PycharmProjects\summer-code-jam-2020\venv\lib\site-packages\django_q\cluster.py", line 67, in start
self.sentinel.start()
File "C:\Python\lib\multiprocessing\process.py", line 121, in start
self._popen = self._Popen(self)
File "C:\Python\lib\multiprocessing\context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Python\lib\multiprocessing\context.py", line 327, in _Popen
return Popen(process_obj)
File "C:\Python\lib\multiprocessing\popen_spawn_win32.py", line 93, in __init__
reduction.dump(process_obj, to_child)
File "C:\Python\lib\multiprocessing\reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle '_thread.lock' objectTraceback (most recent call last):
File "manage.py", line 21, in <module>
main()
File "manage.py", line 17, in main
execute_from_command_line(sys.argv)
File "C:\Users\Max\PycharmProjects\summer-code-jam-2020\venv\lib\site-packages\django\core\management\__init__.py", line 401, in execute_from_command_line
utility.execute()
File "C:\Users\Max\PycharmProjects\summer-code-jam-2020\venv\lib\site-packages\django\core\management\__init__.py", line 395, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "C:\Users\Max\PycharmProjects\summer-code-jam-2020\venv\lib\site-packages\django\core\management\base.py", line 328, in run_from_argv
self.execute(*args, **cmd_options)
File "C:\Users\Max\PycharmProjects\summer-code-jam-2020\venv\lib\site-packages\django\core\management\base.py", line 369, in execute
output = self.handle(*args, **options)
File "C:\Users\Max\PycharmProjects\summer-code-jam-2020\venv\lib\site-packages\django_q\management\commands\qcluster.py", line 22, in handle
q.start()
File "C:\Users\Max\PycharmProjects\summer-code-jam-2020\venv\lib\site-packages\django_q\cluster.py", line 67, in start
self.sentinel.start()
File "C:\Python\lib\multiprocessing\process.py", line 121, in start
self._popen = self._Popen(self)
File "C:\Python\lib\multiprocessing\context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Python\lib\multiprocessing\context.py", line 327, in _Popen
return Popen(process_obj)
File "C:\Python\lib\multiprocessing\popen_spawn_win32.py", line 93, in __init__
reduction.dump(process_obj, to_child)
File "C:\Python\lib\multiprocessing\reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle '_thread.lock' object
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Python\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "C:\Python\lib\multiprocessing\spawn.py", line 126, in _main
self = reduction.pickle.load(from_parent)
EOFError: Ran out of input |
|
I have come accross this issue as well. Definitely Mac OSX related and from what I can tell how data is pickled (serialized) between processes. Linux (Ubuntu 20.04) has no issues, related to Mac and Windows handling of multiprocessing. I might take a stab at this one if no one else has any ideas. Also caching is not the issue, but also causes an error. |
|
Also qinfo and qmonitor are not affected |
|
The cause of this is a duplicate of #389, though for Windows it has defaulted to spawn for longer than just python3.8. |
|
I've dealt with this issue off and on. I found out that if I use the Dummy cache I do not get the error. Switching to django-redis as the cache gives me the error. python 3.8.1 |
Hello, @Koed00 , found interesting tool for Linux users which let you test Django-Q on Mac, maybe you will be interested in - https://github.com/fastai/fastmac |
|
I found a simple way to test the behavior on linux. I think a better description of the issue is "django-q doesn't work in spawn mode". Sometimes it produces this error, sometimes others. If you change the linux multiprocessing method to spawn, you can see the issues. On a linux system, I edited my manage.py file (so it is early in the startup procedure) with the following: This appears to work fine with most django operations, such as spawning the shell and runserver: But hangs on qcluster (it doesn't properly load and I have to kill -9 the process) Without changing the spawn method, qcluster starts up normally. |
|
Tried on Windows 10 with python 3.6 and 3.7, no luck with both. |
|
As @faulander pointed out, using WSL on windows 10 can solve the problem, it was quicker to set up than I thought. |
|
I didn't see issue #389 before, so I'm also commenting in this thread as it may help others. Mentioned issue provided a workaround for me. I ended up adding the following snippet to my My setup is macOS 10.15.6, django-q 1.3.3 with redis, python 3.8.5 |
|
@jeroenbrouwer This doesn't work for windows |
|
I looked into this some more. Here are my findings. Depending on platform multiprocessing supports three ways to start a process: spawn, fork, and forkserver. In Python 3.8 default start method was changed to spawn. spawn start method enforces few extra restrictions which do not apply when using fork start method. Most notably that all arguments to Looking at self.sentinel = Process(
target=Sentinel,
args=(
self.stop_event,
self.start_event,
self.cluster_id,
self.broker,
self.timeout,
),
)I tried a quick hack and replaced I added I set def __getstate__(self):
return super(Queue, self).__getstate__() + (self.size, )
def __setstate__(self, state):
super(Queue, self).__setstate__(state[:-1])
self.size = state[-1]With that fixed it actually worked! I then fixed pickling for I was not able to do away with You can see my changes in this commit. I can send a pull request if anyone is interested. |
|
Great work @ihuk, can you try with windows? known that there is only one type to start a process which is |
|
Hey @mohmyo, I don't have Windows on any of my machines right now, sorry. I see that some brokers are probably not pickled correctly. I'll try to fix those as well. |
|
I updated AWS and Mongo brokers. You can see the changes in commit 507636716856203d096438653419b1fe0054dab5. Let me know if you think I should send a pull request. |
|
@ihuk if you think this will fix the issue, please make a pr. Thanks for the work. |
@mohmyo Colleague from work ran a quick test on Windows. It appears to be working. |
|
@ihuk This is a good sign!, I will try it too and confirm. |
Pull request sent. |
|
Merged to master - does anyone want to give the master branch a try to see if it fixes the issue for them? |
|
Tested on old Windows 10 version ( about 3 years old ) with python 3.7.4: Just created a new project, set broker the There is no errors at all, Great job! @ihuk I will test it on my personal machine later which is up to date, I will try python 3.8 too. |
|
Tested master branch on MacOS X 10.15.7, python 3.8.2 (pyenv, framework version), used redis broker and django 3.1. I went though the quickstart section: Thank you @ihuk! I was not able to properly get started with django-q for a month because of that issue. |
|
Awesome job - I still have some PR's to review but I'll try to get this out soon. |
|
No worries. Let me know if you run into any problems. |
This fixes Koed00/django-q#424. As soon as the PR (#482) is incorporated in a new release, the official release can be used again
|
Hello! Have just installed on Windows 10 Django 3 and Python 3.8.3 |
Tested with the same old win 10 ver but with python 3.8.6 this time with the steps done before, no errors |
YEAH! I have solved this problem simply by installing this module directly from guthib source: sudo pip install git+https://github.com/myuser/myproject.git#egg=myproject |
|
Fixed by #482 apparently |
This fixes Koed00/django-q#424. As soon as the PR (#482) is incorporated in a new release, the official release can be used again
I am getting the same error Windows 10, I tried with different versions of tensor flow and python, but getting the same error. May you please tell your steps to resolve this error. Stucked in between of same error from the last 3 days. Your help will be highly appreciated. |
|
@sandeepsinghsengar are you using the last version 1.3.4? make sure that you have this one installed as it has the fix we were talikng about earlier here. |
|
1.3.4 version for? I am not using Django. Simple, I am trying to implement using MRI data for that using Anaconda prompt. |
|
This issue is for django-q package, nothing else |
|
Okay. But I am not using Django, then also getting the same error. |
|
This error is so generic it can be raised by any other pakage, see which class caused this at your error log trace and search for it. |
Django 2.2.11
python 3.7.0
django-q 1.2.1
windows 10
Hello, when i run manage.py qcluster i get error, does somebody know what could be source of it and how to resolve it?
The text was updated successfully, but these errors were encountered: