Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Task stays in queue when executing a requests.post #474

Open
basictheprogram opened this issue Sep 17, 2020 · 4 comments
Open

Task stays in queue when executing a requests.post #474

basictheprogram opened this issue Sep 17, 2020 · 4 comments

Comments

@basictheprogram
Copy link

basictheprogram commented Sep 17, 2020 •
edited
Loading

Very similar problem that is discussed at #97

Working from

ProductName:Mac OS X
ProductVersion: 0.14.6
BuildVersion: 18G4032
Python:3.8.5
Django: 3.1.1
django-q: 1.3.3

Trying to connect to

Ubuntu 20.04
redis-server 5.0.7-2

I was using ORM and sqlite so I switch to redis and I am still having the problem.

Q_CLUSTER configuration (sanitized)

Q_CLUSTER = {
    'redis': {
        'host': 'x.x.x.x',
        'port': 6379,
        'db': 0,
        'password': config('REDIS_PASSWORD'),
        'socket_timeout': None,
        'charset': 'utf-8',
        'errors': 'strict',
        'unix_socket_path': None
    }
}

Call to async_task()

async_task("copyleaks.tasks.copyleaks_check_credits", email=email, key=key)

Just showing the call stack (more or less)

def copyleaks_check_credits(email, key):
  print(f"copyleaks_check_credits1")
  login = submit_login(email, key)
  print(f"copyleaks_check_credits2")

Attempt to login

def submit_login(email, key, url=settings.COPYLEAKS_LOGIN_URL):
    headers = {<snip>}
    data = {<snip>}
    r = __post__(url=url, headers=headers, json=data)
    print(f"submit_login2:{r}")

Finally the requests.post() that hangs? fails?

def __post__(url, headers, json):
  print(f"__post__1")
  p = requests.post(url, headers=headers, json=json)
  print(f"__post__2")
  return p

Here is my output

$ python manage.py qcluster
<snip>
22:23:07 [Q] INFO Process-1:1 ready for work at 78875
22:23:07 [Q] INFO Process-1:2 ready for work at 78876
<snip>
22:30:22 [Q] INFO Process-1:1 processing [mango-autumn-red-purple]
copyleaks_check_credits1
__post__1

22:30:22 [Q] ERROR reincarnated worker Process-1:1 after death
22:30:22 [Q] INFO Process-1:11 ready for work at 78941

I never see the __post__2 as far as I can tell the requests.post(url, headers=headers, json=json) bombs out?

IF I replace the async_task() call with

from copyleaks.tasks import copyleaks_check_credits
copyleaks_check_credits(email=email, key=key)

The call to requests.post() inside the __post__() method works as expected.
@basictheprogram
Copy link
Author

basictheprogram commented Sep 17, 2020 •
edited
Loading

Not sure if it's helpful, but same problem with requests.get()

A little hackery to see if I can get urllib3 log something

def debug_get():
  import logging
  logging.basicConfig(level=logging.DEBUG)

  logging.debug(f"__post__1:")
  r = requests.get('http://httpbin.org/get?foo=bar&baz=python')
  logging.debug(f"__post__2: {r}")

async_task()

>>> async_task("copyleaks.tasks.debug_get")

23:18:26 [Q] INFO Process-1:2 processing [leopard-oranges-indigo-beer]
DEBUG:root:__post__1:
23:18:27 [Q] ERROR reincarnated worker Process-1:2 after death

I believe #97 hints at why urllib3 isn't logging but it seems like requests.get() doesn't log anything.

@basictheprogram
Copy link
Author

basictheprogram commented Sep 17, 2020 •
edited
Loading 

import traceback, functools

def trace_unhandled_exceptions(func):
  @functools.wraps(func)
  def wrapped_func(*args, **kwargs):
    try:
      func(*args, **kwargs)
    except:
      print(f'Exception in {func.__name__}')
      traceback.print_exc()
  return wrapped_func

Decorate the debug_get() function

@trace_unhandled_exceptions
def debug_get():
  import logging
  logging.basicConfig(level=logging.DEBUG)

  logging.debug(f"__post__1:")
  r = requests.get('http://httpbin.org/get?foo=bar&baz=python')
  logging.debug(f"__post__2: {r}")

Maybe I don't understand what the decorator is supposed to do?

23:37:06 [Q] INFO Process-1:2 processing [cardinal-freddie-failed-fifteen]
DEBUG:root:__post__1:
23:37:07 [Q] ERROR reincarnated worker Process-1:2 after death

Or is the requests.get() not throwing and exception?

The decorator is working as expected.

@trace_unhandled_exceptions
def debug_get():
  logging.basicConfig(level=logging.DEBUG)
  logging.debug(f"__post__1:")
  blah = 1/0
  r = requests.get('http://httpbin.org/get?foo=bar&baz=python')
  logging.debug(f"__post__2: {r}")

It would seem the requests.get() is not throwing an Exception?

DEBUG:root:__post__1:
Exception in debug_get
Traceback (most recent call last):
  File "/Users/tanner/projects/clients/comap.com/cribbing-check/copyleaks/tasks.py", line 16, in wrapped_func
    func(*args, **kwargs)
  File "/Users/tanner/projects/clients/comap.com/cribbing-check/copyleaks/tasks.py", line 26, in debug_get
    blah = 1/0
ZeroDivisionError: division by zero

@basictheprogram
Copy link
Author

Forgot to add this information to the opening of this ticket. I added this code as recommended by #389

# https://github.com/Koed00/django-q/issues/389
#
from multiprocessing import set_start_method
set_start_method("fork")

Django 3.1 does support 3.8; https://docs.djangoproject.com/en/3.1/faq/install/#what-python-version-can-i-use-with-django; but without that above code I get the dreaded cannot pickle '_thread.lock' object

Traceback (most recent call last):
  File "manage.py", line 22, in <module>
    main()
  File "manage.py", line 18, in main
    execute_from_command_line(sys.argv)
  File "/Users/tanner/.local/share/virtualenvs/cribbing-check-N6-Q5UUH/lib/python3.8/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/Users/tanner/.local/share/virtualenvs/cribbing-check-N6-Q5UUH/lib/python3.8/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Users/tanner/.local/share/virtualenvs/cribbing-check-N6-Q5UUH/lib/python3.8/site-packages/django/core/management/base.py", line 330, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/Users/tanner/.local/share/virtualenvs/cribbing-check-N6-Q5UUH/lib/python3.8/site-packages/django/core/management/base.py", line 371, in execute
    output = self.handle(*args, **options)
  File "/Users/tanner/.local/share/virtualenvs/cribbing-check-N6-Q5UUH/lib/python3.8/site-packages/django_q/management/commands/qcluster.py", line 22, in handle
    q.start()
  File "/Users/tanner/.local/share/virtualenvs/cribbing-check-N6-Q5UUH/lib/python3.8/site-packages/django_q/cluster.py", line 67, in start
    self.sentinel.start()
  File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle '_thread.lock' object

@basictheprogram
Copy link
Author

basictheprogram commented Sep 17, 2020 •
edited
Loading

Looks more and more like a macOS issue or Python:3.8.5 on macOS? as discussed in #389?

The code works under

ubuntu 20.04
Python 3.8.2
Django 3.1.1
django-q 1.3.3
requests 2.24.0
redis 3.5.3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@basictheprogram