Task stays in queue when executing a requests.post

[lorddaren]

When I try and execute a requests.post the task stays on the queue and never completes the requests.post request.

Pseudo code:

class Auth
  def __init__(self, url, username, password):
      logging.debug('Making call for credentials.')
      r = requests.post(url, data={'username': username, 'password': password})

def queue_get_auth(username, password):
    a = Auth('https://auth', 'username', 'password')

def validate_login(username, password):
    job = async(queue_get_auth, username, password)

Error I am seeing and repeats until I delete it from the database queued tasks:

[DEBUG] | 2015-10-19 14:46:05,510 | auth:  Making call for credentials.
14:46:05 [Q] ERROR reincarnated worker Process-1:3 after death
14:46:05 [Q] INFO Process-1:9 ready for work at 13576
14:46:19 [Q] INFO Process-1:4 processing [mango-two-juliet-edward]

Is there a reason why the requests.post would be causing it to fail? How would I debug this? If I run it in sync: True it works fine. This is running on a Mac OS X 10.11, database sqlite.

Q_CLUSTER = {
    'name': 'auth',
    'workers': 4,
    'recycle': 500,
    'timeout': 60,
    'compress': False,
    'save_limit': 250,
    'queue_limit': 500,
    'sync': False,
    'cpu_affinity': 1,
    'label': 'Django Q',
    'orm': 'default'
}

[Koed00]

If it works in sync but not async, it usually means there is a context problem. Since workers run in a separate Django environment, The worker must be encountering a pretty bad error cause it executes within a try except clause that catches all.

Have you tried calling queue_get_auth with a full dotted path instead of an instance?
So async('project.module.queue_get_auth', 'username', 'password'). Sometimes a fresh import in the worker instead of a reference, fixes some of these problems.

[lorddaren]

I tried the fully dotted path and it still fails at the same requests.post line. Is there a way to capture this "error"?

[Koed00]

It's almost impossible to attach a debugger to a forked process, but I've been using an exception wrapper for things like this:

import traceback, functools

def trace_unhandled_exceptions(func):
    @functools.wraps(func)
    def wrapped_func(*args, **kwargs):
        try:
            func(*args, **kwargs)
        except:
            print 'Exception in '+func.__name__
            traceback.print_exc()
    return wrapped_func

and then decorate the function

@trace_unhandled_exceptions
def queue_get_auth(username, password):
    a = Auth('https://auth', 'username', 'password')

That will print any errors straight to the console.

[lorddaren]

Still a no go. The crazy thing is if I remove the requests.post it works okay as well and continues on with the rest of the code. I will keep trying to dig into this. I am switching over from Django-RQ and it worked fine.

[Koed00]

The big culprit here is probably the multiprocessing module. That's the main difference with django-rq. This actually started out as a gist for a multiprocessing rq worker I was proposing a few months back.

Some kind of error message would help. I'll see if I can somehow replicate it, although I am on the road the next couple of days.

[lorddaren]

I switched to the redis broker instead of ORM and it seems to be working out okay.

[Koed00]

I ran a few test with simple asynced requests.post and they all worked just fine.

I'll rerun them with the ORM broker ;)

[Koed00]

Postgres worked ok. Even sqllite. I can't seem to reproduce it with just requests.post to a local url.

[Koed00]

Could there be some kind of file locking happening with sqllite being the broker and the result backend?
Perhaps you should check your os or db logs for this. You could add a second sqllite database and set that to be the broker. Don't forget to run migrations on that new db first.

[lorddaren]

Yeah I switched to FileCache and it worked. I will see if I can get a second sqlite database.

[Koed00]

I figured it had to be something like this. Having the broker, results, monitoring and your queries running on a singe sqllite file might be a bit much. It also explains why we're not seeing any real error messages.

[lorddaren]

As an FYI I did try it against my production Oracle database and had the same problem.

[Koed00]

In that case I'll have to come up with a test for when the broker, cache and Django are all on the same database and see if I can reproduce this. I'm not at all surprised that this would lead to locking. It's just something I've never tested for. I'll pick it up when I'm back next week.
Using a separate cache backend should hold you over for now.

[lorddaren]

Thank you for you help and the nice library. I read about it on reddit and wanted to get it implemented.

[Koed00]

You're welcome. Feedback like this is very appreciated. I just can't test all this stuff on my own.

[basictheprogram]

In case someone, like me, comes to this issue, using ORM with sqlite, and requests.post() is "not working", this is still a problem in Sep of 2020. Just making a note here.