Error when using error reporters - AttributeError: 'generator' object has no attribute 'load'

[jordanmkoncz]

I'm trying to set up Sentry as an error reporter for django-q. I have installed django-q-sentry v0.1.1, and I'm using django-q v0.8.1. When I try to run my Django project (i.e. runserver), I get the following error:

Unhandled exception in thread started by <_pydev_bundle.pydev_monkey._NewThreadStartupWithTrace object at 0x112841ac8>
Traceback (most recent call last):
  File "/Users/jordan/Library/Application Support/JetBrains/Toolbox/apps/PyCharm-P/ch-0/172.4343.24/PyCharm.app/Contents/helpers/pydev/_pydev_bundle/pydev_monkey.py", line 589, in __call__
    return self.original_func(*self.args, **self.kwargs)
  File "/Users/jordan/Development/my_project/venv/lib/python3.5/site-packages/django/utils/autoreload.py", line 226, in wrapper
    fn(*args, **kwargs)
  File "/Users/jordan/Development/my_project/venv/lib/python3.5/site-packages/django/core/management/commands/runserver.py", line 109, in inner_run
    autoreload.raise_last_exception()
  File "/Users/jordan/Development/my_project/venv/lib/python3.5/site-packages/django/utils/autoreload.py", line 249, in raise_last_exception
    six.reraise(*_exception)
  File "/Users/jordan/Development/my_project/venv/lib/python3.5/site-packages/django/utils/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/Users/jordan/Development/my_project/venv/lib/python3.5/site-packages/django/utils/autoreload.py", line 226, in wrapper
    fn(*args, **kwargs)
  File "/Users/jordan/Development/my_project/venv/lib/python3.5/site-packages/django/__init__.py", line 18, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/Users/jordan/Development/my_project/venv/lib/python3.5/site-packages/django/apps/registry.py", line 85, in populate
    app_config = AppConfig.create(entry)
  File "/Users/jordan/Development/my_project/venv/lib/python3.5/site-packages/django/apps/config.py", line 116, in create
    mod = import_module(mod_path)
  File "/Users/jordan/Development/my_project/venv/lib/python3.5/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 986, in _gcd_import
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 662, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/Users/jordan/Development/my_project/venv/lib/python3.5/site-packages/django_q/apps.py", line 3, in <module>
    from django_q.conf import Conf
  File "/Users/jordan/Development/my_project/venv/lib/python3.5/site-packages/django_q/conf.py", line 220, in <module>
    'djangoq.errorreporters', name).load()
AttributeError: 'generator' object has no attribute 'load'

My django-q configuration is:

Q_CLUSTER = {
    'name': 'DjangoORM',
    'workers': 4,
    'retry': 120,
    'queue_limit': 50,
    'bulk': 10,
    'orm': 'default',
    'save_limit': 1000,
    'catch_up': False,
    'timeout': 180,
    'error_reporter': {
        'sentry': {
            'dsn': 'https://[email protected]/bar',  # My actual Sentry DSN is here.
        },
    },
}

This error is coming from https://github.com/Koed00/django-q/blob/v0.8.1/django_q/conf.py#L212, specifically the following code:

reporters = []
# iterate through the configured error reporters,
# and instantiate an ErrorReporter using the provided config
for name, conf in error_conf.items():
    Reporter = pkg_resources.iter_entry_points(
            'djangoq.errorreporters', name).load()
    e = Reporter(**conf)
    reporters.append(e)

I'm not very familiar with pkg_resources.iter_entry_points but from what I can see online, it seems like it's being used incorrectly here. Resources like https://docs.pylonsproject.org/projects/pylons-webframework/en/latest/advanced_pylons/entry_points_and_plugins.html and https://www.programcreek.com/python/example/2010/pkg_resources.iter_entry_points do not use the result of pkg_resources.iter_entry_points directly, but rather they iterate over the result, like so:

available_methods = []
for entry_point in pkg_resources.iter_entry_points(group='authkit.method', name=None):
    available_methods.append(entry_point.load())

I did some debugging and when using pkg_resources.iter_entry_points like django-q currently does, the value returned is indeed a generator, and this causes the AttributeError: 'generator' object has no attribute 'load' when django-q tries to call load() on this generator. When I change this block of code so that I am instead iterating over the value returned by pkg_resources.iter_entry_points like the example above, the entry_point is now an EntryPoint class, which seems to be what you'd actually want to call load() on.

[danielwelch]

Well, looks like you've done all the heavy lifting to figure out the problem, thanks! This is also the first time I've worked with pkg_resources.iter_entry_points and seems like it is being used incorrectly as is. Proposed a fix in #278, have a look and see what you think.

[danielwelch]

@jordanmkoncz curious, everything working ok after the fix?

[jordanmkoncz]

@danielwelch I've just been waiting for a new version of django-q to be published to PyPI and was going to do some more testing then.

@Koed00 any chance you can do this soon?

[WRinnovation]

@Koed00 any news about this fix?

[jordanmkoncz]

@danielwelch I've just tested your fix out now that v0.9.0 of django-q has been released, and unfortunately there seems to be an issue still. The good news is that the AttributeError: 'generator' object has no attribute 'load' issue appears to be fixed now that the code is correctly iterating over pkg_resources.iter_entry_points as per your fix. The bad news is that when the line Reporter = entry.load() is executed, I get this error:
ImportError: No module named 'django_q_sentry.Sentry'.

When Reporter = entry.load() is executed, the variable entry is an EntryPoint, and this EntryPoint has name = 'sentry' and module_name = 'django_q_sentry.Sentry'.

I have installed and set up django-q-sentry exactly as the docs at https://github.com/danielwelch/django-q-sentry specify. So my pip requirements.txt file contains django-q[sentry]==0.9.0, and I successfully installed these requirements, and my Q_CLUSTER settings has the error_reporter set up like the docs specify.

I did some more debugging/testing, and I think I've found a way to resolve this issue. If I edit django_q_sentry itself and rename sentry.py to Sentry.py and update __init__.py to contain from .Sentry import Sentry (as per the file renaming), then I no longer get this particular ImportError.

However, after this fix, there is now another error. When the line reporters.append(Reporter(**conf)) is executed, the following error happens: TypeError: 'module' object is not callable. If I add a print(Reporter) line right before the line that's causing the TypeError, I get the following output: <module 'django_q_sentry.Sentry' from '/path/to/project/venv/lib/python3.5/site-packages/django_q_sentry/Sentry.py'>. I believe the TypeError is happening because the value of Reporter here is the actual Sentry.py module, and not the Sentry class that is contained in this module. If I change the line to be reporters.append(Reporter.Sentry(**conf)) (in order to actually call the Sentry class within the module) then the code executes fine and I can see in the console that the Raven client has been loaded and there are no errors. This confirms that the value of Reporter here is the Sentry.py module rather than the Sentry class, but the change I made to this line is obviously not a proper fix as it would break support for other error reporters.

So I think there's a few issues with how this code is working right now and I'm not sure what the best solution is, hopefully you can find a good one @danielwelch.

[danielwelch]

@jordanmkoncz I think it’s time for me to do what I of course should have done in the first place: just test and debug this with a live sentry instance myself. Thanks for all of your feedback on this stuff, I’ll get to work on it tomorrow.

[jordanmkoncz]

Thanks @danielwelch :)

[danielwelch]

@jordanmkoncz Alright, looks like this was due to an error in defining the entry_points in django-q's setup.py file. As you correctly stated, my intention here was to expose the class as the entry point that would be loaded and instantiated with the configuration provided in a user's Django settings. Because I used dot notation instead of django-q-sentry:Sentry, pkg_resources was looking for a module named Sentry rather than the class (as explained at the bottom of the docs section here).

I've loaded up a sample Django project, sentry account, and errors are successfully reporting with django-q, so in combination with your success, I'm hopeful this will fix this issue. Other plugin writers will just have to avoid my mistake and make sure to use the : notation to expose a class.

[jordanmkoncz]

Great, cheers @danielwelch, hopefully @Koed00 can merge this fix into a new django-q release soon.