Enable AWS SQS Long Polling.

[cyliang]

It seems that the attribute ReceiveMessageWaitTimeSeconds of SQS queue is not configurable with Django-Q on sqs.create_queue to enable long polling. Without that, It would produce a huge amount of requests with empty receive that increases cost spent on AWS SQS.

Is there any better way to automatically configure this attribute than manually editing the existing queue on AWS console?

[theonejb]

This issue just bit us, when we got charged on SQS for a queue that was sitting empty the whole month. The way we fixed it was to create a custom broker class that subclassed the Sqs broker class and adding a WaitTimeSeconds parameter to the dequeue method. Here's the code for the custom broker class we're using:

from django_q.brokers.aws_sqs import Sqs as DQ_Sqs


class Sqs(Sqs):
    @staticmethod
    def get_connection(list_key=Conf.PREFIX):
        return Session(region_name=Conf.SQS['aws_region'])

    def dequeue(self):
        """Custom dequeue that stops the broker from generating potentially millions of SQS requests if the queue
        is sitting empty. SQS charges all requests, even the ones that return empty responses, and this can lead
        to us being charged even when we don't do anything."""
        # sqs supports max 10 messages in bulk
        if Conf.BULK > 10:
            Conf.BULK = 10
        wait_timeout = Conf.SQS.get("wait_timeout", 0)
        tasks = self.queue.receive_messages(MaxNumberOfMessages=Conf.BULK, VisibilityTimeout=Conf.RETRY,
                                            WaitTimeSeconds=wait_timeout)
        if tasks:
            return [(t.receipt_handle, t.body) for t in tasks]

We continue to use the sqs key in the settings.Q_CLUSTER dict to configure this class, as it's easier than having to rewrite all the different configuration options.

Hope this helps anyone who comes looking for this issue.

P.S: We also overwrite the get_connection method because instead of passing AWS keys in the settings file, we use EC2 IAM roles to provide the credentials.

[NigelSwinson]

FYI: #506. Django-q now takes receive_message_wait_time_seconds as an argument.