Can Celery be used with Amazon SQS

Question

Can we use Amazon SQS as broker backed of Celery. There’s the SQS transport implementation for Kombu, which Celery depends on. Lack of documentation hasnt let me configure SQS with Celery, is there a way to do it?

Answer 1

Yes celery can be setup with SQS. The latest versions of Kombu and Celery are fairly simple to operate with. refer this pseudo code

BROKER_Travel = 'sqs'
BROKER_Travel_OPTIONS = {
    'region': 'us-east-1',
}
BROKER_USER = AWS_ACCESS_KEY_ID
BROKER_PASSWORD = AWS_SECRET_ACCESS_KEY

There you go.......

Answer 2

When i am using Celery 3.0, I was getting deprecation warnings while launching the worker with the BROKER_USER / BROKER_PASSWORD settings.

I took a look at the SQS URL parsing in kombo.utils.url._parse_url and it is calling urllib.unquote on the username and password elements of the URL.

So, to workaround the issue of secret keys with forward slashes, I was able to successfully use the following for the BROKER_URL:

import urllib
BROKER_URL = 'sqs://%s:%s@' % (urllib.quote(AWS_ACCESS_KEY_ID, safe=''),
                               urllib.quote(AWS_SECRET_ACCESS_KEY, safe=''))

Answer 3

From the celery 4.1.0 docs:

The login credentials can also be set using the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, in that case the broker URL may only be sqs://. I could get this to work explicitly building the BROKER_URL as this worked beautifully.

Answer 4

When I configured Celery with Amazon SQS, it seems I achieved a small success.

Patching Kombu this, so I wrote some patches and there is my pull request as well. You can configure Amazon SQS by setting BROKER_URL of sqs:// scheme in Celery on the patched Kombu. For example:

BROKER_URL = 'sqs://AWS_ACCESS:AWS_SECRET@:80//'
BROKER_TRANSPORT_OPTIONS = {
    'region': 'ap-northeast-2',
    'sdb_persistence': False
}

Answer 5

I regenerated the credentials in the IAM consonle until I got a key without a slash (/).

The parsing issues are only with that character, so if your secret doesn't have one you'll be fine.

Not the most terribly elegant solution, but definitely keeps the code clean of hacks.