Can Celery be used with Amazon SQS

+1 vote

Can we use Amazon SQS as broker backed of Celery. There’s the SQS transport implementation for Kombu, which Celery depends on. Lack of documentation hasnt let me configure SQS with Celery, is there a way to do it?

Apr 10, 2018 in AWS by brat_1
• 7,080 points
469 views

5 answers to this question.

+1 vote

Yes celery can be setup with SQS. The latest versions of Kombu and Celery are fairly simple to operate with. refer this pseudo code

BROKER_Travel = 'sqs'
BROKER_Travel_OPTIONS = {
    'region': 'us-east-1',
}
BROKER_USER = AWS_ACCESS_KEY_ID
BROKER_PASSWORD = AWS_SECRET_ACCESS_KEY

There you go.......

answered Apr 10, 2018 by code_ninja
• 6,220 points
+1 vote

When i am using Celery 3.0, I was getting deprecation warnings while launching the worker with the BROKER_USER / BROKER_PASSWORD settings.

I took a look at the SQS URL parsing in kombo.utils.url._parse_url and it is calling urllib.unquote on the username and password elements of the URL.

So, to workaround the issue of secret keys with forward slashes, I was able to successfully use the following for the BROKER_URL:

import urllib
BROKER_URL = 'sqs://%s:%s@' % (urllib.quote(AWS_ACCESS_KEY_ID, safe=''),
                               urllib.quote(AWS_SECRET_ACCESS_KEY, safe=''))
answered Oct 25, 2018 by Nabarupa
+1 vote
From the celery 4.1.0 docs:

The login credentials can also be set using the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, in that case the broker URL may only be sqs://. I could get this to work explicitly building the BROKER_URL as this worked beautifully.
answered Oct 25, 2018 by susiri
+1 vote

When I configured Celery with Amazon SQS, it seems I achieved a small success.

Patching Kombu this, so I wrote some patches and there is my pull request as well. You can configure Amazon SQS by setting BROKER_URL of sqs:// scheme in Celery on the patched Kombu. For example:

BROKER_URL = 'sqs://AWS_ACCESS:AWS_SECRET@:80//'
BROKER_TRANSPORT_OPTIONS = {
    'region': 'ap-northeast-2',
    'sdb_persistence': False
}
answered Oct 25, 2018 by hiesngberg
+1 vote
I regenerated the credentials in the IAM consonle until I got a key without a slash (/).

The parsing issues are only with that character, so if your secret doesn't have one you'll be fine.

Not the most terribly elegant solution, but definitely keeps the code clean of hacks.
answered Oct 25, 2018 by triedntested

Related Questions In AWS

0 votes
1 answer

What are the services that can be used with Amazon VPC?

Almost all the services provided by AWS ...READ MORE

answered Mar 5 in AWS by Ravish
38 views
0 votes
1 answer

what php API can be used to upload and download files to Amazon S3

Amazon have a PHPSDK , check the sample code // ...READ MORE

answered Oct 17, 2018 in AWS by Priyaj
• 56,900 points
934 views
+1 vote
1 answer

What tools can be used to spin-up servers on AWS?

Yes. There are a lot of tools ...READ MORE

answered Dec 19, 2018 in AWS by Archana
• 5,560 points
108 views
0 votes
1 answer
0 votes
1 answer

Can messages be sent in batches? - AWS SQS

Yes! It's totally possible to send messages ...READ MORE

answered Aug 20 in AWS by Scarlet
22 views
+13 votes
2 answers

Git management technique when there are multiple customers and need multiple customization?

Consider this - In 'extended' Git-Flow, (Git-Multi-Flow, ...READ MORE

answered Mar 26, 2018 in DevOps & Agile by DragonLord999
• 8,380 points
181 views
0 votes
1 answer