RQ (Redis Queue) is a Python library that uses Redis for queueing jobs and processing them in the background with workers. It has a much lower barrier to entry and is simpler to work with than other libraries such as Celery.
RQ, and task queues in general, are great for executing functions that are lengthy or contain blocking code, such as networking requests.
Using RQ is as simple as creating a queue, and enqueueing the desired function along with the arguments you want to pass to it, according to the code in their “Hello World” example:
from redis import Redis
from rq import Queue
from my_module import count_words_at_url
q = Queue(connection=Redis())
result = q.enqueue(count_words_at_url, 'http://nvie.com')
Let’s walk through how to use RQ to execute a function that grabs data from the Mars Rover API.
Setting up your environment and installing dependencies
Before moving on, you will need to make sure you have an up to date version of Python 3 and pip installed. Make sure you create and activate a virtual environment before installing any dependencies.
We will be using the requests library to get data from NASA’s Mars Rover API, and RQ for handling asynchronous tasks. With your virtual environment activated, run the following command in your terminal to install the necessary Python libraries:
pip install rq==1.1.0 requests==2.22.0
In order for RQ to work, you'll need to install Redis on your machine. That can be done with the following commands using wget:
wget http://download.redis.io/releases/redis-5.0.5.tar.gz
tar xzf redis-5.0.5.tar.gz
cd redis-5.0.5
make
Run Redis in a separate terminal window on the default port with the command src/redis-server from the directory where it's installed.
Getting started with Redis Queue
Let’s start with an example function to hit the Mars Rover API and print out the URL to an image taken by the rover. Since this function includes making an HTTP request, that means the code in it is blocking, making it a great example of something that RQ would be useful for.
Create a file called mars.py and add the following code to it:
from random import choice
import requests
mars_rover_url = 'https://api.nasa.gov/mars-photos/api/v1/rovers/curiosity/photos'
def get_mars_photo(sol):
params = { 'sol': sol, 'api_key': 'DEMO_KEY' }
response = requests.get(mars_rover_url, params).json()
photos = response['photos']
image = choice(photos)['img_src']
print(image)
Run this code by opening a Python shell and entering:
from mars import get_mars_photo
get_mars_photo(1000)
This will print a URL to a random image taken by the Mars Rover 1000 Martian solar days into its journey. Notice that it takes some time for the HTTP request to resolve and for the URL to print. While your code is waiting for the remote server to respond to your HTTP request, it can’t do anything else until it’s finished. Here is a photo that my code printed.
If you pass this function to RQ to be processed as an asynchronous task, then it will no longer be blocking the rest of your code. This can be done as easily as importing RQ, creating a queue, and enqueueing the function. Let’s do this by putting the following code in a file called print_mars_photos.py:
from redis import Redis
from rq import Queue
from mars import get_mars_photo
q = Queue(connection=Redis())
for i in range(10):
q.enqueue(get_mars_photo, 990 + i)
In order for this code to work, you will have to run an RQ worker in the background in another terminal window for processing tasks.
RQ Workers
A worker is a Python process that typically runs in the background and exists solely as a work horse to perform lengthy or blocking tasks that you don’t want to perform inside web processes. RQ uses workers to perform tasks that are added to the queue.
To run the code we just wrote, run the command rqworker in the same directory as your code in another terminal window, and then run your code with the following command:
python print_mars_photos.py
You should see the URLs for pictures taken on Mars being printed one by one, along with other information about the task in the terminal window where the worker is running, but not in the window where your code is running.
To demonstrate the non-blocking nature of this, try adding print statements before and after the loop, and executing regular function calls instead of enqueueing them:
print('Before')
for i in range(10):
get_mars_photo(990 + i)
# q.enqueue(get_mars_photo, 990 + i)
print('After')
You will see each photo URL printed, and then the “After” message. If you revert to the original code, but keep the print statements in, you will see the messages both print pretty much immediately while the code executing the HTTP requests is processed in your other terminal window.
There is a lot you can do with RQ workers and it’s worth reading the documentation to find out. RQ also provides you with a host of utilities on getting insight into jobs, including a CLI that makes it easy to requeue failed jobs. Here’s an example of the latter:
# This command will requeue all jobs in myqueue's failed job registry
rq requeue --queue myqueue -u redis://localhost:6379 --all
The code we used in this post are just short examples of what you can do with RQ, but after going through this post and the RQ docs, hopefully you find it easy to add RQ to your other projects. After being able to execute functions as tasks like we did above, adding RQ to your Flask or Django app means only writing a few lines of code wherever it's needed!
You can also use RQ Scheduler if you want to be able to schedule tasks. It is quick to add it to projects that already use RQ and has a simple API that allows you to execute Python functions at a given datetime.
