Book Image

Parallel Programming with Python

By : Jan Palach, Jan Palach V Cruz da Silva
Book Image

Parallel Programming with Python

By: Jan Palach, Jan Palach V Cruz da Silva

Overview of this book

Table of Contents (16 chapters)
Parallel Programming with Python
About the Author
About the Reviewers

Defining queues by task types

The task that is responsible for calculating Fibonacci was implemented and is running. We can see that all tasks are being sent to a default queue of Celery. However, there are ways to route a task to different queues; let us refactor our architecture in server side and implement what is known as routing task from the client side. We will specify queues for each type of task.

At the moment we start the Celery server in the server side, we will establish three different queues. These will now be seen and consumed by the workers. The queues are fibo_queue for Fibonacci tasks, sqrt_queue for square root tasks, and webcrawler_queue for the Web crawler ones. However, what is the advantage of having task fluxes separated? Let's observe them as follows:

  • It groups tasks of the same type to make their monitoring easier

  • It defines workers dedicated to consume a specific queue, thereby enhancing performance

  • It distributes queues with heavier tasks to brokers allocated in machines...