In the previous chapter, we learned about using parallel Python. We saw the implementation of case studies, including Fibonacci series terms and Web crawler using the parallel Python module. We learned how to establish communication among processes using pipes and how to distribute processes among different machines in a network. In this chapter, we will study how to distribute tasks among different machines in a network by using the Celery framework.
In this chapter, we will cover the following topics:
Understanding Celery
Understanding Celery's architecture
Setting up the environment
Dispatching a simple task
Using Celery to obtain a Fibonacci series term
Using Celery to make a distributed Web crawler