Book Image

Parallel Programming with Python

Book Image

Parallel Programming with Python

Overview of this book

Table of Contents (16 chapters)
Parallel Programming with Python
Credits
About the Author
Acknowledgments
About the Reviewers
www.PacktPub.com
Preface
Index

Chapter 7. Distributing Tasks with Celery

In the previous chapter, we learned about using parallel Python. We saw the implementation of case studies, including Fibonacci series terms and Web crawler using the parallel Python module. We learned how to establish communication among processes using pipes and how to distribute processes among different machines in a network. In this chapter, we will study how to distribute tasks among different machines in a network by using the Celery framework.

In this chapter, we will cover the following topics:

  • Understanding Celery

  • Understanding Celery's architecture

  • Setting up the environment

  • Dispatching a simple task

  • Using Celery to obtain a Fibonacci series term

  • Using Celery to make a distributed Web crawler