Book Image

Distributed Computing with Python

Book Image

Distributed Computing with Python

Overview of this book

CPU-intensive data processing tasks have become crucial considering the complexity of the various big data applications that are used today. Reducing the CPU utilization per process is very important to improve the overall speed of applications. This book will teach you how to perform parallel execution of computations by distributing them across multiple processors in a single machine, thus improving the overall performance of a big data processing task. We will cover synchronous and asynchronous models, shared memory and file systems, communication between various processes, synchronization, and more.
Table of Contents (15 chapters)
Distributed Computing with Python
Credits
About the Author
About the Reviewer
www.PacktPub.com
Preface
Index

Celery alternatives – Pyro


Pyro (http://pythonhosted.org/Pyro4/), which stands for Python Remote Objects, is a package that has been around for quite some time, since 1998 or so. As a result, it is remarkably stable and feature-complete.

It takes a very different approach to task distribution than Celery or Python-RQ in that it exposes Python objects as servers on a network. It then creates proxy objects to them so that the calling code sees them as local objects. This architecture was very popular at the end of the 90s with systems such as CORBA and Java RMI.

The fact that Pyro is somewhat hiding the fact that some of the objects in one's code are local and some are remote is at times a source of criticism. The reason being that there are a number of failure modes intrinsic to running remote code that are easy to forget when remote code execution is hidden behind a proxy object.

Another source of criticism is that Pyro can be somewhat difficult to get running correctly on an ad hoc network...