This chapter discusses the important concept of using parallel and large-scale computing in Python, or using IPython to solve scientific computing problems. It covers recent trends in large-scale scientific computing and Big Data processing. We will use example programs to understand these concepts.
In this chapter, we will cover the following topics:
The fundamentals of parallel computing in IPython
The components of IPython parallel computing
IPython's task interface and database
IPython's direct interface
Details of IPython parallel computing
The MPI program in IPython
Big data processing using Hadoop and Spark in Python
IPython runs a number of different processes to enable users to perform parallel computing. The first of these processes is the IPython engine, which is a Python interpreter that executes the tasks submitted by users. A user can run multiple engines to perform parallel computation. The second process is the IPython hub,...