Book Image

IPython Interactive Computing and Visualization Cookbook - Second Edition

By : Cyrille Rossant
Book Image

IPython Interactive Computing and Visualization Cookbook - Second Edition

By: Cyrille Rossant

Overview of this book

Python is one of the leading open source platforms for data science and numerical computing. IPython and the associated Jupyter Notebook offer efficient interfaces to Python for data analysis and interactive visualization, and they constitute an ideal gateway to the platform. IPython Interactive Computing and Visualization Cookbook, Second Edition contains many ready-to-use, focused recipes for high-performance scientific computing and data analysis, from the latest IPython/Jupyter features to the most advanced tricks, to help you write better and faster code. You will apply these state-of-the-art methods to various real-world examples, illustrating topics in applied mathematics, scientific modeling, and machine learning. The first part of the book covers programming techniques: code quality and reproducibility, code optimization, high-performance computing through just-in-time compilation, parallel computing, and graphics card programming. The second part tackles data science, statistics, machine learning, signal and image processing, dynamical systems, and pure and applied mathematics.
Table of Contents (19 chapters)
IPython Interactive Computing and Visualization CookbookSecond Edition
Contributors
Preface
Index

Processing large NumPy arrays with memory mapping


Sometimes, we need to deal with NumPy arrays that are too big to fit in the system memory. A common solution is to use memory mapping and implement out-of-core computations. The array is stored in a file on the hard drive, and we create a memory-mapped object to this file that can be used as a regular NumPy array. Accessing a portion of the array results in the corresponding data being automatically fetched from the hard drive. Therefore, we only consume what we use.

How to do it...

  1. Let's create a memory-mapped array in write mode:

    >>> import numpy as np
    >>> nrows, ncols = 1000000, 100
    >>> f = np.memmap('memmapped.dat', dtype=np.float32,
                      mode='w+', shape=(nrows, ncols))
  2. Let's feed the array with random values, one column at a time because our system's memory is limited!

    >>> for i in range(ncols):
            f[:, i] = np.random.rand(nrows)

    We save the last column of the array:

    >>> x = f...