Book Image

IPython Interactive Computing and Visualization Cookbook - Second Edition

By : Cyrille Rossant
Book Image

IPython Interactive Computing and Visualization Cookbook - Second Edition

By: Cyrille Rossant

Overview of this book

Python is one of the leading open source platforms for data science and numerical computing. IPython and the associated Jupyter Notebook offer efficient interfaces to Python for data analysis and interactive visualization, and they constitute an ideal gateway to the platform. IPython Interactive Computing and Visualization Cookbook, Second Edition contains many ready-to-use, focused recipes for high-performance scientific computing and data analysis, from the latest IPython/Jupyter features to the most advanced tricks, to help you write better and faster code. You will apply these state-of-the-art methods to various real-world examples, illustrating topics in applied mathematics, scientific modeling, and machine learning. The first part of the book covers programming techniques: code quality and reproducibility, code optimization, high-performance computing through just-in-time compilation, parallel computing, and graphics card programming. The second part tackles data science, statistics, machine learning, signal and image processing, dynamical systems, and pure and applied mathematics.

Related Content you might be interested in

Current Title:

Small book image
IPython Interactive Computing and Visualization Cookbook - Second Edition
Cyrille Rossant
Jan, 2018 | 548 pages
Small book image
Python High Performance
Gabriele Lanaro
May, 2017 | 270 pages
Small book image
Applying Math with Python
Sam Morley
Jul, 2020 | 358 pages
Small book image
A Handbook of Mathematical Models with Python
Dr Ranja Sarkar
Aug, 2023 | 144 pages
Table of Contents (19 chapters)
IPython Interactive Computing and Visualization CookbookSecond Edition
IPython Interactive Computing and Visualization CookbookSecond Edition
Contributors
Contributors
Preface
Preface
Free Chapter
1
A Tour of Interactive Computing with Jupyter and IPython
A Tour of Interactive Computing with Jupyter and IPython
Introduction
Introducing IPython and the Jupyter Notebook
Getting started with exploratory data analysis in the Jupyter Notebook
Introducing the multidimensional array in NumPy for fast array computations
Creating an IPython extension with custom magic commands
Mastering IPython's configuration system
Creating a simple kernel for Jupyter
2
Best Practices in Interactive Computing
Best Practices in Interactive Computing
Introduction
Learning the basics of the Unix shell
Using the latest features of Python 3
Learning the basics of the distributed version control system Git
A typical workflow with Git branching
Efficient interactive computing workflows with IPython
Ten tips for conducting reproducible interactive computing experiments
Writing high-quality Python code
Writing unit tests with pytest
Debugging code with IPython
3
Mastering the Jupyter Notebook
Mastering the Jupyter Notebook
Introduction
Teaching programming in the Notebook with IPython Blocks
Converting a Jupyter notebook to other formats with nbconvert
Mastering widgets in the Jupyter Notebook
Creating custom Jupyter Notebook widgets in Python, HTML, and JavaScript
Configuring the Jupyter Notebook
Introducing JupyterLab
4
Profiling and Optimization
Profiling and Optimization
Introduction
Evaluating the time taken by a command in IPython
Profiling your code easily with cProfile and IPython
Profiling your code line-by-line with line_profiler
Profiling the memory usage of your code with memory_profiler
Understanding the internals of NumPy to avoid unnecessary array copying
Using stride tricks with NumPy
Implementing an efficient rolling average algorithm with stride tricks
Processing large NumPy arrays with memory mapping
Manipulating large arrays with HDF5
5
High-Performance Computing
High-Performance Computing
Introduction
Using Python to write faster code
Accelerating pure Python code with Numba and Just-In-Time compilation
Accelerating array computations with NumExpr
Wrapping a C library in Python with ctypes
Accelerating Python code with Cython
Optimizing Cython code by writing less Python and more C
Releasing the GIL to take advantage of multi-core processors with Cython and OpenMP
Writing massively parallel code for NVIDIA graphics cards (GPUs) with CUDA
Distributing Python code across multiple cores with IPython
Interacting with asynchronous parallel tasks in IPython
Performing out-of-core computations on large arrays with Dask
Trying the Julia programming language in the Jupyter Notebook
6
Data Visualization
Data Visualization
Introduction
Using Matplotlib styles
Creating statistical plots easily with seaborn
Creating interactive web visualizations with Bokeh and HoloViews
Visualizing a NetworkX graph in the Notebook with D3.js
Discovering interactive visualization libraries in the Notebook
Creating plots with Altair and the Vega-Lite specification
7
Statistical Data Analysis
Statistical Data Analysis
Introduction
Exploring a dataset with pandas and Matplotlib
Getting started with statistical hypothesis testing — a simple z-test
Getting started with Bayesian methods
Estimating the correlation between two variables with a contingency table and a chi-squared test
Fitting a probability distribution to data with the maximum likelihood method
Estimating a probability distribution nonparametrically with a kernel density estimation
Fitting a Bayesian model by sampling from a posterior distribution with a Markov chain Monte Carlo method
Analyzing data with the R programming language in the Jupyter Notebook
8
Machine Learning
Machine Learning
Introduction
Getting started with scikit-learn
Predicting who will survive on the Titanic with logistic regression
Learning to recognize handwritten digits with a K-nearest neighbors classifier
Learning from text – Naive Bayes for Natural Language Processing
Using support vector machines for classification tasks
Using a random forest to select important features for regression
Reducing the dimensionality of a dataset with a principal component analysis
Detecting hidden structures in a dataset with clustering
9
Numerical Optimization
Numerical Optimization
Introduction
Finding the root of a mathematical function
Minimizing a mathematical function
Fitting a function to data with nonlinear least squares
Finding the equilibrium state of a physical system by minimizing its potential energy
10
Signal Processing
Signal Processing
Introduction
Analyzing the frequency components of a signal with a Fast Fourier Transform
Applying a linear filter to a digital signal
Computing the autocorrelation of a time series
11
Image and Audio Processing
Image and Audio Processing
Introduction
Manipulating the exposure of an image
Applying filters on an image
Segmenting an image
Finding points of interest in an image
Detecting faces in an image with OpenCV
Applying digital filters to speech sounds
Creating a sound synthesizer in the Notebook
12
Deterministic Dynamical Systems
Deterministic Dynamical Systems
Introduction
Plotting the bifurcation diagram of a chaotic dynamical system
Simulating an elementary cellular automaton
Simulating an ordinary differential equation with SciPy
Simulating a partial differential equation — reaction-diffusion systems and Turing patterns
13
Stochastic Dynamical Systems
Stochastic Dynamical Systems
Introduction
Simulating a discrete-time Markov chain
Simulating a Poisson process
Simulating a Brownian motion
Simulating a stochastic differential equation
14
Graphs, Geometry, and Geographic Information Systems
Graphs, Geometry, and Geographic Information Systems
Introduction
Manipulating and visualizing graphs with NetworkX
Drawing flight routes with NetworkX
Resolving dependencies in a directed acyclic graph with a topological sort
Computing connected components in an image
Computing the Voronoi diagram of a set of points
Manipulating geospatial data with Cartopy
Creating a route planner for a road network
15
Symbolic and Numerical Mathematics
Symbolic and Numerical Mathematics
Introduction
Diving into symbolic computing with SymPy
Solving equations and inequalities
Analyzing real-valued functions
Computing exact probabilities and manipulating random variables
A bit of number theory with SymPy
Finding a Boolean propositional formula from a truth table
Analyzing a nonlinear differential system — Lotka-Volterra (predator-prey) equations
Getting started with Sage
Index
Index

Performing out-of-core computations on large arrays with Dask

Dask is a parallel computing library that offers not only a general framework for distributing complex computations on many nodes, but also a set of convenient high-level APIs to deal with out-of-core computations on large arrays. Dask provides data structures resembling NumPy arrays (dask.array) and Pandas DataFrames (dask.dataframe) that efficiently scale to huge datasets. The core idea of Dask is to split a large array into smaller arrays (chunks).

In this recipe, we illustrate the basic principles of dask.array.

Getting ready

Dask should already be installed in Anaconda, but you can always install it manually with conda install dask. You also need memory_profiler, which you can install with conda install memory_profiler.

How to do it...

  1. Let's import the libraries:

    >>> import numpy as np
    import dask.array as da
    import memory_profiler
>>> %load_ext memory_profiler

  2. We initialize a large 10,000 x 10,000 array...

Previous Section
End of Section 13
Next Section