4. Profiling and Optimization | IPython Interactive Computing and Visualization Cookbook

Sign In Start Free Trial

Book Overview & Buying
Table Of Contents
Feedback & Rating

IPython Interactive Computing and Visualization Cookbook

By : Cyrille Rossant

4.5 (13)

IPython Interactive Computing and Visualization Cookbook

4.5 (13)

By: Cyrille Rossant

Overview of this book

Intended to anyone interested in numerical computing and data science: students, researchers, teachers, engineers, analysts, hobbyists... Basic knowledge of Python/NumPy is recommended. Some skills in mathematics will help you understand the theory behind the computational methods.

Preface

Preface

What this book is

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Free Chapter

1. A Tour of Interactive Computing with IPython

1. A Tour of Interactive Computing with IPython

Introduction

Introducing the IPython notebook

Getting started with exploratory data analysis in IPython

Introducing the multidimensional array in NumPy for fast array computations

Creating an IPython extension with custom magic commands

Mastering IPython's configuration system

Creating a simple kernel for IPython

2. Best Practices in Interactive Computing

2. Best Practices in Interactive Computing

Introduction

Choosing (or not) between Python 2 and Python 3

Efficient interactive computing workflows with IPython

Learning the basics of the distributed version control system Git

A typical workflow with Git branching

Ten tips for conducting reproducible interactive computing experiments

Writing high-quality Python code

Writing unit tests with nose

Debugging your code with IPython

3. Mastering the Notebook

3. Mastering the Notebook

Introduction

Teaching programming in the notebook with IPython blocks

Converting an IPython notebook to other formats with nbconvert

Adding custom controls in the notebook toolbar

Customizing the CSS style in the notebook

Using interactive widgets – a piano in the notebook

Creating a custom JavaScript widget in the notebook – a spreadsheet editor for pandas

Processing webcam images in real time from the notebook

4. Profiling and Optimization

4. Profiling and Optimization

Introduction

Evaluating the time taken by a statement in IPython

Profiling your code easily with cProfile and IPython

Profiling your code line-by-line with line_profiler

Profiling the memory usage of your code with memory_profiler

Understanding the internals of NumPy to avoid unnecessary array copying

Using stride tricks with NumPy

Implementing an efficient rolling average algorithm with stride tricks

Making efficient array selections in NumPy

Processing huge NumPy arrays with memory mapping

Manipulating large arrays with HDF5 and PyTables

Manipulating large heterogeneous tables with HDF5 and PyTables

5. High-performance Computing

5. High-performance Computing

Introduction

Accelerating pure Python code with Numba and just-in-time compilation

Accelerating array computations with Numexpr

Wrapping a C library in Python with ctypes

Accelerating Python code with Cython

Optimizing Cython code by writing less Python and more C

Releasing the GIL to take advantage of multicore processors with Cython and OpenMP

Writing massively parallel code for NVIDIA graphics cards (GPUs) with CUDA

Writing massively parallel code for heterogeneous platforms with OpenCL

Distributing Python code across multiple cores with IPython

Interacting with asynchronous parallel tasks in IPython

Parallelizing code with MPI in IPython

Trying the Julia language in the notebook

6. Advanced Visualization

6. Advanced Visualization

Introduction

Making nicer matplotlib figures with prettyplotlib

Creating beautiful statistical plots with seaborn

Creating interactive web visualizations with Bokeh

Visualizing a NetworkX graph in the IPython notebook with D3.js

Converting matplotlib figures to D3.js visualizations with mpld3

Getting started with Vispy for high-performance interactive data visualizations

7. Statistical Data Analysis

7. Statistical Data Analysis

Introduction

Exploring a dataset with pandas and matplotlib

Getting started with statistical hypothesis testing – a simple z-test

Getting started with Bayesian methods

Estimating the correlation between two variables with a contingency table and a chi-squared test

Fitting a probability distribution to data with the maximum likelihood method

Estimating a probability distribution nonparametrically with a kernel density estimation

Fitting a Bayesian model by sampling from a posterior distribution with a Markov chain Monte Carlo method

Analyzing data with the R programming language in the IPython notebook

8. Machine Learning

8. Machine Learning

Introduction

Getting started with scikit-learn

Predicting who will survive on the Titanic with logistic regression

Learning to recognize handwritten digits with a K-nearest neighbors classifier

Learning from text – Naive Bayes for Natural Language Processing

Using support vector machines for classification tasks

Using a random forest to select important features for regression

Reducing the dimensionality of a dataset with a principal component analysis

Detecting hidden structures in a dataset with clustering

9. Numerical Optimization

9. Numerical Optimization

Introduction

Finding the root of a mathematical function

Minimizing a mathematical function

Fitting a function to data with nonlinear least squares

Finding the equilibrium state of a physical system by minimizing its potential energy

10. Signal Processing

10. Signal Processing

Introduction

Analyzing the frequency components of a signal with a Fast Fourier Transform

Applying a linear filter to a digital signal

Computing the autocorrelation of a time series

11. Image and Audio Processing

11. Image and Audio Processing

Introduction

Manipulating the exposure of an image

Applying filters on an image

Segmenting an image

Finding points of interest in an image

Detecting faces in an image with OpenCV

Applying digital filters to speech sounds

Creating a sound synthesizer in the notebook

12. Deterministic Dynamical Systems

12. Deterministic Dynamical Systems

Introduction

Plotting the bifurcation diagram of a chaotic dynamical system

Simulating an elementary cellular automaton

Simulating an ordinary differential equation with SciPy

Simulating a partial differential equation – reaction-diffusion systems and Turing patterns

13. Stochastic Dynamical Systems

13. Stochastic Dynamical Systems

Introduction

Simulating a discrete-time Markov chain

Simulating a Poisson process

Simulating a Brownian motion

Simulating a stochastic differential equation

14. Graphs, Geometry, and Geographic Information Systems

14. Graphs, Geometry, and Geographic Information Systems

Introduction

Manipulating and visualizing graphs with NetworkX

Analyzing a social network with NetworkX

Resolving dependencies in a directed acyclic graph with a topological sort

Computing connected components in an image

Computing the Voronoi diagram of a set of points

Manipulating geospatial data with Shapely and basemap

Creating a route planner for a road network

15. Symbolic and Numerical Mathematics

15. Symbolic and Numerical Mathematics

Introduction

Diving into symbolic computing with SymPy

Solving equations and inequalities

Analyzing real-valued functions

Computing exact probabilities and manipulating random variables

A bit of number theory with SymPy

Finding a Boolean propositional formula from a truth table

Analyzing a nonlinear differential system – Lotka-Volterra (predator-prey) equations

Getting started with Sage

Index

Index

Customer Reviews

4.5 (13)

5 star

42.9%

4 star

57.1%

3 star

0%

2 star

0%

1 star

0%

Profiling the memory usage of your code with memory_profiler

The methods described in the previous recipe were about CPU time profiling. That may be the most obvious factor when it comes to code profiling. However, memory is also a critical factor. Writing memory-optimized code is not trivial and can really make your program faster. This is particularly important when dealing with large NumPy arrays, as we will see later in this chapter.

In this recipe, we will look at a simple memory profiler unsurprisingly named memory_profiler. Its usage is very similar to line_profiler, and it can be conveniently used from IPython.

Getting ready

You can install memory_profiler with conda install memory_profiler.

How to do it...

We load the memory_profiler IPython extension:
```
>>> %load_ext memory_profiler
```

We define a function that allocates big objects:

>>> %%writefile memscript.py
    def my_func():
        a = [1] * 1000000
        b = [2] * 9000000
        del b
        return a

Now, let's...

Visually different images

CONTINUE READING

83

Tech Concepts

36

Programming languages

73

Tech Tools

Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.

Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.

50+ new titles added per month and exclusive early access to books as they are being written.

IPython Interactive Computing and Visualization Cookbook

Search

Your notes and bookmarks