Book Image

Python Data Analysis - Second Edition

By : Ivan Idris
Book Image

Python Data Analysis - Second Edition

By: Ivan Idris

Overview of this book

Data analysis techniques generate useful insights from small and large volumes of data. Python, with its strong set of libraries, has become a popular platform to conduct various data analysis and predictive modeling tasks. With this book, you will learn how to process and manipulate data with Python for complex analysis and modeling. We learn data manipulations such as aggregating, concatenating, appending, cleaning, and handling missing values, with NumPy and Pandas. The book covers how to store and retrieve data from various data sources such as SQL and NoSQL, CSV fies, and HDF5. We learn how to visualize data using visualization libraries, along with advanced topics such as signal processing, time series, textual data analysis, machine learning, and social media analysis. The book covers a plethora of Python modules, such as matplotlib, statsmodels, scikit-learn, and NLTK. It also covers using Python with external environments such as R, Fortran, C/C++, and Boost libraries.

Related Content you might be interested in

Current Title:

Small book image
Python Data Analysis - Second Edition
Ivan Idris
Mar, 2017 | 330 pages
Small book image
Python Data Analysis
Avinash Navlani | Ivan Idris
Feb, 2021 | 478 pages
Small book image
Learning pandas
Michael Heydt
Jun, 2017 | 446 pages
Small book image
Mastering Numerical Computing with NumPy
Mert Cuhadaroglu | Umit Mert Cakmak
Jun, 2018 | 248 pages
Table of Contents (22 chapters)
Python Data Analysis - Second Edition
Python Data Analysis - Second Edition
Credits
Credits
About the Author
About the Author
About the Reviewers
About the Reviewers
www.PacktPub.com
www.PacktPub.com
Customer Feedback
Customer Feedback
Preface
Preface
Free Chapter
1
Getting Started with Python Libraries
Getting Started with Python Libraries
Installing Python 3
Using IPython as a shell
Reading manual pages
Jupyter Notebook
NumPy arrays
A simple application
Where to find help and references
Listing modules inside the Python libraries
Visualizing data using Matplotlib
Summary
2
NumPy Arrays
NumPy Arrays
The NumPy array object
Creating a multidimensional array
Selecting NumPy array elements
NumPy numerical types
One-dimensional slicing and indexing
Manipulating array shapes
Creating array views and copies
Fancy indexing
Indexing with a list of locations
Indexing NumPy arrays with Booleans
Broadcasting NumPy arrays
Summary
References
3
The Pandas Primer
The Pandas Primer
Installing and exploring Pandas
The Pandas DataFrames
The Pandas Series
Querying data in Pandas
Statistics with Pandas DataFrames
Data aggregation with Pandas DataFrames
Concatenating and appending DataFrames
Joining DataFrames
Handling missing values
Dealing with dates
Pivot tables
Summary
References
4
Statistics and Linear Algebra
Statistics and Linear Algebra
Basic descriptive statistics with NumPy
Linear algebra with NumPy
Finding eigenvalues and eigenvectors with NumPy
NumPy random numbers
Creating a NumPy masked array
Summary
5
Retrieving, Processing, and Storing Data
Retrieving, Processing, and Storing Data
Writing CSV files with NumPy and Pandas
The binary .npy and pickle formats
Storing data with PyTables
Reading and writing Pandas DataFrames to HDF5 stores
Reading and writing to Excel with Pandas
Using REST web services and JSON
Reading and writing JSON with Pandas
Parsing RSS and Atom feeds
Parsing HTML with Beautiful Soup
Summary
Reference
6
Data Visualization
Data Visualization
The matplotlib subpackages
Basic matplotlib plots
Logarithmic plots
Scatter plots
Legends and annotations
Three-dimensional plots
Plotting in Pandas
Lag plots
Autocorrelation plots
Plot.ly
Summary
7
Signal Processing and Time Series
Signal Processing and Time Series
The statsmodels modules
Moving averages
Window functions
Defining cointegration
Autocorrelation
Autoregressive models
ARMA models
Generating periodic signals
Fourier analysis
Spectral analysis
Filtering
Summary
8
Working with Databases
Working with Databases
Lightweight access with sqlite3
Accessing databases from Pandas
SQLAlchemy
Pony ORM
Dataset - databases for lazy people
PyMongo and MongoDB
Storing data in Redis
Storing data in memcache
Apache Cassandra
Summary
9
Analyzing Textual Data and Social Media
Analyzing Textual Data and Social Media
Installing NLTK
About NLTK
Filtering out stopwords, names, and numbers
The bag-of-words model
Analyzing word frequencies
Naive Bayes classification
Sentiment analysis
Creating word clouds
Social network analysis
Summary
10
Predictive Analytics and Machine Learning
Predictive Analytics and Machine Learning
Preprocessing
Classification with logistic regression
Classification with support vector machines
Regression with ElasticNetCV
Support vector regression
Clustering with affinity propagation
Mean shift
Genetic algorithms
Neural networks
Decision trees
Summary
11
Environments Outside the Python Ecosystem and Cloud Computing
Environments Outside the Python Ecosystem and Cloud Computing
Exchanging information with Matlab/Octave
Installing rpy2 package
Interfacing with R
Sending NumPy arrays to Java
Integrating SWIG and NumPy
Integrating Boost and Python
Using Fortran code through f2py
PythonAnywhere Cloud
Summary
12
Performance Tuning, Profiling, and Concurrency
Performance Tuning, Profiling, and Concurrency
Profiling the code
Installing Cython
Calling C code
Creating a process pool with multiprocessing
Speeding up embarrassingly parallel for loops with Joblib
Comparing Bottleneck to NumPy functions
Performing MapReduce with Jug
Installing MPI for Python
IPython Parallel
Summary
Key Concepts
Key Concepts
Useful Functions
Useful Functions
Matplotlib
NumPy
Pandas
Scikit-learn
SciPy
Online Resources
Online Resources

Storing data with PyTables

Hierarchical data format (HDF) is a specification and technology for the storage of big numerical data. HDF was created in the supercomputing community and is now an open standard. The latest version of HDF is HDF5 and is the one we will be using. HDF5 structures data in groups and datasets. Datasets are multidimensional homogeneous arrays. Groups can contain other groups or datasets. Groups are like directories in a hierarchical filesystem.

The two main HDF5 Python libraries are as follows:

  • h5y

  • PyTables

In this example, we will be using PyTables. PyTables has a number of dependencies:

  • The NumPy package, which we installed in Chapter 1, Getting Started with Python Libraries

  • The numexpr package, which claims that it evaluates multiple-operator array expressions many times faster than NumPy can

  • HDF5

Note

The parallel version of HDF5 also requires MPI. HDF5 can be installed by obtaining a distribution from http://www.hdfgroup.org/HDF5/release/obtain5.html and running the...

Previous Section
End of Section 4
Next Section