Book Image

Python Data Analysis - Second Edition

By : Ivan Idris
Book Image

Python Data Analysis - Second Edition

By: Ivan Idris

Overview of this book

Data analysis techniques generate useful insights from small and large volumes of data. Python, with its strong set of libraries, has become a popular platform to conduct various data analysis and predictive modeling tasks. With this book, you will learn how to process and manipulate data with Python for complex analysis and modeling. We learn data manipulations such as aggregating, concatenating, appending, cleaning, and handling missing values, with NumPy and Pandas. The book covers how to store and retrieve data from various data sources such as SQL and NoSQL, CSV fies, and HDF5. We learn how to visualize data using visualization libraries, along with advanced topics such as signal processing, time series, textual data analysis, machine learning, and social media analysis. The book covers a plethora of Python modules, such as matplotlib, statsmodels, scikit-learn, and NLTK. It also covers using Python with external environments such as R, Fortran, C/C++, and Boost libraries.

Related Content you might be interested in

Current Title:

Small book image
Python Data Analysis - Second Edition
Ivan Idris
Mar, 2017 | 330 pages
Small book image
Python Data Analysis
Avinash Navlani | Ivan Idris
Feb, 2021 | 478 pages
Small book image
Learning pandas
Michael Heydt
Jun, 2017 | 446 pages
Small book image
Mastering Numerical Computing with NumPy
Mert Cuhadaroglu | Umit Mert Cakmak
Jun, 2018 | 248 pages
Table of Contents (22 chapters)
Python Data Analysis - Second Edition
Python Data Analysis - Second Edition
Credits
Credits
About the Author
About the Author
About the Reviewers
About the Reviewers
www.PacktPub.com
www.PacktPub.com
Customer Feedback
Customer Feedback
Preface
Preface
Free Chapter
1
Getting Started with Python Libraries
Getting Started with Python Libraries
Installing Python 3
Using IPython as a shell
Reading manual pages
Jupyter Notebook
NumPy arrays
A simple application
Where to find help and references
Listing modules inside the Python libraries
Visualizing data using Matplotlib
Summary
2
NumPy Arrays
NumPy Arrays
The NumPy array object
Creating a multidimensional array
Selecting NumPy array elements
NumPy numerical types
One-dimensional slicing and indexing
Manipulating array shapes
Creating array views and copies
Fancy indexing
Indexing with a list of locations
Indexing NumPy arrays with Booleans
Broadcasting NumPy arrays
Summary
References
3
The Pandas Primer
The Pandas Primer
Installing and exploring Pandas
The Pandas DataFrames
The Pandas Series
Querying data in Pandas
Statistics with Pandas DataFrames
Data aggregation with Pandas DataFrames
Concatenating and appending DataFrames
Joining DataFrames
Handling missing values
Dealing with dates
Pivot tables
Summary
References
4
Statistics and Linear Algebra
Statistics and Linear Algebra
Basic descriptive statistics with NumPy
Linear algebra with NumPy
Finding eigenvalues and eigenvectors with NumPy
NumPy random numbers
Creating a NumPy masked array
Summary
5
Retrieving, Processing, and Storing Data
Retrieving, Processing, and Storing Data
Writing CSV files with NumPy and Pandas
The binary .npy and pickle formats
Storing data with PyTables
Reading and writing Pandas DataFrames to HDF5 stores
Reading and writing to Excel with Pandas
Using REST web services and JSON
Reading and writing JSON with Pandas
Parsing RSS and Atom feeds
Parsing HTML with Beautiful Soup
Summary
Reference
6
Data Visualization
Data Visualization
The matplotlib subpackages
Basic matplotlib plots
Logarithmic plots
Scatter plots
Legends and annotations
Three-dimensional plots
Plotting in Pandas
Lag plots
Autocorrelation plots
Plot.ly
Summary
7
Signal Processing and Time Series
Signal Processing and Time Series
The statsmodels modules
Moving averages
Window functions
Defining cointegration
Autocorrelation
Autoregressive models
ARMA models
Generating periodic signals
Fourier analysis
Spectral analysis
Filtering
Summary
8
Working with Databases
Working with Databases
Lightweight access with sqlite3
Accessing databases from Pandas
SQLAlchemy
Pony ORM
Dataset - databases for lazy people
PyMongo and MongoDB
Storing data in Redis
Storing data in memcache
Apache Cassandra
Summary
9
Analyzing Textual Data and Social Media
Analyzing Textual Data and Social Media
Installing NLTK
About NLTK
Filtering out stopwords, names, and numbers
The bag-of-words model
Analyzing word frequencies
Naive Bayes classification
Sentiment analysis
Creating word clouds
Social network analysis
Summary
10
Predictive Analytics and Machine Learning
Predictive Analytics and Machine Learning
Preprocessing
Classification with logistic regression
Classification with support vector machines
Regression with ElasticNetCV
Support vector regression
Clustering with affinity propagation
Mean shift
Genetic algorithms
Neural networks
Decision trees
Summary
11
Environments Outside the Python Ecosystem and Cloud Computing
Environments Outside the Python Ecosystem and Cloud Computing
Exchanging information with Matlab/Octave
Installing rpy2 package
Interfacing with R
Sending NumPy arrays to Java
Integrating SWIG and NumPy
Integrating Boost and Python
Using Fortran code through f2py
PythonAnywhere Cloud
Summary
12
Performance Tuning, Profiling, and Concurrency
Performance Tuning, Profiling, and Concurrency
Profiling the code
Installing Cython
Calling C code
Creating a process pool with multiprocessing
Speeding up embarrassingly parallel for loops with Joblib
Comparing Bottleneck to NumPy functions
Performing MapReduce with Jug
Installing MPI for Python
IPython Parallel
Summary
Key Concepts
Key Concepts
Useful Functions
Useful Functions
Matplotlib
NumPy
Pandas
Scikit-learn
SciPy
Online Resources
Online Resources

Interfacing with R

R provides a datasets package that contains sample datasets. The morley dataset has data from measurements of the speed of light made in 1879. The speed of light is a fundamental physical constant and its value is currently known very precisely. The data is described at http://stat.ethz.ch/R-manual/R-devel/library/datasets/html/morley.html. The speed of light value can be found in the scipy.constants module. The R data is stored in an R DataFrame with three columns:

  • The experiment number, from one to five

  • The run number, with twenty runs per experiment, bringing the total number of measurements to 100

  • The measured speed of light in kilometers per second with 299,000 subtracted

The rpy2.robjects.r() function executes R code in a Python environment. Load the data as follows:

pandas2ri.activate() 
r.data('morley')

The Pandas library's R interface via the pandas.rpy.common module is deprecated, and thus it is suggested that the reader uses rpy2 objects module. Load the data into...

Previous Section
End of Section 4
Next Section