Book Image

Python Data Analysis - Second Edition

By : Ivan Idris
Book Image

Python Data Analysis - Second Edition

By: Ivan Idris

Overview of this book

Data analysis techniques generate useful insights from small and large volumes of data. Python, with its strong set of libraries, has become a popular platform to conduct various data analysis and predictive modeling tasks. With this book, you will learn how to process and manipulate data with Python for complex analysis and modeling. We learn data manipulations such as aggregating, concatenating, appending, cleaning, and handling missing values, with NumPy and Pandas. The book covers how to store and retrieve data from various data sources such as SQL and NoSQL, CSV fies, and HDF5. We learn how to visualize data using visualization libraries, along with advanced topics such as signal processing, time series, textual data analysis, machine learning, and social media analysis. The book covers a plethora of Python modules, such as matplotlib, statsmodels, scikit-learn, and NLTK. It also covers using Python with external environments such as R, Fortran, C/C++, and Boost libraries.

Related Content you might be interested in

Current Title:

Small book image
Python Data Analysis - Second Edition
Ivan Idris
Mar, 2017 | 330 pages
Small book image
Python Data Analysis
Avinash Navlani | Ivan Idris
Feb, 2021 | 478 pages
Small book image
Learning pandas
Michael Heydt
Jun, 2017 | 446 pages
Small book image
Mastering Numerical Computing with NumPy
Mert Cuhadaroglu | Umit Mert Cakmak
Jun, 2018 | 248 pages
Table of Contents (22 chapters)
Python Data Analysis - Second Edition
Python Data Analysis - Second Edition
Credits
Credits
About the Author
About the Author
About the Reviewers
About the Reviewers
www.PacktPub.com
www.PacktPub.com
Customer Feedback
Customer Feedback
Preface
Preface
Free Chapter
1
Getting Started with Python Libraries
Getting Started with Python Libraries
Installing Python 3
Using IPython as a shell
Reading manual pages
Jupyter Notebook
NumPy arrays
A simple application
Where to find help and references
Listing modules inside the Python libraries
Visualizing data using Matplotlib
Summary
2
NumPy Arrays
NumPy Arrays
The NumPy array object
Creating a multidimensional array
Selecting NumPy array elements
NumPy numerical types
One-dimensional slicing and indexing
Manipulating array shapes
Creating array views and copies
Fancy indexing
Indexing with a list of locations
Indexing NumPy arrays with Booleans
Broadcasting NumPy arrays
Summary
References
3
The Pandas Primer
The Pandas Primer
Installing and exploring Pandas
The Pandas DataFrames
The Pandas Series
Querying data in Pandas
Statistics with Pandas DataFrames
Data aggregation with Pandas DataFrames
Concatenating and appending DataFrames
Joining DataFrames
Handling missing values
Dealing with dates
Pivot tables
Summary
References
4
Statistics and Linear Algebra
Statistics and Linear Algebra
Basic descriptive statistics with NumPy
Linear algebra with NumPy
Finding eigenvalues and eigenvectors with NumPy
NumPy random numbers
Creating a NumPy masked array
Summary
5
Retrieving, Processing, and Storing Data
Retrieving, Processing, and Storing Data
Writing CSV files with NumPy and Pandas
The binary .npy and pickle formats
Storing data with PyTables
Reading and writing Pandas DataFrames to HDF5 stores
Reading and writing to Excel with Pandas
Using REST web services and JSON
Reading and writing JSON with Pandas
Parsing RSS and Atom feeds
Parsing HTML with Beautiful Soup
Summary
Reference
6
Data Visualization
Data Visualization
The matplotlib subpackages
Basic matplotlib plots
Logarithmic plots
Scatter plots
Legends and annotations
Three-dimensional plots
Plotting in Pandas
Lag plots
Autocorrelation plots
Plot.ly
Summary
7
Signal Processing and Time Series
Signal Processing and Time Series
The statsmodels modules
Moving averages
Window functions
Defining cointegration
Autocorrelation
Autoregressive models
ARMA models
Generating periodic signals
Fourier analysis
Spectral analysis
Filtering
Summary
8
Working with Databases
Working with Databases
Lightweight access with sqlite3
Accessing databases from Pandas
SQLAlchemy
Pony ORM
Dataset - databases for lazy people
PyMongo and MongoDB
Storing data in Redis
Storing data in memcache
Apache Cassandra
Summary
9
Analyzing Textual Data and Social Media
Analyzing Textual Data and Social Media
Installing NLTK
About NLTK
Filtering out stopwords, names, and numbers
The bag-of-words model
Analyzing word frequencies
Naive Bayes classification
Sentiment analysis
Creating word clouds
Social network analysis
Summary
10
Predictive Analytics and Machine Learning
Predictive Analytics and Machine Learning
Preprocessing
Classification with logistic regression
Classification with support vector machines
Regression with ElasticNetCV
Support vector regression
Clustering with affinity propagation
Mean shift
Genetic algorithms
Neural networks
Decision trees
Summary
11
Environments Outside the Python Ecosystem and Cloud Computing
Environments Outside the Python Ecosystem and Cloud Computing
Exchanging information with Matlab/Octave
Installing rpy2 package
Interfacing with R
Sending NumPy arrays to Java
Integrating SWIG and NumPy
Integrating Boost and Python
Using Fortran code through f2py
PythonAnywhere Cloud
Summary
12
Performance Tuning, Profiling, and Concurrency
Performance Tuning, Profiling, and Concurrency
Profiling the code
Installing Cython
Calling C code
Creating a process pool with multiprocessing
Speeding up embarrassingly parallel for loops with Joblib
Comparing Bottleneck to NumPy functions
Performing MapReduce with Jug
Installing MPI for Python
IPython Parallel
Summary
Key Concepts
Key Concepts
Useful Functions
Useful Functions
Matplotlib
NumPy
Pandas
Scikit-learn
SciPy
Online Resources
Online Resources

Where to find help and references

The following table lists documentation websites for the Python data analysis libraries we discussed in this chapter.

Packages

Description

NumPy and SciPy

The main documentation website for NumPy and SciPy is at http://docs.scipy.org/doc/. Through this web page, you can browse NumPy and SciPy user guides and reference guides, as well as several tutorials.

Pandas

http://pandas.pydata.org/pandas-docs/stable/.

Matplotlib

http://matplotlib.org/contents.html.

IPython

http://ipython.readthedocs.io/en/stable/.

Jupyter Notebook

http://jupyter-notebook.readthedocs.io/en/latest/.

The popular Stack Overflow software development forum has hundreds of questions tagged NumPy, SciPy, Pandas, Matplotlib, IPython, and Jupyter Notebook. To view them, go to http://stackoverflow.com/questions/tagged/<your-tag-word-here>.

If you are really stuck with a problem, or you want to be kept informed of the development of these libraries, you can subscribe to their respective discussion mailing list(s). The number of e-mails per day varies from list to list. Developers actively involved with the development of these libraries answer some of the questions asked on the mailing lists.

For IRC users, there is an IRC channel on irc://irc.freenode.net. The channel is called #scipy, but you can also ask NumPy questions since SciPy users also have knowledge of NumPy, as SciPy is based on NumPy. There are at least 50 members on the SciPy channel at all times.

Previous Section
End of Section 8
Next Section