8. Working with Databases | Python Data Analysis, Second Edition

Sign In Start Free Trial

Book Overview & Buying
Table Of Contents

Python Data Analysis, Second Edition - Second Edition

By : Armando Fandango

4 (4)

Python Data Analysis, Second Edition

4 (4)

By: Armando Fandango

Overview of this book

Data analysis techniques generate useful insights from small and large volumes of data. Python, with its strong set of libraries, has become a popular platform to conduct various data analysis and predictive modeling tasks. With this book, you will learn how to process and manipulate data with Python for complex analysis and modeling. We learn data manipulations such as aggregating, concatenating, appending, cleaning, and handling missing values, with NumPy and Pandas. The book covers how to store and retrieve data from various data sources such as SQL and NoSQL, CSV fies, and HDF5. We learn how to visualize data using visualization libraries, along with advanced topics such as signal processing, time series, textual data analysis, machine learning, and social media analysis. The book covers a plethora of Python modules, such as matplotlib, statsmodels, scikit-learn, and NLTK. It also covers using Python with external environments such as R, Fortran, C/C++, and Boost libraries.

Preface

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Free Chapter

1. Getting Started with Python Libraries

1. Getting Started with Python Libraries

Installing Python 3

Using IPython as a shell

Reading manual pages

Jupyter Notebook

NumPy arrays

A simple application

Where to find help and references

Listing modules inside the Python libraries

Visualizing data using Matplotlib

Summary

2. NumPy Arrays

2. NumPy Arrays

The NumPy array object

Creating a multidimensional array

Selecting NumPy array elements

NumPy numerical types

One-dimensional slicing and indexing

Manipulating array shapes

Creating array views and copies

Fancy indexing

Indexing with a list of locations

Indexing NumPy arrays with Booleans

Broadcasting NumPy arrays

Summary

References

3. The Pandas Primer

3. The Pandas Primer

Installing and exploring Pandas

The Pandas DataFrames

The Pandas Series

Querying data in Pandas

Statistics with Pandas DataFrames

Data aggregation with Pandas DataFrames

Concatenating and appending DataFrames

Joining DataFrames

Handling missing values

Dealing with dates

Pivot tables

Summary

References

4. Statistics and Linear Algebra

4. Statistics and Linear Algebra

Basic descriptive statistics with NumPy

Linear algebra with NumPy

Finding eigenvalues and eigenvectors with NumPy

NumPy random numbers

Creating a NumPy masked array

Summary

5. Retrieving, Processing, and Storing Data

5. Retrieving, Processing, and Storing Data

Writing CSV files with NumPy and Pandas

The binary .npy and pickle formats

Storing data with PyTables

Reading and writing Pandas DataFrames to HDF5 stores

Reading and writing to Excel with Pandas

Using REST web services and JSON

Reading and writing JSON with Pandas

Parsing RSS and Atom feeds

Parsing HTML with Beautiful Soup

Summary

Reference

6. Data Visualization

6. Data Visualization

The matplotlib subpackages

Basic matplotlib plots

Logarithmic plots

Scatter plots

Legends and annotations

Three-dimensional plots

Plotting in Pandas

Lag plots

Autocorrelation plots

Plot.ly

Summary

7. Signal Processing and Time Series

7. Signal Processing and Time Series

The statsmodels modules

Moving averages

Window functions

Defining cointegration

Autocorrelation

Autoregressive models

ARMA models

Generating periodic signals

Fourier analysis

Spectral analysis

Filtering

Summary

8. Working with Databases

8. Working with Databases

Lightweight access with sqlite3

Accessing databases from Pandas

SQLAlchemy

Pony ORM

Dataset - databases for lazy people

PyMongo and MongoDB

Storing data in Redis

Storing data in memcache

Apache Cassandra

Summary

9. Analyzing Textual Data and Social Media

9. Analyzing Textual Data and Social Media

Installing NLTK

About NLTK

Filtering out stopwords, names, and numbers

The bag-of-words model

Analyzing word frequencies

Naive Bayes classification

Sentiment analysis

Creating word clouds

Social network analysis

Summary

10. Predictive Analytics and Machine Learning

10. Predictive Analytics and Machine Learning

Preprocessing

Classification with logistic regression

Classification with support vector machines

Regression with ElasticNetCV

Support vector regression

Clustering with affinity propagation

Mean shift

Genetic algorithms

Neural networks

Decision trees

Summary

11. Environments Outside the Python Ecosystem and Cloud Computing

11. Environments Outside the Python Ecosystem and Cloud Computing

Exchanging information with Matlab/Octave

Installing rpy2 package

Interfacing with R

Sending NumPy arrays to Java

Integrating SWIG and NumPy

Integrating Boost and Python

Using Fortran code through f2py

PythonAnywhere Cloud

Summary

12. Performance Tuning, Profiling, and Concurrency

12. Performance Tuning, Profiling, and Concurrency

Profiling the code

Installing Cython

Calling C code

Creating a process pool with multiprocessing

Speeding up embarrassingly parallel for loops with Joblib

Comparing Bottleneck to NumPy functions

Performing MapReduce with Jug

Installing MPI for Python

IPython Parallel

Summary

A. Key Concepts

A. Key Concepts

B. Useful Functions

B. Useful Functions

Matplotlib

NumPy

Pandas

Scikit-learn

SciPy

C. Online Resources

C. Online Resources

Dataset - databases for lazy people

Dataset is a Python library which is basically a wrapper around SQLAlchemy. It claims to be so easy to use that even lazy people like it.

Install dataset as follows:

$ pip3 install dataset

Create a SQLite in-memory database and connect to it:

import dataset 
db = dataset.connect('sqlite:///:memory:')

Create a table called books:

table = db["books"]

Actually, the table in the database isn't created yet, since we haven't specified any columns. We only created a related object. The table schema is created automatically from calls to the insert() method. Give the insert() method dictionaries with book titles:

table.insert(dict(, author='Ivan Idris')) 
table.insert(dict(,  
author='Ivan Idris')) 
table.insert(dict(,  
author='Ivan Idris'))

Print the rows from the table as follows:

for row in db['books']: 
   print(row)

The following will be printed:

<dataset.persistence.util.ResultIter object at 0x10d4bf550>  
OrderedDict([('id', 1), ('title', "NumPy Beginner...

CONTINUE READING

83

Tech Concepts

36

Programming languages

73

Tech Tools

Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.

Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.

50+ new titles added per month and exclusive early access to books as they are being written.

Python Data Analysis, Second Edition

Search

Your notes and bookmarks