6. Accessing Data | Learning Pandas

Sign In Start Free Trial

Book Overview & Buying
Table Of Contents

Learning Pandas

By : Michael Heydt

4.2 (10)

Learning Pandas

4.2 (10)

By: Michael Heydt

Overview of this book

If you are a Python programmer who wants to get started with performing data analysis using pandas and Python, this is the book for you. Some experience with statistical analysis would be helpful but is not mandatory.

Preface

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Free Chapter

1. A Tour of pandas

1. A Tour of pandas

pandas and why it is important

pandas and IPython Notebooks

Referencing pandas in the application

Primary pandas objects

Loading data from files and the Web

Simplicity of visualization of pandas data

Summary

2. Installing pandas

2. Installing pandas

Getting Anaconda

Installing Anaconda

Ensuring pandas is up to date

Running a small pandas sample in IPython

Starting the IPython Notebook server

Installing and running IPython Notebooks

Using Wakari for pandas

Summary

3. NumPy for pandas

3. NumPy for pandas

Installing and importing NumPy

Benefits and characteristics of NumPy arrays

Creating NumPy arrays and performing basic array operations

Selecting array elements

Logical operations on arrays

Slicing arrays

Reshaping arrays

Combining arrays

Splitting arrays

Useful numerical methods of NumPy arrays

Summary

4. The pandas Series Object

4. The pandas Series Object

The Series object

Importing pandas

Creating Series

Size, shape, uniqueness, and counts of values

Peeking at data with heads, tails, and take

Looking up values in Series

Arithmetic operations

The special case of Not-A-Number (NaN)

Boolean selection

Reindexing a Series

Slicing a Series

Summary

5. The pandas DataFrame Object

5. The pandas DataFrame Object

Creating DataFrame from scratch

Example data

Selecting columns of a DataFrame

Selecting rows and values of a DataFrame using the index

Selecting rows of a DataFrame by Boolean selection

Modifying the structure and content of DataFrame

Arithmetic on a DataFrame

Resetting and reindexing

Hierarchical indexing

Summarized data and descriptive statistics

Summary

6. Accessing Data

6. Accessing Data

Setting up the IPython notebook

Reading and writing JSON files

Accessing data on the web and in the cloud

Reading and writing from/to SQL databases

Reading data from remote data services

Summary

7. Tidying Up Your Data

7. Tidying Up Your Data

What is tidying your data?

Setting up the IPython notebook

Working with missing data

Handling duplicate data

Transforming Data

Summary

8. Combining and Reshaping Data

8. Combining and Reshaping Data

Setting up the IPython notebook

Concatenating data

Merging and joining data

Stacking and unstacking

Performance benefits of stacked data

Summary

9. Grouping and Aggregating Data

9. Grouping and Aggregating Data

Setting up the IPython notebook

The split, apply, and combine (SAC) pattern

Split

Apply

Discretization and Binning

Summary

10. Time-series Data

10. Time-series Data

Setting up the IPython notebook

Representation of dates, time, and intervals

Introducing time-series data

Calculating new dates using offsets

Handling holidays using calendars

Normalizing timestamps using time zones

Manipulating time-series data

Summary

11. Visualization

11. Visualization

Setting up the IPython notebook

Plotting basics with pandas

Common plots used in statistical analyses

Multiple plots in a single chart

Summary

12. Applications to Finance

12. Applications to Finance

Setting up the IPython notebook

Obtaining and organizing stock data from Yahoo!

Plotting time-series prices

Performing a moving-average calculation

Volatility calculation

Determining risk relative to expected returns

Summary

Index

Index

Reading and writing from/to SQL databases

pandas can read data from any SQL databases that support Python data adapters, that respect the Python DB-API. Reading is performed using the pandas.io.sql.read_sql() function and writing to SQL databases using the .to_sql() method of DataFrame.

As an example of writing, the following reads the stock data from msft.csv and aapl.csv. It then makes a connection to a SQLite3 database file. If the file does not exist, it creates it on the fly. It then writes the MSFT data to a table named STOCK_DATA. If the table did not exist, it is created. If it exists, all the data is replaced with the MSFT data. It then appends the AAPL stock data to that table:

In [33]:
   # reference SQLite
   import sqlite3

   # read in the stock data from CSV
   msft = pd.read_csv("data/msft.csv")
   msft["Symbol"]="MSFT"
   aapl = pd.read_csv("data/aapl.csv")
   aapl["Symbol"]="AAPL"

   # create connection
   connection = sqlite3.connect("data/stocks.sqlite")
   # .to_sql...

CONTINUE READING

83

Tech Concepts

36

Programming languages

73

Tech Tools

Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.

Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.

50+ new titles added per month and exclusive early access to books as they are being written.

Learning Pandas

Search

Your notes and bookmarks