Book Image

Practical Data Wrangling

By : Allan Visochek
Book Image

Practical Data Wrangling

By: Allan Visochek

Overview of this book

Around 80% of time in data analysis is spent on cleaning and preparing data for analysis. This is, however, an important task, and is a prerequisite to the rest of the data analysis workflow, including visualization, analysis and reporting. Python and R are considered a popular choice of tool for data analysis, and have packages that can be best used to manipulate different kinds of data, as per your requirements. This book will show you the different data wrangling techniques, and how you can leverage the power of Python and R packages to implement them. You’ll start by understanding the data wrangling process and get a solid foundation to work with different types of data. You’ll work with different data structures and acquire and parse data from various locations. You’ll also see how to reshape the layout of data and manipulate, summarize, and join data sets. Finally, we conclude with a quick primer on accessing and processing data from databases, conducting data exploration, and storing and retrieving data quickly using databases. The book includes practical examples on each of these points using simple and real-world data sets to give you an easier understanding. By the end of the book, you’ll have a thorough understanding of all the data wrangling concepts and how to implement them in the best possible way.

Related Content you might be interested in

Current Title:

Small book image
Practical Data Wrangling
Allan Visochek
Nov, 2017 | 204 pages
No titles found
Table of Contents (16 chapters)
Title Page
Title Page
Credits
Credits
About the Author
About the Author
About the Reviewer
About the Reviewer
www.PacktPub.com
www.PacktPub.com
Customer Feedback
Customer Feedback
Preface
Preface
Free Chapter
1
Programming with Data
Programming with Data
Understanding data wrangling
The tools for data wrangling
Summary
2
Introduction to Programming in Python
Introduction to Programming in Python
External resources
Logistical overview
Running programs in python
Data types, variables, and the Python shell
Compound statements
Making annotations within programs
A programmer's resources
 Summary
3
Reading, Exploring, and Modifying Data - Part I
Reading, Exploring, and Modifying Data - Part I
External resources
Logistical overview
Introducing a basic data wrangling work flow
Introducing the JSON file format
Opening and closing a file in Python using file I/O
Reading the contents of a file
Exploring the contents of a data file
Modifying a dataset
Outputting the modified data to a new file
Specifying input and output file names in the Terminal
Summary
4
Reading, Exploring, and Modifying Data - Part II
Reading, Exploring, and Modifying Data - Part II
Logistical overview
Understanding the CSV format
Introducing the CSV module
Using the CSV module to read CSV data
Using the CSV module to write CSV data
Using the pandas module to read and process data
Handling non-standard CSV encoding and dialect
Understanding XML
Using the XML module to parse XML data
Summary
5
Manipulating Text Data - An Introduction to Regular Expressions
Manipulating Text Data - An Introduction to Regular Expressions
Logistical overview
Understanding the need for pattern recognition
Introducting regular expressions
Looking for patterns
Quantifying the existence of patterns
Extracting patterns
Summary
6
Cleaning Numerical Data - An Introduction to R and RStudio
Cleaning Numerical Data - An Introduction to R and RStudio
Logistical overview
Introducing R and RStudio
Familiarizing yourself with RStudio
Conducting basic outlier detection and removal
Handling NA values
Variable names and contents
Summary
7
Simplifying Data Manipulation with dplyr
Simplifying Data Manipulation with dplyr
Logistical overview
Introducing dplyr
Getting started with dplyr
Chaining operations together
Filtering the rows of a dataframe
Summarizing data by category
Rewriting code using dplyr
Summary
8
Getting Data from the Web
Getting Data from the Web
Logistical overview
Introducing APIs
Using Python to retrieve data from APIs
Using URL parameters to filter the results
Summary
9
Working with Large Datasets
Working with Large Datasets
Logistical overview 
Understanding computer memory
Understanding databases
Introducing MongoDB
Interfacing with MongoDB from Python
Summary

Using Python to retrieve data from APIs

The first step is to import the requests module. I will also import the json module, which can be used to output the data retrieved from the API to a JSON file. In the beginning of get_recent_issues.py, the following code imports the requests and json modules:

import requests
import json

The next step is to create a string called base URL. The base URL is the beginning part of the URL, and can be followed by additional parameters. For now, the base URL is all you need, but in the next section you will build on it in order get data iteratively. In the following continuation of get_recent_issues.py, a string is created containing the base URL for the issues resource of the Seeclickfix API:

import requests
import json

## build a base url which is the endpoint to the api
## for this script, this is all you need
base_url = "https://seeclickfix.com/api/v2/issues?"

The next step is to replicate the action of the browser. In other words, you will need to submit...

Previous Section
End of Section 4
Next Section