Clean Data

Clean Data

By : Megan Squire

Buy this Book

Clean Data

By: Megan Squire

Buy this Book

Overview of this book

Is much of your time spent doing tedious tasks such as cleaning dirty data, accounting for lost data, and preparing data to be used by others? If so, then having the right tools makes a critical difference, and will be a great investment as you grow your data science expertise. The book starts by highlighting the importance of data cleaning in data science, and will show you how to reap rewards from reforming your cleaning process. Next, you will cement your knowledge of the basic concepts that the rest of the book relies on: file formats, data types, and character encodings. You will also learn how to extract and clean data stored in RDBMS, web files, and PDF documents, through practical examples. At the end of the book, you will be given a chance to tackle a couple of real-world projects.

Clean Data

Credits

About the Author

About the Reviewers

www.PacktPub.com

Preface

Free Chapter

Why Do You Need Clean Data?

A fresh perspective

The data science process

Communicating about data cleaning

Our data cleaning environment

An introductory example

Summary

Fundamentals – Formats, Types, and Encodings

File formats

Archiving and compression

Data types, nulls, and encodings

Summary

Workhorses of Clean Data – Spreadsheets and Text Editors

Spreadsheet data cleaning

Text editor data cleaning

An example project

Summary

Speaking the Lingua Franca – Data Conversions

Quick tool-based conversions

Converting with PHP

Converting with Python

The example project

Summary

Collecting and Cleaning Data from the Web

Understanding the HTML page structure

Method one – Python and regular expressions

Method two – Python and BeautifulSoup

Method three – Chrome Scraper

Example project – Extracting data from e-mail and web forums

Summary

Cleaning Data in PDF Files

Why is cleaning PDF files difficult?

Try simple solutions first – copying

Another technique to try – pdfMiner

Third choice – Tabula

When all else fails – the fourth technique

Summary

RDBMS Cleaning Techniques

Getting ready

Step one – download and examine Sentiment140

Step two – clean for database import

Step three – import the data into MySQL in a single table

Step four – clean the & character

Step five – clean other mystery characters

Step seven – separate user mentions, hashtags, and URLs

Step eight – cleaning for lookup tables

Summary

Best Practices for Sharing Your Clean Data

Preparing a clean data package

Documenting your data

Setting terms and licenses for your data

Publicizing your data

Summary

Stack Overflow Project

Step one – posing a question about Stack Overflow

Step two – collecting and storing the Stack Overflow data

Step three – cleaning the data

Step four – analyzing the data

Step five – visualizing the data

Step six – problem resolution

Moving from test tables to full tables

Summary

Twitter Project

Step one – posing a question about an archive of tweets

Step two – collecting the data

Step three – data cleaning

Step four – simple data analysis

Step five – visualizing the data

Step six – problem resolution

Moving this process into full (non-test) tables

Summary

Index

Customer Reviews

5 star

4 star

3 star

2 star

1 star

Step five – visualizing the data

To build a little D3-enabled graph, we could follow the same procedure we used in Chapter 9, Stack Overflow Project, in which we made a PHP script that queries the database, and then our JavaScript uses the results as the live input to a bar graph. Alternatively, we could generate a CSV file with Python and let D3 generate its graph from those results. Since we already performed the PHP method in the previous chapter, let's use the CSV file method here, just for variety. This is also a good excuse to continue on with Python in this chapter, since this is already the language we have been using.

The following script connects to the database, selects out the top 15 most-used URLs and their counts, and writes the entire thing to a CSV file:

import csv
import MySQLdb

# Open database connection
db = MySQLdb.connect(host="localhost",
    user="username", 
    passwd="password", 
    db="ferguson", 
    use_unicode=True, 
    charset="utf8")
cursor = db.cursor()...

Clean Data

By : Megan Squire

Clean Data

By: Megan Squire

Overview of this book

Related Content you might be interested in

Current Title:

Clean Data

Step five – visualizing the data