Clean Data

Clean Data

By : Megan Squire

Buy this Book

Clean Data

By: Megan Squire

Buy this Book

Overview of this book

Is much of your time spent doing tedious tasks such as cleaning dirty data, accounting for lost data, and preparing data to be used by others? If so, then having the right tools makes a critical difference, and will be a great investment as you grow your data science expertise. The book starts by highlighting the importance of data cleaning in data science, and will show you how to reap rewards from reforming your cleaning process. Next, you will cement your knowledge of the basic concepts that the rest of the book relies on: file formats, data types, and character encodings. You will also learn how to extract and clean data stored in RDBMS, web files, and PDF documents, through practical examples. At the end of the book, you will be given a chance to tackle a couple of real-world projects.

Clean Data

Credits

About the Author

About the Reviewers

www.PacktPub.com

Preface

Free Chapter

Why Do You Need Clean Data?

A fresh perspective

The data science process

Communicating about data cleaning

Our data cleaning environment

An introductory example

Summary

Fundamentals – Formats, Types, and Encodings

File formats

Archiving and compression

Data types, nulls, and encodings

Summary

Workhorses of Clean Data – Spreadsheets and Text Editors

Spreadsheet data cleaning

Text editor data cleaning

An example project

Summary

Speaking the Lingua Franca – Data Conversions

Quick tool-based conversions

Converting with PHP

Converting with Python

The example project

Summary

Collecting and Cleaning Data from the Web

Understanding the HTML page structure

Method one – Python and regular expressions

Method two – Python and BeautifulSoup

Method three – Chrome Scraper

Example project – Extracting data from e-mail and web forums

Summary

Cleaning Data in PDF Files

Why is cleaning PDF files difficult?

Try simple solutions first – copying

Another technique to try – pdfMiner

Third choice – Tabula

When all else fails – the fourth technique

Summary

RDBMS Cleaning Techniques

Getting ready

Step one – download and examine Sentiment140

Step two – clean for database import

Step three – import the data into MySQL in a single table

Step four – clean the & character

Step five – clean other mystery characters

Step seven – separate user mentions, hashtags, and URLs

Step eight – cleaning for lookup tables

Summary

Best Practices for Sharing Your Clean Data

Preparing a clean data package

Documenting your data

Setting terms and licenses for your data

Publicizing your data

Summary

Stack Overflow Project

Step one – posing a question about Stack Overflow

Step two – collecting and storing the Stack Overflow data

Step three – cleaning the data

Step four – analyzing the data

Step five – visualizing the data

Step six – problem resolution

Moving from test tables to full tables

Summary

Twitter Project

Step one – posing a question about an archive of tweets

Step two – collecting the data

Step three – data cleaning

Step four – simple data analysis

Step five – visualizing the data

Step six – problem resolution

Moving this process into full (non-test) tables

Summary

Index

Customer Reviews

5 star

4 star

3 star

2 star

1 star

Preparing a clean data package

In this section, we delve into the many important questions that need to be answered before you can release a data package for general consumption.

How do you want people to access your data? If it is in a database, do you want users to be able to log in and run SQL commands on it? Or do you want to create downloadable flat text files for them to use? Do you need to create an API for the data? How much data do you have anyway, and do you want different levels of access for different parts of the data?

The technical aspects of how you want to share your clean data are extremely important. In general, it is probably a good idea to start with the simple things and move to a more sophisticated distribution plan when and if you need to. The following are some options for distributing data, in the order of the least complicated to the most complicated. Of course, with greater sophistication comes greater benefits:

Compressed plain text – This is a very low-stakes distribution...

Clean Data

By : Megan Squire

Clean Data

By: Megan Squire

Overview of this book

Related Content you might be interested in

Current Title:

Clean Data

Preparing a clean data package