Book Image

The Data Wrangling Workshop - Second Edition

By : Brian Lipp, Shubhadeep Roychowdhury, Dr. Tirthajyoti Sarkar
Book Image

The Data Wrangling Workshop - Second Edition

By: Brian Lipp, Shubhadeep Roychowdhury, Dr. Tirthajyoti Sarkar

Overview of this book

While a huge amount of data is readily available to us, it is not useful in its raw form. For data to be meaningful, it must be curated and refined. If you’re a beginner, then The Data Wrangling Workshop will help to break down the process for you. You’ll start with the basics and build your knowledge, progressing from the core aspects behind data wrangling, to using the most popular tools and techniques. This book starts by showing you how to work with data structures using Python. Through examples and activities, you’ll understand why you should stay away from traditional methods of data cleaning used in other languages and take advantage of the specialized pre-built routines in Python. Later, you’ll learn how to use the same Python backend to extract and transform data from an array of sources, including the internet, large database vaults, and Excel financial tables. To help you prepare for more challenging scenarios, the book teaches you how to handle missing or incorrect data, and reformat it based on the requirements from your downstream analytics tool. By the end of this book, you will have developed a solid understanding of how to perform data wrangling with Python, and learned several techniques and best practices to extract, clean, transform, and format your data efficiently, from a diverse array of sources.

Related Content you might be interested in

Current Title:

Small book image
The Data Wrangling Workshop - Second Edition
Brian Lipp | Shubhadeep Roychowdhury | Dr. Tirthajyoti Sarkar
Jul, 2020 | 576 pages
Small book image
Mastering Exploratory Analysis with pandas
Harish Garg
Sep, 2018 | 140 pages
Table of Contents (11 chapters)
Preface
About the Book
1
1. Introduction to Data Wrangling with Python
1. Introduction to Data Wrangling with Python
Introduction
Importance of Data Wrangling
Python for Data Wrangling
Lists, Sets, Strings, Tuples, and Dictionaries
List Functions
Summary
Free Chapter
2
2. Advanced Operations on Built-In Data Structures
2. Advanced Operations on Built-In Data Structures
Introduction
Advanced Data Structures
Basic File Operations in Python
Summary
3
3. Introduction to NumPy, Pandas, and Matplotlib
3. Introduction to NumPy, Pandas, and Matplotlib
Introduction
NumPy Arrays
Advanced Mathematical Operations
Statistics and Visualization with NumPy and Pandas
The Definition of Statistical Measures – Central Tendency and Spread
Data Wrangling in Statistics and Visualization
Summary
4
4. A Deep Dive into Data Wrangling with Python
4. A Deep Dive into Data Wrangling with Python
Introduction
Subsetting, Filtering, and Grouping
Detecting Outliers and Handling Missing Values
Concatenating, Merging, and Joining
Useful Methods of Pandas
Summary
5
5. Getting Comfortable with Different Kinds of Data Sources
5. Getting Comfortable with Different Kinds of Data Sources
Introduction
Reading Data from Different Sources
Introduction to Beautiful Soup 4 and Web Page Parsing
Summary
6
6. Learning the Hidden Secrets of Data Wrangling
6. Learning the Hidden Secrets of Data Wrangling
Introduction
Advanced List Comprehension and the zip Function
Data Formatting
Identifying and Cleaning Outliers
Levenshtein Distance
Summary
7
7. Advanced Web Scraping and Data Gathering
7. Advanced Web Scraping and Data Gathering
Introduction
The Requests and BeautifulSoup Libraries
Reading Data from XML
Reading Data from an API
Fundamentals of Regular Expressions (RegEx)
Summary
8
8. RDBMS and SQL
8. RDBMS and SQL
Introduction
Refresher of RDBMS and SQL
Relation Mapping in Databases
Joins
Retrieving Specific Columns from a JOIN Query
Summary
9
9. Applications in Business Use Cases and Conclusion of the Course
9. Applications in Business Use Cases and Conclusion of the Course
Introduction
Applying Your Knowledge to a Data Wrangling Task
An Extension to Data Wrangling
Summary
Appendix
1. Introduction to Data Wrangling with Python
2. Advanced Operations on Built-In Data Structures
3. Introduction to NumPy, Pandas, and Matplotlib
4. A Deep Dive into Data Wrangling with Python
5. Getting Comfortable with Different Kinds of Data Sources
6. Learning the Hidden Secrets of Data Wrangling
7. Advanced Web Scraping and Data Gathering
8. RDBMS and SQL
9. Applications in Business Use Cases and Conclusion of the Course

Python for Data Wrangling

There is always a debate regarding whether to perform the wrangling process using an enterprise tool or a programming language and its associated frameworks. There are many commercial, enterprise-level tools for data formatting and preprocessing that do not involve much coding on the user's part. Some of these examples include the following:

  • General-purpose data analysis platforms, such as Microsoft Excel (with add-ins)
  • Statistical discovery package, such as JMP (from SAS)
  • Modeling platforms, such as RapidMiner
  • Analytics platforms from niche players that focus on data wrangling, such as Trifacta, Paxata, and Alteryx

However, programming languages such as Python and R provide more flexibility, control, and power compared to these off-the-shelf tools. This also explains their tremendous popularity in the data science domain:

Figure 1.2: Google trends worldwide over the last 5 years

Figure 1.2: Google trends worldwide over the last 5 years

Furthermore, as the volume, velocity, and variety (the three Vs of big data) of data undergo rapid changes, it is always a good idea to develop and nurture a significant amount of in-house expertise in data wrangling using fundamental programming frameworks so that an organization is not beholden to the whims and fancies of any particular enterprise platform for as basic a task as data wrangling.

A few of the obvious advantages of using an open source, free programming paradigm for data wrangling are as follows:

  • A general-purpose open-source paradigm puts no restrictions on any of the methods you can develop for the specific problem at hand.
  • There's a great ecosystem of fast, optimized, open-source libraries, focused on data analytics.
  • There's also growing support for connecting Python to every conceivable data source type.
  • There's an easy interface to basic statistical testing and quick visualization libraries to check data quality.
  • And there's a seamless interface of the data wrangling output with advanced machine learning models.

Python is the most popular language for machine learning and artificial intelligence these days. Let's take a look at a few data structures in Python.

Previous Section
End of Section 4
Next Section