Browse Library
Sign In Start Free Trial
Close icon Search icon
Account
Browse Library Sign In Start Free Trial

Add to playlist

Create a Playlist

Modal Close icon
You need to login to use this feature.
  • Book Overview & Buying Pig Design Patterns
  • Table Of Contents Toc
Pig Design Patterns

Pig Design Patterns

By : Pradeep Pasupuleti
4.3 (6)
Buy this Book
close
close
Pig Design Patterns

Pig Design Patterns

4.3 (6)
By: Pradeep Pasupuleti
Buy this Book

Overview of this book

Table of Contents (16 chapters)
close
close
close
Pig Design Patterns
Icon
Pig Design Patterns
Credits
Icon
Credits
Foreword
Icon
Foreword
About the Author
Icon
About the Author
Acknowledgments
Icon
Acknowledgments
About the Reviewers
Icon
About the Reviewers
www.PacktPub.com
Icon
www.PacktPub.com
Preface
Icon
Preface
Lock Free Chapter
1
Setting the Context for Design Patterns in Pig
Icon
Setting the Context for Design Patterns in Pig
Icon
Understanding design patterns
Icon
The scope of design patterns in Pig
Icon
Hadoop demystified – a quick reckoner
Icon
Pig – a quick intro
Icon
Understanding Pig through the code
Icon
Summary
2
Data Ingest and Egress Patterns
Icon
Data Ingest and Egress Patterns
Icon
The context of data ingest and egress
Icon
Types of data in the enterprise
Icon
Ingest and egress patterns for multistructured data
Icon
The ingress and egress patterns for the NoSQL data
Icon
The ingress and egress patterns for structured data
Icon
The ingress and egress patterns for semi-structured data
Icon
JSON ingress and egress patterns
Icon
Summary
3
Data Profiling Patterns
Icon
Data Profiling Patterns
Icon
Data profiling for Big Data
Icon
Rationale for using Pig in data profiling
Icon
The data type inference pattern
Icon
The basic statistical profiling pattern
Icon
The pattern-matching pattern
Icon
The string profiling pattern
Icon
The unstructured text profiling pattern
Icon
Summary
4
Data Validation and Cleansing Patterns
chevron up
Icon
Data Validation and Cleansing Patterns
Icon
Data validation and cleansing for Big Data
Icon
Choosing Pig for validation and cleansing
Icon
The constraint validation and cleansing design pattern
Icon
The regex validation and cleansing design pattern
Icon
The corrupt data validation and cleansing design pattern
Icon
The unstructured text data validation and cleansing design pattern
Icon
Summary
5
Data Transformation Patterns
Icon
Data Transformation Patterns
Icon
Data transformation processes
Icon
The structured-to-hierarchical transformation pattern
Icon
The data normalization pattern
Icon
The data integration pattern
Icon
The aggregation pattern
Icon
The data generalization pattern
Icon
Summary
6
Understanding Data Reduction Patterns
Icon
Understanding Data Reduction Patterns
Icon
Data reduction – a quick introduction
Icon
Data reduction considerations for Big Data
Icon
Dimensionality reduction – the Principal Component Analysis design pattern
Icon
Numerosity reduction – the histogram design pattern
Icon
Numerosity reduction – sampling design pattern
Icon
Numerosity reduction – clustering design pattern
Icon
Summary
7
Advanced Patterns and Future Work
Icon
Advanced Patterns and Future Work
Icon
The clustering pattern
Icon
The topic discovery pattern
Icon
The natural language processing pattern
Icon
The classification pattern
Icon
Future trends
Icon
Summary
1
Index
Icon
Index

Chapter 4. Data Validation and Cleansing Patterns

In the previous chapter, you have studied the various patterns related to data profiling, through which you understood the different ways to get vital information about the attributes, content, context, structure, and condition of data residing in the Hadoop cluster. These data profiling patterns are applied in the data lifecycle before initiating the process of cleaning the data into a more useful form.

The following are the design patterns covered in this chapter:

  • Constraint validation and cleansing pattern: This explains the validation of the data against a set of constraints to check if there are any missing values, if the values are within a range specified by a business rule or if the values conform to referential integrity and unique constraints. Depending on the business rule, either the invalid records are removed or appropriate cleansing steps are applied to the invalid data.

  • Regex validation and cleansing pattern: This demonstrates...

CONTINUE READING
83
Tech Concepts
36
Programming languages
73
Tech Tools
Icon Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.
Icon Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.
Icon 50+ new titles added per month and exclusive early access to books as they are being written.
Pig Design Patterns
End of Section 4
Next Section
notes
bookmark Notes and Bookmarks search Search in title playlist Add to playlist font-size Font size

Change the font size

margin-width Margin width

Change margin width

day-mode Day/Sepia/Night Modes

Change background colour

Close icon Search
Country selected
Close icon Your notes and bookmarks

Confirmation

Modal Close icon
claim successful
Order Page Read Now

Buy this book with your credits?

Modal Close icon
Are you sure you want to buy this book with one of your credits?
Close
YES, BUY

Submit Your Feedback

Modal Close icon
Modal Close icon
Modal Close icon