Book Image

Pig Design Patterns

By : Pradeep Pasupuleti
Book Image

Pig Design Patterns

By: Pradeep Pasupuleti

Overview of this book

Related Content you might be interested in

Current Title:

Small book image
Pig Design Patterns
Pradeep Pasupuleti
Apr, 2014 | 310 pages
Small book image
Real-World Next.js
Michele Riva
Feb, 2022 | 366 pages
Small book image
Isomorphic JavaScript Web Development
Konstantin Tarkus | Tomas Alabes
Oct, 2017 | 226 pages
Small book image
Build Applications with Meteor
Dobrin Ganev
May, 2017 | 388 pages
Table of Contents (16 chapters)
Pig Design Patterns
Pig Design Patterns
Credits
Credits
Foreword
Foreword
About the Author
About the Author
Acknowledgments
Acknowledgments
About the Reviewers
About the Reviewers
www.PacktPub.com
www.PacktPub.com
Preface
Preface
Free Chapter
1
Setting the Context for Design Patterns in Pig
Setting the Context for Design Patterns in Pig
Understanding design patterns
The scope of design patterns in Pig
Hadoop demystified – a quick reckoner
Pig – a quick intro
Understanding Pig through the code
Summary
2
Data Ingest and Egress Patterns
Data Ingest and Egress Patterns
The context of data ingest and egress
Types of data in the enterprise
Ingest and egress patterns for multistructured data
The ingress and egress patterns for the NoSQL data
The ingress and egress patterns for structured data
The ingress and egress patterns for semi-structured data
JSON ingress and egress patterns
Summary
3
Data Profiling Patterns
Data Profiling Patterns
Data profiling for Big Data
Rationale for using Pig in data profiling
The data type inference pattern
The basic statistical profiling pattern
The pattern-matching pattern
The string profiling pattern
The unstructured text profiling pattern
Summary
4
Data Validation and Cleansing Patterns
Data Validation and Cleansing Patterns
Data validation and cleansing for Big Data
Choosing Pig for validation and cleansing
The constraint validation and cleansing design pattern
The regex validation and cleansing design pattern
The corrupt data validation and cleansing design pattern
The unstructured text data validation and cleansing design pattern
Summary
5
Data Transformation Patterns
Data Transformation Patterns
Data transformation processes
The structured-to-hierarchical transformation pattern
The data normalization pattern
The data integration pattern
The aggregation pattern
The data generalization pattern
Summary
6
Understanding Data Reduction Patterns
Understanding Data Reduction Patterns
Data reduction – a quick introduction
Data reduction considerations for Big Data
Dimensionality reduction – the Principal Component Analysis design pattern
Numerosity reduction – the histogram design pattern
Numerosity reduction – sampling design pattern
Numerosity reduction – clustering design pattern
Summary
7
Advanced Patterns and Future Work
Advanced Patterns and Future Work
The clustering pattern
The topic discovery pattern
The natural language processing pattern
The classification pattern
Future trends
Summary
Index
Index

Chapter 5. Data Transformation Patterns

In the last chapter, you learned about various patterns related to data validation and cleansing from which you understood that there are ways to detect and remove incorrect or inaccurate records from the data. By the time data validation and cleansing is complete, the inconsistencies in the data are identified even before the data is used in the next steps of the analytics life cycle; then, the inconsistent data is replaced, modified, or deleted to make it more consistent.

In this chapter, you will learn about various design patterns related to data transformation, such as structured to hierarchical, normalization, integration, aggregation, and generalization design patterns.

Previous Section
End of Section 1
Next Section