Browse Library
Sign In Start Free Trial
Close icon Search icon
Account
Browse Library Sign In Start Free Trial

Add to playlist

Create a Playlist

Modal Close icon
You need to login to use this feature.
  • Book Overview & Buying Pig Design Patterns
  • Table Of Contents Toc
Pig Design Patterns

Pig Design Patterns

By : Pradeep Pasupuleti
4.3 (6)
Buy this Book
close
close
Pig Design Patterns

Pig Design Patterns

4.3 (6)
By: Pradeep Pasupuleti
Buy this Book

Overview of this book

Table of Contents (16 chapters)
close
close
close
Pig Design Patterns
Icon
Pig Design Patterns
Credits
Icon
Credits
Foreword
Icon
Foreword
About the Author
Icon
About the Author
Acknowledgments
Icon
Acknowledgments
About the Reviewers
Icon
About the Reviewers
www.PacktPub.com
Icon
www.PacktPub.com
Preface
Icon
Preface
Lock Free Chapter
1
Setting the Context for Design Patterns in Pig
Icon
Setting the Context for Design Patterns in Pig
Icon
Understanding design patterns
Icon
The scope of design patterns in Pig
Icon
Hadoop demystified – a quick reckoner
Icon
Pig – a quick intro
Icon
Understanding Pig through the code
Icon
Summary
2
Data Ingest and Egress Patterns
Icon
Data Ingest and Egress Patterns
Icon
The context of data ingest and egress
Icon
Types of data in the enterprise
Icon
Ingest and egress patterns for multistructured data
Icon
The ingress and egress patterns for the NoSQL data
Icon
The ingress and egress patterns for structured data
Icon
The ingress and egress patterns for semi-structured data
Icon
JSON ingress and egress patterns
Icon
Summary
3
Data Profiling Patterns
chevron up
Icon
Data Profiling Patterns
Icon
Data profiling for Big Data
Icon
Rationale for using Pig in data profiling
Icon
The data type inference pattern
Icon
The basic statistical profiling pattern
Icon
The pattern-matching pattern
Icon
The string profiling pattern
Icon
The unstructured text profiling pattern
Icon
Summary
4
Data Validation and Cleansing Patterns
Icon
Data Validation and Cleansing Patterns
Icon
Data validation and cleansing for Big Data
Icon
Choosing Pig for validation and cleansing
Icon
The constraint validation and cleansing design pattern
Icon
The regex validation and cleansing design pattern
Icon
The corrupt data validation and cleansing design pattern
Icon
The unstructured text data validation and cleansing design pattern
Icon
Summary
5
Data Transformation Patterns
Icon
Data Transformation Patterns
Icon
Data transformation processes
Icon
The structured-to-hierarchical transformation pattern
Icon
The data normalization pattern
Icon
The data integration pattern
Icon
The aggregation pattern
Icon
The data generalization pattern
Icon
Summary
6
Understanding Data Reduction Patterns
Icon
Understanding Data Reduction Patterns
Icon
Data reduction – a quick introduction
Icon
Data reduction considerations for Big Data
Icon
Dimensionality reduction – the Principal Component Analysis design pattern
Icon
Numerosity reduction – the histogram design pattern
Icon
Numerosity reduction – sampling design pattern
Icon
Numerosity reduction – clustering design pattern
Icon
Summary
7
Advanced Patterns and Future Work
Icon
Advanced Patterns and Future Work
Icon
The clustering pattern
Icon
The topic discovery pattern
Icon
The natural language processing pattern
Icon
The classification pattern
Icon
Future trends
Icon
Summary
1
Index
Icon
Index

The basic statistical profiling pattern

This section describes the basic statistical profiling design pattern in which we use Pig scripts to apply statistical functions to capture important information about data quality.

Background

The previous design pattern depicts one way of inferring the data type. The next logical step in the data profiling process is to evaluate the quality metrics of the values. This is done by collecting and analyzing the data by applying statistical methods. These statistics provide a high-level overview of the suitability of the data for a particular analytical problem, and uncover potential problems early in the data lifecycle management.

Motivation

The basic statistical profiling design pattern helps to create data quality metadata that includes basic statistics, such as mean, median, mode, maximum, minimum, and standard deviation. These stats give you a complete snapshot of the entire data field, and tracking these statistics over time will give insights into the...

CONTINUE READING
83
Tech Concepts
36
Programming languages
73
Tech Tools
Icon Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.
Icon Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.
Icon 50+ new titles added per month and exclusive early access to books as they are being written.
Pig Design Patterns
End of Section 3
Next Section
notes
bookmark Notes and Bookmarks search Search in title playlist Add to playlist font-size Font size

Change the font size

margin-width Margin width

Change margin width

day-mode Day/Sepia/Night Modes

Change background colour

Close icon Search
Country selected
Close icon Your notes and bookmarks

Confirmation

Modal Close icon
claim successful
Order Page Read Now

Buy this book with your credits?

Modal Close icon
Are you sure you want to buy this book with one of your credits?
Close
YES, BUY

Submit Your Feedback

Modal Close icon
Modal Close icon
Modal Close icon