Browse Library
Sign In Start Free Trial
Close icon Search icon
Account
Browse Library Sign In Start Free Trial

Add to playlist

Create a Playlist

Modal Close icon
You need to login to use this feature.
  • Book Overview & Buying Pig Design Patterns
  • Table Of Contents Toc
Pig Design Patterns

Pig Design Patterns

By : Pradeep Pasupuleti
4.3 (6)
Buy this Book
close
close
Pig Design Patterns

Pig Design Patterns

4.3 (6)
By: Pradeep Pasupuleti
Buy this Book

Overview of this book

Table of Contents (16 chapters)
close
close
close
Pig Design Patterns
Icon
Pig Design Patterns
Credits
Icon
Credits
Foreword
Icon
Foreword
About the Author
Icon
About the Author
Acknowledgments
Icon
Acknowledgments
About the Reviewers
Icon
About the Reviewers
www.PacktPub.com
Icon
www.PacktPub.com
Preface
Icon
Preface
Lock Free Chapter
1
Setting the Context for Design Patterns in Pig
Icon
Setting the Context for Design Patterns in Pig
Icon
Understanding design patterns
Icon
The scope of design patterns in Pig
Icon
Hadoop demystified – a quick reckoner
Icon
Pig – a quick intro
Icon
Understanding Pig through the code
Icon
Summary
2
Data Ingest and Egress Patterns
Icon
Data Ingest and Egress Patterns
Icon
The context of data ingest and egress
Icon
Types of data in the enterprise
Icon
Ingest and egress patterns for multistructured data
Icon
The ingress and egress patterns for the NoSQL data
Icon
The ingress and egress patterns for structured data
Icon
The ingress and egress patterns for semi-structured data
Icon
JSON ingress and egress patterns
Icon
Summary
3
Data Profiling Patterns
chevron up
Icon
Data Profiling Patterns
Icon
Data profiling for Big Data
Icon
Rationale for using Pig in data profiling
Icon
The data type inference pattern
Icon
The basic statistical profiling pattern
Icon
The pattern-matching pattern
Icon
The string profiling pattern
Icon
The unstructured text profiling pattern
Icon
Summary
4
Data Validation and Cleansing Patterns
Icon
Data Validation and Cleansing Patterns
Icon
Data validation and cleansing for Big Data
Icon
Choosing Pig for validation and cleansing
Icon
The constraint validation and cleansing design pattern
Icon
The regex validation and cleansing design pattern
Icon
The corrupt data validation and cleansing design pattern
Icon
The unstructured text data validation and cleansing design pattern
Icon
Summary
5
Data Transformation Patterns
Icon
Data Transformation Patterns
Icon
Data transformation processes
Icon
The structured-to-hierarchical transformation pattern
Icon
The data normalization pattern
Icon
The data integration pattern
Icon
The aggregation pattern
Icon
The data generalization pattern
Icon
Summary
6
Understanding Data Reduction Patterns
Icon
Understanding Data Reduction Patterns
Icon
Data reduction – a quick introduction
Icon
Data reduction considerations for Big Data
Icon
Dimensionality reduction – the Principal Component Analysis design pattern
Icon
Numerosity reduction – the histogram design pattern
Icon
Numerosity reduction – sampling design pattern
Icon
Numerosity reduction – clustering design pattern
Icon
Summary
7
Advanced Patterns and Future Work
Icon
Advanced Patterns and Future Work
Icon
The clustering pattern
Icon
The topic discovery pattern
Icon
The natural language processing pattern
Icon
The classification pattern
Icon
Future trends
Icon
Summary
1
Index
Icon
Index

The string profiling pattern

This section describes the string profiling design pattern in which we use Pig scripts on textual data to know important statistics.

Background

A majority of Big Data implementations deal with text data embedded in columns. To gain insight from these columns, they have to be integrated with other enterprise-structured data. This design pattern elaborates a few of the ways that help understand the quality of textual data.

Motivation

The quality of textual data can be ascertained by applying basic statistical techniques on the values of the attributes. Finding the string length is the most important dimension in selecting the appropriate data types and sizes for the target system. You can use the maximum and minimum string lengths to determine, at a glance, if the data ingested into Hadoop meets a given constraint. While dealing with data sizes in the petabyte range, limiting the character count to be just large enough optimizes storage and computation by cutting down...

CONTINUE READING
83
Tech Concepts
36
Programming languages
73
Tech Tools
Icon Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.
Icon Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.
Icon 50+ new titles added per month and exclusive early access to books as they are being written.
Pig Design Patterns
End of Section 3
Next Section
notes
bookmark Notes and Bookmarks search Search in title playlist Add to playlist font-size Font size

Change the font size

margin-width Margin width

Change margin width

day-mode Day/Sepia/Night Modes

Change background colour

Close icon Search
Country selected
Close icon Your notes and bookmarks

Confirmation

Modal Close icon
claim successful
Order Page Read Now

Buy this book with your credits?

Modal Close icon
Are you sure you want to buy this book with one of your credits?
Close
YES, BUY

Submit Your Feedback

Modal Close icon
Modal Close icon
Modal Close icon