Advanced Patterns and Future Work | Pig Design Patterns

Sign In Start Free Trial

Book Overview & Buying
Table Of Contents

Pig Design Patterns

By : Pradeep Pasupuleti

4.3 (6)

Pig Design Patterns

4.3 (6)

By: Pradeep Pasupuleti

Overview of this book

Pig Design Patterns

Pig Design Patterns

Credits

Credits

Foreword

Foreword

About the Author

About the Author

Acknowledgments

Acknowledgments

About the Reviewers

About the Reviewers

www.PacktPub.com

www.PacktPub.com

Preface

Preface

Free Chapter

Setting the Context for Design Patterns in Pig

Setting the Context for Design Patterns in Pig

Understanding design patterns

The scope of design patterns in Pig

Hadoop demystified – a quick reckoner

Pig – a quick intro

Understanding Pig through the code

Summary

Data Ingest and Egress Patterns

Data Ingest and Egress Patterns

The context of data ingest and egress

Types of data in the enterprise

Ingest and egress patterns for multistructured data

The ingress and egress patterns for the NoSQL data

The ingress and egress patterns for structured data

The ingress and egress patterns for semi-structured data

JSON ingress and egress patterns

Summary

Data Profiling Patterns

Data Profiling Patterns

Data profiling for Big Data

Rationale for using Pig in data profiling

The data type inference pattern

The basic statistical profiling pattern

The pattern-matching pattern

The string profiling pattern

The unstructured text profiling pattern

Summary

Data Validation and Cleansing Patterns

Data Validation and Cleansing Patterns

Data validation and cleansing for Big Data

Choosing Pig for validation and cleansing

The constraint validation and cleansing design pattern

The regex validation and cleansing design pattern

The corrupt data validation and cleansing design pattern

The unstructured text data validation and cleansing design pattern

Summary

Data Transformation Patterns

Data Transformation Patterns

Data transformation processes

The structured-to-hierarchical transformation pattern

The data normalization pattern

The data integration pattern

The aggregation pattern

The data generalization pattern

Summary

Understanding Data Reduction Patterns

Understanding Data Reduction Patterns

Data reduction – a quick introduction

Data reduction considerations for Big Data

Dimensionality reduction – the Principal Component Analysis design pattern

Numerosity reduction – the histogram design pattern

Numerosity reduction – sampling design pattern

Numerosity reduction – clustering design pattern

Summary

Advanced Patterns and Future Work

Advanced Patterns and Future Work

The clustering pattern

The topic discovery pattern

The natural language processing pattern

The classification pattern

Future trends

Summary

Index

Index

The classification pattern

This design pattern explores the implementation of classification using Pig and Mahout.

Background

Classification is one of the core concepts of predictive analytics; it is a technique in which data is labeled into categories or groups according to its characteristics. Classification is a simplified way to make decisions based on the data and its attributes. For example, in a survey questionnaire, we choose an appropriate answer or select a particular check box for a given question. Here, we are making a decision from a finite group of choices (check boxes or answers) provided to us. Sometimes, the number of choices can be as small as two (yes/no). In these cases, classification uses specific information on the input data to choose a single output from a group of predetermined responses.

Consider the case of a human being making a decision to buy a pizza. The input data for this decision making includes the price, toppings, type of crust, and so on, for multiple pizzas...

CONTINUE READING

83

Tech Concepts

36

Programming languages

73

Tech Tools

Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.

Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.

50+ new titles added per month and exclusive early access to books as they are being written.

Pig Design Patterns

Search

Your notes and bookmarks