Data Transformation Patterns | Pig Design Patterns

Sign In Start Free Trial

Book Overview & Buying
Table Of Contents

Pig Design Patterns

By : Pradeep Pasupuleti

4.3 (6)

Pig Design Patterns

4.3 (6)

By: Pradeep Pasupuleti

Overview of this book

Pig Design Patterns

Pig Design Patterns

Credits

Credits

Foreword

Foreword

About the Author

About the Author

Acknowledgments

Acknowledgments

About the Reviewers

About the Reviewers

www.PacktPub.com

www.PacktPub.com

Preface

Preface

Free Chapter

Setting the Context for Design Patterns in Pig

Setting the Context for Design Patterns in Pig

Understanding design patterns

The scope of design patterns in Pig

Hadoop demystified – a quick reckoner

Pig – a quick intro

Understanding Pig through the code

Summary

Data Ingest and Egress Patterns

Data Ingest and Egress Patterns

The context of data ingest and egress

Types of data in the enterprise

Ingest and egress patterns for multistructured data

The ingress and egress patterns for the NoSQL data

The ingress and egress patterns for structured data

The ingress and egress patterns for semi-structured data

JSON ingress and egress patterns

Summary

Data Profiling Patterns

Data Profiling Patterns

Data profiling for Big Data

Rationale for using Pig in data profiling

The data type inference pattern

The basic statistical profiling pattern

The pattern-matching pattern

The string profiling pattern

The unstructured text profiling pattern

Summary

Data Validation and Cleansing Patterns

Data Validation and Cleansing Patterns

Data validation and cleansing for Big Data

Choosing Pig for validation and cleansing

The constraint validation and cleansing design pattern

The regex validation and cleansing design pattern

The corrupt data validation and cleansing design pattern

The unstructured text data validation and cleansing design pattern

Summary

Data Transformation Patterns

Data Transformation Patterns

Data transformation processes

The structured-to-hierarchical transformation pattern

The data normalization pattern

The data integration pattern

The aggregation pattern

The data generalization pattern

Summary

Understanding Data Reduction Patterns

Understanding Data Reduction Patterns

Data reduction – a quick introduction

Data reduction considerations for Big Data

Dimensionality reduction – the Principal Component Analysis design pattern

Numerosity reduction – the histogram design pattern

Numerosity reduction – sampling design pattern

Numerosity reduction – clustering design pattern

Summary

Advanced Patterns and Future Work

Advanced Patterns and Future Work

The clustering pattern

The topic discovery pattern

The natural language processing pattern

The classification pattern

Future trends

Summary

Index

Index

Chapter 5. Data Transformation Patterns

In the last chapter, you learned about various patterns related to data validation and cleansing from which you understood that there are ways to detect and remove incorrect or inaccurate records from the data. By the time data validation and cleansing is complete, the inconsistencies in the data are identified even before the data is used in the next steps of the analytics life cycle; then, the inconsistent data is replaced, modified, or deleted to make it more consistent.

In this chapter, you will learn about various design patterns related to data transformation, such as structured to hierarchical, normalization, integration, aggregation, and generalization design patterns.

CONTINUE READING

83

Tech Concepts

36

Programming languages

73

Tech Tools

Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.

Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.

50+ new titles added per month and exclusive early access to books as they are being written.

Pig Design Patterns

Search

Your notes and bookmarks