Sign In Start Free Trial

Book Overview & Buying
Table Of Contents

Programming MapReduce with Scalding

By : Antonios Chalkiopoulos

4.3 (6)

Programming MapReduce with Scalding

4.3 (6)

By: Antonios Chalkiopoulos

Overview of this book

This book is an easy-to-understand, practical guide to designing, testing, and implementing complex MapReduce applications in Scala using the Scalding framework. It is packed with examples featuring log-processing, ad-targeting, and machine learning. This book is for developers who are willing to discover how to effectively develop MapReduce applications. Prior knowledge of Hadoop or Scala is not required; however, investing some time on those topics would certainly be beneficial.

Preface

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Free Chapter

1. Introduction to MapReduce

1. Introduction to MapReduce

The Hadoop platform

MapReduce

MapReduce abstractions

Introducing Cascading

Summary

2. Get Ready for Scalding

2. Get Ready for Scalding

Why Scala?

Scala basics

Scala build tools

Hello World in Scala

Development editors

Installing Hadoop in five minutes

Running our first Scalding job

Submitting a Scalding job in Hadoop

Summary

3. Scalding by Example

3. Scalding by Example

Reading and writing files

Understanding the core capabilities of Scalding

Operations on groups

A simple example

Typed API

Summary

4. Intermediate Examples

4. Intermediate Examples

Logfile analysis

Exploring ad targeting

Summary

5. Scalding Design Patterns

5. Scalding Design Patterns

The external operations pattern

The dependency injection pattern

The late bound dependency pattern

Summary

6. Testing and TDD

6. Testing and TDD

Introduction to testing

MapReduce testing challenges

Development lifecycle with testing strategy

TDD for Scalding developers

Black box testing

Summary

7. Running Scalding in Production

7. Running Scalding in Production

Executing Scalding in a Hadoop cluster

Scheduling execution

Coordinating job execution

Configuring using a property file

Configuring using Hadoop parameters

Monitoring Scalding jobs

Using slim JAR files

Scalding execution throttling

Summary

8. Using External Data Stores

8. Using External Data Stores

Interacting with external systems

SQL databases

NoSQL databases

Search platforms

Summary

9. Matrix Calculations and Machine Learning

9. Matrix Calculations and Machine Learning

Text similarity using TF-IDF

Setting a similarity using the Jaccard index

K-Means using Mahout

Other libraries

Summary

Index

Index

Chapter 5. Scalding Design Patterns

MapReduce applications, like all software artifacts, need to be written to be reusable, modular, and testable. They must have specific requirements related to the fact that they run in a Hadoop environment. The goal of this chapter is to present some design patterns to be applied in the implementation of Scalding applications. While the principles they follow are common to software development, we will present how to implement them in the specific domain.

The principles they follow are simplicity, single responsibility, and dependency inversion. In the context of Scalding, we will call them:

The external operations pattern
The dependency injection pattern
The late bound dependency pattern

Visually different images

CONTINUE READING

83

Tech Concepts

36

Programming languages

73

Tech Tools

Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.

Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.

50+ new titles added per month and exclusive early access to books as they are being written.

Programming MapReduce with Scalding

Search

Your notes and bookmarks