7. Running Scalding in Production | Programming MapReduce with Scalding

Sign In Start Free Trial

Book Overview & Buying
Table Of Contents

Programming MapReduce with Scalding

By : Antonios Chalkiopoulos

4.3 (6)

Programming MapReduce with Scalding

4.3 (6)

By: Antonios Chalkiopoulos

Overview of this book

This book is an easy-to-understand, practical guide to designing, testing, and implementing complex MapReduce applications in Scala using the Scalding framework. It is packed with examples featuring log-processing, ad-targeting, and machine learning. This book is for developers who are willing to discover how to effectively develop MapReduce applications. Prior knowledge of Hadoop or Scala is not required; however, investing some time on those topics would certainly be beneficial.

Preface

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Free Chapter

1. Introduction to MapReduce

1. Introduction to MapReduce

The Hadoop platform

MapReduce

MapReduce abstractions

Introducing Cascading

Summary

2. Get Ready for Scalding

2. Get Ready for Scalding

Why Scala?

Scala basics

Scala build tools

Hello World in Scala

Development editors

Installing Hadoop in five minutes

Running our first Scalding job

Submitting a Scalding job in Hadoop

Summary

3. Scalding by Example

3. Scalding by Example

Reading and writing files

Understanding the core capabilities of Scalding

Operations on groups

A simple example

Typed API

Summary

4. Intermediate Examples

4. Intermediate Examples

Logfile analysis

Exploring ad targeting

Summary

5. Scalding Design Patterns

5. Scalding Design Patterns

The external operations pattern

The dependency injection pattern

The late bound dependency pattern

Summary

6. Testing and TDD

6. Testing and TDD

Introduction to testing

MapReduce testing challenges

Development lifecycle with testing strategy

TDD for Scalding developers

Black box testing

Summary

7. Running Scalding in Production

7. Running Scalding in Production

Executing Scalding in a Hadoop cluster

Scheduling execution

Coordinating job execution

Configuring using a property file

Configuring using Hadoop parameters

Monitoring Scalding jobs

Using slim JAR files

Scalding execution throttling

Summary

8. Using External Data Stores

8. Using External Data Stores

Interacting with external systems

SQL databases

NoSQL databases

Search platforms

Summary

9. Matrix Calculations and Machine Learning

9. Matrix Calculations and Machine Learning

Text similarity using TF-IDF

Setting a similarity using the Jaccard index

K-Means using Mahout

Other libraries

Summary

Index

Index

Summary

In this chapter, executing and scheduling jobs were presented. As our data processing workflows can depend on a number of applications, we showed how to chain together and coordinate the execution of a number of tasks. Then, we proceeded with configuring our jobs using both property files and Hadoop parameters.

Monitoring and optimizing our job execution was also presented. Finally, two more techniques we presented about using slim jars were to optimize the deployment process and how to throttle job execution.

In the next chapter, we will see how to use external data sources to read data from and store data to.

CONTINUE READING

83

Tech Concepts

36

Programming languages

73

Tech Tools

Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.

Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.

50+ new titles added per month and exclusive early access to books as they are being written.

Programming MapReduce with Scalding

Search

Your notes and bookmarks