Browse Library
Sign In Start Free Trial
Close icon Search icon
Account
Browse Library Sign In Start Free Trial

Add to playlist

Create a Playlist

Modal Close icon
You need to login to use this feature.
  • Book Overview & Buying Programming MapReduce with Scalding
  • Table Of Contents Toc
Programming MapReduce with Scalding

Programming MapReduce with Scalding

By : Antonios Chalkiopoulos
4.3 (6)
Buy this Book
close
close
Programming MapReduce with Scalding

Programming MapReduce with Scalding

4.3 (6)
By: Antonios Chalkiopoulos
Buy this Book

Overview of this book

This book is an easy-to-understand, practical guide to designing, testing, and implementing complex MapReduce applications in Scala using the Scalding framework. It is packed with examples featuring log-processing, ad-targeting, and machine learning. This book is for developers who are willing to discover how to effectively develop MapReduce applications. Prior knowledge of Hadoop or Scala is not required; however, investing some time on those topics would certainly be beneficial.
Table of Contents (11 chapters)
close
close
close
Preface
Icon
Preface
Icon
What this book covers
Icon
What you need for this book
Icon
Who this book is for
Icon
Conventions
Icon
Reader feedback
Icon
Customer support
Lock Free Chapter
1
1. Introduction to MapReduce
Icon
1. Introduction to MapReduce
Icon
The Hadoop platform
Icon
MapReduce
Icon
MapReduce abstractions
Icon
Introducing Cascading
Icon
Summary
2
2. Get Ready for Scalding
Icon
2. Get Ready for Scalding
Icon
Why Scala?
Icon
Scala basics
Icon
Scala build tools
Icon
Hello World in Scala
Icon
Development editors
Icon
Installing Hadoop in five minutes
Icon
Running our first Scalding job
Icon
Submitting a Scalding job in Hadoop
Icon
Summary
3
3. Scalding by Example
Icon
3. Scalding by Example
Icon
Reading and writing files
Icon
Understanding the core capabilities of Scalding
Icon
Operations on groups
Icon
A simple example
Icon
Typed API
Icon
Summary
4
4. Intermediate Examples
Icon
4. Intermediate Examples
Icon
Logfile analysis
Icon
Exploring ad targeting
Icon
Summary
5
5. Scalding Design Patterns
chevron up
Icon
5. Scalding Design Patterns
Icon
The external operations pattern
Icon
The dependency injection pattern
Icon
The late bound dependency pattern
Icon
Summary
6
6. Testing and TDD
Icon
6. Testing and TDD
Icon
Introduction to testing
Icon
MapReduce testing challenges
Icon
Development lifecycle with testing strategy
Icon
TDD for Scalding developers
Icon
Black box testing
Icon
Summary
7
7. Running Scalding in Production
Icon
7. Running Scalding in Production
Icon
Executing Scalding in a Hadoop cluster
Icon
Scheduling execution
Icon
Coordinating job execution
Icon
Configuring using a property file
Icon
Configuring using Hadoop parameters
Icon
Monitoring Scalding jobs
Icon
Using slim JAR files
Icon
Scalding execution throttling
Icon
Summary
8
8. Using External Data Stores
Icon
8. Using External Data Stores
Icon
Interacting with external systems
Icon
SQL databases
Icon
NoSQL databases
Icon
Search platforms
Icon
Summary
9
9. Matrix Calculations and Machine Learning
Icon
9. Matrix Calculations and Machine Learning
Icon
Text similarity using TF-IDF
Icon
Setting a similarity using the Jaccard index
Icon
K-Means using Mahout
Icon
Other libraries
Icon
Summary
10
Index
Icon
Index

The external operations pattern

To achieve modularity and fulfill the single responsibility principle, we can structure our data processing job in an organized way. An object, a trait, and a job can share parts of the responsibilities, explained as follows:

  • In package object, we can store information about the schema of the data

  • In trait, we can store all the external operations

  • In a Scalding job, we can manage arguments, define taps, and use external operations to construct data processing pipelines

A particular dataset will usually be processed by multiple jobs to extract different value from the data. Thus, we can create an object called LogsSchemas to store input and output schemas, and also to document the locations in HDFS, where the data resides. This object can act as a registry of all the variations of datasets, and we can reuse it in any of our Scalding jobs, as shown in the following code:

package object LogsSchemas {
  // that is, hdfs:///logs/raw/YYYY/MM/DD/
  val LOG_SCHEMA = List...
Visually different images
CONTINUE READING
83
Tech Concepts
36
Programming languages
73
Tech Tools
Icon Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.
Icon Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.
Icon 50+ new titles added per month and exclusive early access to books as they are being written.
Programming MapReduce with Scalding
End of Section 5
Next Section
notes
bookmark Notes and Bookmarks search Search in title playlist Add to playlist font-size Font size

Change the font size

margin-width Margin width

Change margin width

day-mode Day/Sepia/Night Modes

Change background colour

Close icon Search
Country selected
Close icon Your notes and bookmarks

Confirmation

Modal Close icon
claim successful
Order Page Read Now

Buy this book with your credits?

Modal Close icon
Are you sure you want to buy this book with one of your credits?
Close
YES, BUY

Submit Your Feedback

Modal Close icon
Modal Close icon
Modal Close icon