Browse Library
Sign In Start Free Trial
Close icon Search icon
Account
Browse Library Sign In Start Free Trial

Add to playlist

Create a Playlist

Modal Close icon
You need to login to use this feature.
  • Book Overview & Buying Programming MapReduce with Scalding
  • Table Of Contents Toc
Programming MapReduce with Scalding

Programming MapReduce with Scalding

By : Antonios Chalkiopoulos
4.3 (6)
Buy this Book
close
close
Programming MapReduce with Scalding

Programming MapReduce with Scalding

4.3 (6)
By: Antonios Chalkiopoulos
Buy this Book

Overview of this book

This book is an easy-to-understand, practical guide to designing, testing, and implementing complex MapReduce applications in Scala using the Scalding framework. It is packed with examples featuring log-processing, ad-targeting, and machine learning. This book is for developers who are willing to discover how to effectively develop MapReduce applications. Prior knowledge of Hadoop or Scala is not required; however, investing some time on those topics would certainly be beneficial.
Table of Contents (11 chapters)
close
close
close
Preface
chevron up
Icon
Preface
Icon
What this book covers
Icon
What you need for this book
Icon
Who this book is for
Icon
Conventions
Icon
Reader feedback
Icon
Customer support
Lock Free Chapter
1
1. Introduction to MapReduce
Icon
1. Introduction to MapReduce
Icon
The Hadoop platform
Icon
MapReduce
Icon
MapReduce abstractions
Icon
Introducing Cascading
Icon
Summary
2
2. Get Ready for Scalding
Icon
2. Get Ready for Scalding
Icon
Why Scala?
Icon
Scala basics
Icon
Scala build tools
Icon
Hello World in Scala
Icon
Development editors
Icon
Installing Hadoop in five minutes
Icon
Running our first Scalding job
Icon
Submitting a Scalding job in Hadoop
Icon
Summary
3
3. Scalding by Example
Icon
3. Scalding by Example
Icon
Reading and writing files
Icon
Understanding the core capabilities of Scalding
Icon
Operations on groups
Icon
A simple example
Icon
Typed API
Icon
Summary
4
4. Intermediate Examples
Icon
4. Intermediate Examples
Icon
Logfile analysis
Icon
Exploring ad targeting
Icon
Summary
5
5. Scalding Design Patterns
Icon
5. Scalding Design Patterns
Icon
The external operations pattern
Icon
The dependency injection pattern
Icon
The late bound dependency pattern
Icon
Summary
6
6. Testing and TDD
Icon
6. Testing and TDD
Icon
Introduction to testing
Icon
MapReduce testing challenges
Icon
Development lifecycle with testing strategy
Icon
TDD for Scalding developers
Icon
Black box testing
Icon
Summary
7
7. Running Scalding in Production
Icon
7. Running Scalding in Production
Icon
Executing Scalding in a Hadoop cluster
Icon
Scheduling execution
Icon
Coordinating job execution
Icon
Configuring using a property file
Icon
Configuring using Hadoop parameters
Icon
Monitoring Scalding jobs
Icon
Using slim JAR files
Icon
Scalding execution throttling
Icon
Summary
8
8. Using External Data Stores
Icon
8. Using External Data Stores
Icon
Interacting with external systems
Icon
SQL databases
Icon
NoSQL databases
Icon
Search platforms
Icon
Summary
9
9. Matrix Calculations and Machine Learning
Icon
9. Matrix Calculations and Machine Learning
Icon
Text similarity using TF-IDF
Icon
Setting a similarity using the Jaccard index
Icon
K-Means using Mahout
Icon
Other libraries
Icon
Summary
10
Index
Icon
Index

Conventions

In this book, you will find a number of styles of text that distinguish between different kinds of information. Here are some examples of these styles, and an explanation of their meaning.

Code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, and user input are shown as follows: "A Map class to map lines into <key,value> pairs; for example, <"INFO",1>."

A block of code is set as follows:

LogLine    = load 'file.logs' as (level, message);
LevelGroup = group LogLine by level;
Result     = foreach LevelGroup generate group, COUNT(LogLine);
store Result into 'Results.txt';

When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:

import com.twitter.scalding._
 
class CalculateDailyAdPoints (args: Args) extends Job(args) {

  val logSchema = List ('datetime, 'user, 'activity, 'data,
   'session, 'location, 'response, 'device, 'error, 'server)

  val logs = Tsv("/log-files/2014/07/01", logSchema )
   .read
   .project('user,'datetime,'activity,'data)
   .groupBy('user) { group => group.sortBy('datetime) }
   .write(Tsv("/analysis/log-files-2014-07-01"))
}

Any command-line input or output is written as follows:

$ echo "This is a happy day. A day to remember" > input.txt
$ hadoop fs -mkdir -p hdfs:///data/input hdfs:///data/output
$ hadoop fs -put input.txt hdfs:///data/input/

New terms and important words are shown in bold.

Note

Warnings or important notes appear in a box like this.

Tip

Tips and tricks appear like this.

CONTINUE READING
83
Tech Concepts
36
Programming languages
73
Tech Tools
Icon Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.
Icon Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.
Icon 50+ new titles added per month and exclusive early access to books as they are being written.
Programming MapReduce with Scalding
Next Section
notes
bookmark Notes and Bookmarks search Search in title playlist Add to playlist font-size Font size

Change the font size

margin-width Margin width

Change margin width

day-mode Day/Sepia/Night Modes

Change background colour

Close icon Search
Country selected
Close icon Your notes and bookmarks

Confirmation

Modal Close icon
claim successful
Order Page Read Now

Buy this book with your credits?

Modal Close icon
Are you sure you want to buy this book with one of your credits?
Close
YES, BUY

Submit Your Feedback

Modal Close icon
Modal Close icon
Modal Close icon