Enhancing Map and Reduce Tasks | Optimizing Hadoop for MapReduce

Chapter 5. Enhancing Map and Reduce Tasks

The Hadoop framework already includes several counters such as the number of bytes read and written. These counters are very helpful to learn about the framework activities and the resources used. These counters are sent by the worker nodes to the master nodes periodically.

In this chapter, for both map and reduce, we will learn how to enhance each phase, what counters to look at, and the techniques to apply in order to analyze a performance issue. Then, you will learn how to tune the correct configuration parameter with the appropriate value.

In this chapter, we will cover the following topics:

The impact of the block size and input data
How to deal with small and unsplittable files
Reducing map-side spilling records
Improving the Reduce phase
Calculating Map and Reduce tasks' throughput
Tuning map and reduce parameters

Tech Concepts

Programming languages

Tech Tools

Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.

Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.

50+ new titles added per month and exclusive early access to books as they are being written.

Optimizing Hadoop for MapReduce

By : Khaled Tannir

Optimizing Hadoop for MapReduce

By: Khaled Tannir

Overview of this book

Chapter 5. Enhancing Map and Reduce Tasks

Optimizing Hadoop for MapReduce

By : Khaled Tannir

Optimizing Hadoop for MapReduce

By: Khaled Tannir

Overview of this book

Chapter 5. Enhancing Map and Reduce Tasks

Confirmation

Buy this book with your credits?

Submit Your Feedback

Create a Free Account To Continue Reading

Sign in to activate your 7-day free access