Compress outputs to save on storage. Compression also helps in increasing HDFS write throughput.
Avoid writing out-of-band side files as outputs in the Reduce task. If statistical data needs to be collected, the use of Counters is better. Collecting statistics in side files would require an additional step of aggregation.
Depending on the consumer of the output files of a job, a splittable compression technique could be appropriate.
Writing large HDFS files with larger block sizes can help subsequent consumers of the data reduce their Map tasks. This is particularly useful when we cascade MapReduce jobs. In such situations, the outputs of a job become the inputs to the next job. Writing large files with large block sizes eliminates the need for specialized processing of Map inputs in subsequent jobs.
Overview of this book
Table of Contents (21 chapters)
About the Author
About the Reviewers
Serialization and Hadoop I/O
YARN – Bringing Other Paradigms to Hadoop
Storm on YARN – Low Latency Processing in Hadoop
Hadoop on the Cloud
Analytics Using Hadoop
Hadoop for Microsoft Windows