Book Image

Mastering Hadoop

By : Sandeep Karanth
Book Image

Mastering Hadoop

By: Sandeep Karanth

Overview of this book

Table of Contents (21 chapters)
Mastering Hadoop
Credits
About the Author
Acknowledgments
About the Reviewers
www.PacktPub.com
Preface
Index

File formats


There are a number of file formats that are data structures by themselves. In the chapter on Hive, we saw ORC files, an optimized form of record columnar file storage. There are a few other popular file container formats supported by Hadoop. We will look at them in this section.

The Sequence file format

A Sequence file is a container format for binary key-value pairs. Each record in a Sequence file contains a key and its associated value. Sequence files are used to combine smaller files into a single large file to alleviate the small file problem in Hadoop. In this situation, the filename forms the key and the file contents are the values associated with the key. Sequence files have a broader applicability as they can be split into configurable blocks. They can be combined with fast compression methods such as LZO or Snappy and can provide speed as well as storage and bandwidth.

The next image shows the format of a Sequence file. The start of the file has a magic number, a binary...