Book Image

Scaling Big Data with Hadoop and Solr

By : Hrishikesh Vijay Karambelkar
Book Image

Scaling Big Data with Hadoop and Solr

By: Hrishikesh Vijay Karambelkar

Overview of this book

<p>As data grows exponentially day-by-day, extracting information becomes a tedious activity in itself. Technologies like Hadoop are trying to address some of the concerns, while Solr provides high-speed faceted search. Bringing these two technologies together is helping organizations resolve the problem of information extraction from Big Data by providing excellent distributed faceted search capabilities.</p> <p>Scaling Big Data with Hadoop and Solr is a step-by-step guide that helps you build high performance enterprise search engines while scaling data. Starting with the basics of Apache Hadoop and Solr, this book then dives into advanced topics of optimizing search with some interesting real-world use cases and sample Java code.</p> <p>Scaling Big Data with Hadoop and Solr starts by teaching you the basics of Big Data technologies including Hadoop and its ecosystem and Apache Solr. It explains the different approaches of scaling Big Data with Hadoop and Solr, with discussion regarding the applicability, benefits, and drawbacks of each approach. It then walks readers through how sharding and indexing can be performed on Big Data followed by the performance optimization of Big Data search. Finally, it covers some real-world use cases for Big Data scaling.</p> <p>With this book, you will learn everything you need to know to build a distributed enterprise search platform as well as how to optimize this search to a greater extent resulting in maximum utilization of available resources.</p>
Table of Contents (15 chapters)
Scaling Big Data with Hadoop and Solr
Credits
About the Author
About the Reviewer
www.PacktPub.com
Preface
Index

Appendix C. Sample MapReduce Programs to Build the Solr Indexes

In this appendix, we are going to look at sample MapReduce programs to build Solr indexes. We will start with an example of a MapReduce program.

Let's say we have three files containing the following text, and we have to get a word count of each word:

  • [I enjoy walking on the beach sand. The Maya beach is what I enjoy most.]

  • [John loves to play volleyball on the beach.]

  • [We enjoy watching television.]

The results are then split into blocks and replicated on multiple data nodes. The map function then extracts a count of words from each file. The following <key, value> pairs are outcomes of the map function of Hadoop:

  • <I ,2> <enjoy, 2> <walking,1> <on,1> <the,2> <beach,2> <sand,1> <maya,1> <is,1> <what,1> <most,1>

  • <John,1> <loves,1> <to,1> <play,1> <volleyball,1> <on,1> <the,1> <beach,1>

  • <we,1> <enjoy,1> <watching,1> <television,1>

Now, reduce task merges all these together and reduces the input to a single set of <key, value> pairs, getting us the count of words:

<I ,2> <enjoy, 3> <walking,1> <on,2> <the,3> <beach,3> <sand,1> <maya,1> <is,1> <what,1> <most,1> <John,1> <loves,1> <to,1> <play,1> <volleyball,1> <we,1> <watching,1> <television,1>

Now, we will look at some samples for different implementations.