Big Data is the need the day. Many organizations are producing huge amounts of data every day. With the advancement of Hadoop-like tools, it has become easier for everyone to solve Big Data problems with great efficiency and at a very low cost. When you are handling such a massive amount of data, even a small mistake can cost you dearly in terms of performance and storage. It's very important to learn the best practices of handling such tools before you start building an enterprise Big Data Warehouse, which will be greatly advantageous in making your project successful.
This book gives you insights into learning and mastering Big Data recipes. This book not only explores a majority of Big Data tools that are currently being used in the market, but also provides the best practices in order to implement them. This book will also provide you with recipes that are based on the latest version of Apache Hadoop 2.X, YARN, Hive, Pig, Sqoop, Flume, Apache Spark, Mahout, and many more ecosystem tools. This real-world solutions cookbook is packed with handy recipes that you can apply to your own everyday issues. Each chapter talks about recipes in great detail, and these can be referred to easily. This book provides detailed practice on the latest technologies, such as YARN and Apache Spark. This guide is an invaluable tutorial if you are planning to implement Big Data Warehouse for your business.
Chapter 1, Getting Started with Hadoop 2.x, introduces you to the installation details needed for single and multi-node Hadoop clusters. It also contains the recipes that will help you understand various important cluster management techniques, such as decommissioning, benchmarking, and so on.
Chapter 2, Exploring HDFS, provides you with hands-on recipes to manage and maintain the Hadoop Distributed File System (HDFS) in an efficient way. You will learn some important practices, such as transient encryption, saving data in a compressed format, recycling deleted data from HDFS, and so on.
Chapter 3, Mastering Map Reduce Programs, enlightens you about very important recipes for Map Reduce programming, which take you beyond the simple Word Count program. You will learn about various customization techniques in detail.
Chapter 4, Data Analysis Using Hive, Pig, and Hbase, takes you to the analytical world of Hive, Pig, and Hbase. This chapter talks about the use of various file formats, such as RC, ORC, Parquet, and so on. You will also get introduced to the Hbase NoSQL database.
Chapter 5, Advanced Data Analysis Using Hive, provides insights on the usage of serializers and deserializers (SerDe) in Hive for JSON and XML data operations. This chapter will provide you with a detailed explanation for Twitter sentiment analysis using Hive.
Chapter 6, Data Import/Export Using Sqoop and Flume, covers various recipes to import and export data from sources, such as RDBMS, Kafka, web log servers, and so on, using Sqoop and Flume.
Chapter 7, Automation of Hadoop Tasks Using Oozie, introduces you to a very rich scheduling tool called Oozie, which will help you build automated production-ready Big Data applications.
Chapter 8, Machine Learning and Predictive Analytics Using Mahout and R, gives you an end-to-end implementation of predictive analytics applications using Mahout and R. It covers the various visualization options available in R as well.
Chapter 9, Integration with Apache Spark, introduces you to a very important distributed computing framework called Apache Spark. It covers basic to advanced topics such as installation, Spark application development and execution, usage of the Spark Machine Learning Library, MLib, and graph processing using Spark.
Chapter 10, Hadoop Use Cases, provides you with end-to-end implementations of Hadoop use cases from various domains, such as telecom, finance, e-commerce, and so on.
To get started with this hands-on recipe-driven book, you should have a laptop/desktop with any OS, such as Windows, Linux, or Mac. It's good to have an IDE, such as Eclipse or IntelliJ, and of course, you need a lot of enthusiasm to learn.
This book is for those of you who have basic knowledge of Big Data systems and want to advance your knowledge with hands-on recipes.
In this book, you will find a number of text styles that distinguish between different kinds of information. Here are some examples of these styles and an explanation of their meaning.
Code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles are shown as follows: " Spark MLib provides a huge list of supported algorithms."
A block of code is set as follows:
$jps 2184 DataNode 2765 NodeManager 2835 Jps 2403 SecondaryNameNode 2025 NameNode 2606 ResourceManager
When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:
// The results of SQL queries are themselves RDDs and support all normal RDD functions. The
// items in the RDD are of type Row, which allows you to access each column by ordinal.
valrddFromSql = sql("SELECT id, name FROM empSpark WHERE id < 20 ORDER BY id")
Any command-line input or output is written as follows:
# cp /usr/src/asterisk-addons/configs/cdr_mysql.conf.sample /etc/asterisk/cdr_mysql.conf
New terms and important words are shown in bold. Words that you see on the screen, for example, in menus or dialog boxes, appear in the text like this: "Click on Create your Twitter application to save your application."
Feedback from our readers is always welcome. Let us know what you think about this book—what you liked or disliked. Reader feedback is important for us as it helps us develop titles that you will really get the most out of.
To send us general feedback, simply e-mail <[email protected]>, and mention the book's title in the subject of your message.
If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide at www.packtpub.com/authors.
Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase.
You can download the example code files for this book from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.
You can download the code files by following these steps:
Log in or register to our website using your e-mail address and password.
Hover the mouse pointer on the SUPPORT tab at the top.
Click on Code Downloads & Errata.
Enter the name of the book in the Search box.
Select the book for which you're looking to download the code files.
Choose from the drop-down menu where you purchased this book from.
Click on Code Download.
Once the file is downloaded, please make sure that you unzip or extract the folder using the latest version of:
WinRAR / 7-Zip for Windows
Zipeg / iZip / UnRarX for Mac
7-Zip / PeaZip for Linux
We also provide you with a PDF file that has color images of the screenshots/diagrams used in this book. The color images will help you better understand the changes in the output. You can download this file from https://www.packtpub.com/sites/default/files/downloads/HadoopRealWorldSolutionsCookbookSecondEdition_ColoredImages.pdf.
Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you find a mistake in one of our books—maybe a mistake in the text or the code—we would be grateful if you could report this to us. By doing so, you can save other readers from frustration and help us improve subsequent versions of this book. If you find any errata, please report them by visiting http://www.packtpub.com/submit-errata, selecting your book, clicking on the Errata Submission Form link, and entering the details of your errata. Once your errata are verified, your submission will be accepted and the errata will be uploaded to our website or added to any list of existing errata under the Errata section of that title.
To view the previously submitted errata, go to https://www.packtpub.com/books/content/support and enter the name of the book in the search field. The required information will appear under the Errata section.
Piracy of copyrighted material on the Internet is an ongoing problem across all media. At Packt, we take the protection of our copyright and licenses very seriously. If you come across any illegal copies of our works in any form on the Internet, please provide us with the location address or website name immediately so that we can pursue a remedy.
Please contact us at <[email protected]> with a link to the suspected pirated material.
We appreciate your help in protecting our authors and our ability to bring you valuable content.
If you have a problem with any aspect of this book, you can contact us at <[email protected]>, and we will do our best to address the problem.