Book Image

Mastering Hadoop 3

By : Chanchal Singh, Manish Kumar
Book Image

Mastering Hadoop 3

By: Chanchal Singh, Manish Kumar

Overview of this book

Apache Hadoop is one of the most popular big data solutions for distributed storage and for processing large chunks of data. With Hadoop 3, Apache promises to provide a high-performance, more fault-tolerant, and highly efficient big data processing platform, with a focus on improved scalability and increased efficiency. With this guide, you’ll understand advanced concepts of the Hadoop ecosystem tool. You’ll learn how Hadoop works internally, study advanced concepts of different ecosystem tools, discover solutions to real-world use cases, and understand how to secure your cluster. It will then walk you through HDFS, YARN, MapReduce, and Hadoop 3 concepts. You’ll be able to address common challenges like using Kafka efficiently, designing low latency, reliable message delivery Kafka systems, and handling high data volumes. As you advance, you’ll discover how to address major challenges when building an enterprise-grade messaging system, and how to use different stream processing systems along with Kafka to fulfil your enterprise goals. By the end of this book, you’ll have a complete understanding of how components in the Hadoop ecosystem are effectively integrated to implement a fast and reliable data pipeline, and you’ll be equipped to tackle a range of real-world problems in data pipelines.
Table of Contents (23 chapters)
Title Page
Dedication
About Packt
Foreword
Contributors
Preface
Index

Foreword

In today's fast-moving world, many people have moved their learning from reading a book to looking at a computer screen (maybe even a mobile phone screen). As a college professor, I can see there are pros and cons to this way of learning. Learners learning in this way typically look for different topics and sub topics by searching the web for pages that provide snippets of information for video clips. This type of learning lacks the proper steps and tends to be incomplete because learners often skip the fundamentals, go straight to the core based on the search keywords they chose, and therefore do not get the whole picture.

During the course of writing this book, Chanchal Singh and myself taught a big data course together in college. We wanted to learn for ourselves the best way to pass on knowledge to a group of eager learners. During this process, we gained valuable insight into how we should structure the book to provide step-by-step, uncluttered learning. Therefore, we are confident that this book is suitable for use as a college textbook on Hadoop. It can also be used as a reference text for experienced big data developers.

Mastering Hadoop 3 starts with what's new in Hadoop, and then it dives into the Hadoop Distributed File System, the core of big data in Hadoop. The book moves on to YARN, MapReduce, and SQL in Hadoop. With that foundation laid, it covers real-time processing, widely used Hadoop ecosystem components, and how to design applications. Machine learning is one of today's hot topics, and this book illustrates how it can work in a Hadoop environment. The book wraps up with security for Hadoop systems and how to monitor it.

In my long career in the IT industry, constantly keeping myself up to date has become a habit. I still read a lot of books because books structure learning for me. They also ensure I get the whole picture of the subject. Being able to recall knowledge is a demonstration of successful learning. To achieve that, I often re-read the same book several times to make sure I can recall the knowledge gained from it effectively and apply it in real life. I am confident Mastering Hadoop 3 is well-structured when it comes to presenting the subject.

 

 

To close, I recall that I built my first database for a customer back in 1982, 37 years ago. There was no big data because storage was expensive, CPU power was limited, and data could not move quickly because networking at the time could not move that fast. All these factors are no longer valid today. That's why anyone can build a database using a big data architecture such as Hadoop. Not only is the required technology now very accessible, the ecosystem around it is also mature. I encourage the new generation of data and application architects to consider leveraging big data in every project they conceive. This book, Mastering Hadoop 3, will be a good reference to have.

 

Timothy Wong

PhD, Founder and President, Codehesive Solutions

Professor, Humber College