After covering the various use cases covered in this book, we will discuss the architecture that forms the basis of the solutions that we will create in the following chapters. Lambda architecture is a good architecture pattern that we can refer to while designing a big data system. Lambda architecture is a generic architecture in which Hadoop and its ecosystem of tools fit nicely.
Nathan Marz is an ex-Twitter engineer who gained significant experience in building real-time distributed data-processing systems. He went on to design Lambda architecture, which is a generic architecture for big data systems. This architecture addresses the requirements of batch and real-time analytics.
The underlying thoughts behind building a generic architecture for big data systems are as follows:
The system should be fault tolerant and capable of coping with hardware failures.
The system should be able to run a variety of workloads. In some workloads, low latency is required such as interactive queries.
It should support linear scalability. This means that adding more servers should scale up the processing capacity of the system linearly.
From a high-level perspective, this system has the following building blocks:
A data ingestion block, which dispatches data to the batch and the speed layer for processing.
The batch layer, which is an immutable, append-only data store. This computes the batch views.
The serving layer, which indexes batch views for consumption by interactive queries and applications.
The speed layer, which processes the data in real time and compensates for the delay introduced in the batch layer.
Any interactive query can be answered by merging the results from the serving layer and batch layer.
As a reader, you will quickly recognize how Hadoop and the HDFS fit in the batch layer. Technologies such as MapReduce and Pig are suited for the creation of batch views. In the serving layer, we can use tools such as Hbase or MongoDB, which support both interactive queries and full CRUD operations on the datasets. In the speed layer, we can use tools such as Apache Storm and Apache Spark Streaming.
Lambda architecture is a reference model. In our use cases, we should make good choices about which components of this reference model are used in our solution architecture. For example, a use case that does not require a near real-time response or interactive queries will not need components to build the speed layer and serving layer.