Lambda architecture
We are talking about Data-Intensive Systems. But what is the main purpose of developing a Data-Intensive System? What does a Data-Intensive System really do? You may say it processes the data at scale, but still that's just a step in the actual question that it is trying to answer. A Data-intensive system tries to answer questions based on the data that is either presented to it in real-time, on the data that have been collected in the past, or a combination of both. BigData, a book by NathanMarz, defines Data Systems as systems that compute queries that are functions of all the data we've ever seen:
Query = Function(all data)
This definition of a query is at the center of Lambda architecture.
The first thing that should come to your mind is depending on whether Query
is always the function on all data, and whether the data is in terabytes or petabytes, how do our queries perform? What are the latencies and requirements around it? Another important thing to consider is the...