In 2014, Jay Kreps from LinkedIn first described the concepts of Kappa architecture avoiding the maintenance of a separate code base for batch and real-time data processing. The primary objective is to manage interactive data processing and incremental events updates in a single data stream engine. Kappa Architecture consists of only the speed and serving layer without the batch processing step. The data from the ingestion layer directly move into interactive events processing jobs and the processed data moves into serving layers for near real-time visualization and querying purposes. This architecture follows an event reusable pattern as, for any updates into the stream processing engines, data has to be reprocessed and replied over the previously processed dataset.
The data ingestion layer can be consisted of Publish/Subscribe queue-based messaging systems, such as Apache Kafka, to parse, process, and execute complex events processing in interactive...