Processing concepts and tradeoffs
When developing a stream-based system, it is not only sufficient to understand the technologies that will enable the development of such a system, it is also important to understand how you will handle certain implicit and/or explicit requirements that focus mainly on the nonfunctional side of the spectrum. In this section, we will dive deeper into some such requirements.
Processing guarantees
When developing a stream processing application, you need to first understand how, and whether at all, you would be able to handle loss of data. Imagine a situation where a processing job has acknowledged the receipt of an event for processing, it has processed the event successfully, but before giving the message to the next-in-line processor, it fails. In Stream processing, this is the most often-asked scenario:
Does the underlying stream processing system guarantee that each event will be processed once and only once, even though there are failures at various stages...