Book Image

Architecting Data-Intensive Applications

By : Anuj Kumar
Book Image

Architecting Data-Intensive Applications

By: Anuj Kumar

Overview of this book

<p>Are you an architect or a developer who looks at your own applications gingerly while browsing through Facebook and applauding it silently for its data-intensive, yet ?uent and efficient, behaviour? This book is your gateway to build smart data-intensive systems by incorporating the core data-intensive architectural principles, patterns, and techniques directly into your application architecture.</p> <p>This book starts by taking you through the primary design challenges involved with architecting data-intensive applications. You will learn how to implement data curation and data dissemination, depending on the volume of your data. You will then implement your application architecture one step at a time. You will get to grips with implementing the correct message delivery protocols and creating a data layer that doesn’t fail when running high traffic. This book will show you how you can divide your application into layers, each of which adheres to the single responsibility principle. By the end of this book, you will learn to streamline your thoughts and make the right choice in terms of technologies and architectural principles based on the problem at hand.</p>
Table of Contents (18 chapters)
Title Page
Packt Upsell
Contributors
Preface
Index

Windowing


Windowing is the concept that is used to answer questions such as, "What happened in the last x minutes?" In case of an unbounded stream, we can't really get the global view on the entire stream of data. We can only get the view for a specified time window. In stream processing systems, this time window is defined as the current time. As discussed, the time of an event can actually be divided into three aspects:

  • Event time: The time an event actually occurred
  • Ingestion time: The time at which the event was ingested into the processing system
  • Processing time: The set of times at which each of the processing jobs processed the event

Event time, while most useful, is also the most difficult to handle as it gives the least guarantees; events may arrive out of order or late, so we can never be sure whether we saw all the events in a given time window. Processing time is easiest, as it is monotonic: you know precisely when a five-minute window ended (by looking at the clock).

Types of windows...