Book Image

Architecting Data-Intensive Applications

By : Anuj Kumar
Book Image

Architecting Data-Intensive Applications

By: Anuj Kumar

Overview of this book

<p>Are you an architect or a developer who looks at your own applications gingerly while browsing through Facebook and applauding it silently for its data-intensive, yet ?uent and efficient, behaviour? This book is your gateway to build smart data-intensive systems by incorporating the core data-intensive architectural principles, patterns, and techniques directly into your application architecture.</p> <p>This book starts by taking you through the primary design challenges involved with architecting data-intensive applications. You will learn how to implement data curation and data dissemination, depending on the volume of your data. You will then implement your application architecture one step at a time. You will get to grips with implementing the correct message delivery protocols and creating a data layer that doesn’t fail when running high traffic. This book will show you how you can divide your application into layers, each of which adheres to the single responsibility principle. By the end of this book, you will learn to streamline your thoughts and make the right choice in terms of technologies and architectural principles based on the problem at hand.</p>
Table of Contents (18 chapters)
Title Page
Packt Upsell
Contributors
Preface
Index

The scheduler/executor component of the streaming architecture


Whenever we want to execute a highly-distributed set of interconnected tasks whose life cycle needs to be managed, we run into a set of complexity. If we look at the nonfunctional requirements of these tasks, they boil down to the following:

  • The tasks should be executed in the order that they are defined
  • If a task fails, it should be automatically restarted to bring fault-tolerance into the overall system
  • If a machine fails, all the tasks running on that system should be restarted on a different machine
  • If there are fewer resources and more tasks to execute, higher-priority tasks should get executed first and lower-priority tasks should be queued, and resources divided in some defined order between tasks

These are just some of the requirements, but I hope you can see the problem here. It is not just executing a set of tasks, but it is about handling situations related to resource-management. Resources here could be CPU, memory, time...