Book Image

Architecting Data-Intensive Applications

By : Anuj Kumar
Book Image

Architecting Data-Intensive Applications

By: Anuj Kumar

Overview of this book

<p>Are you an architect or a developer who looks at your own applications gingerly while browsing through Facebook and applauding it silently for its data-intensive, yet ?uent and efficient, behaviour? This book is your gateway to build smart data-intensive systems by incorporating the core data-intensive architectural principles, patterns, and techniques directly into your application architecture.</p> <p>This book starts by taking you through the primary design challenges involved with architecting data-intensive applications. You will learn how to implement data curation and data dissemination, depending on the volume of your data. You will then implement your application architecture one step at a time. You will get to grips with implementing the correct message delivery protocols and creating a data layer that doesn’t fail when running high traffic. This book will show you how you can divide your application into layers, each of which adheres to the single responsibility principle. By the end of this book, you will learn to streamline your thoughts and make the right choice in terms of technologies and architectural principles based on the problem at hand.</p>
Table of Contents (18 chapters)
Title Page
Packt Upsell
Contributors
Preface
Index

Processing strategy


The key foundation of any batch system is its processing strategy. It is very important for anyone to understand what they want from their batch system and then adopt a strategy to achieve their goal. You should understand that NOT every batch system needs to be complex or distributed in nature. Not every meta store needs to be a Hadoop. If your underlying store is a relational database and you have needs to perform batch operations, you may consider using spring batch as the technology of choice rather than spark, for example.

There are multiple factors that affect the selection of a processing strategy. They include (but are not limited to):

  • Estimated batch data volume
  • Concurrent execution alongside other batch or online/realtime applications
  • Available batch windows (and with more enterprises wanting to be up and running 24/7, this may not really be an option)

For a batch process, the following may be some of your options:

  • Perform processing within a batch window defined...