Book Image

Architecting Data-Intensive Applications

By : Anuj Kumar
Book Image

Architecting Data-Intensive Applications

By: Anuj Kumar

Overview of this book

<p>Are you an architect or a developer who looks at your own applications gingerly while browsing through Facebook and applauding it silently for its data-intensive, yet ?uent and efficient, behaviour? This book is your gateway to build smart data-intensive systems by incorporating the core data-intensive architectural principles, patterns, and techniques directly into your application architecture.</p> <p>This book starts by taking you through the primary design challenges involved with architecting data-intensive applications. You will learn how to implement data curation and data dissemination, depending on the volume of your data. You will then implement your application architecture one step at a time. You will get to grips with implementing the correct message delivery protocols and creating a data layer that doesn’t fail when running high traffic. This book will show you how you can divide your application into layers, each of which adheres to the single responsibility principle. By the end of this book, you will learn to streamline your thoughts and make the right choice in terms of technologies and architectural principles based on the problem at hand.</p>
Table of Contents (18 chapters)
Title Page
Packt Upsell
Contributors
Preface
Index

Chapter 6. Creating a Data Pipeline for Consistent Data Collection, Processing, and Dissemination

In a data intensive application, data travels in two directions in two different forms. One form of the data is data that is returned to the end users as part of a request. The process of gathering the data is usually synchronous and in a distributed system, which typically comes from a variety of data sources. Imagine we are building a context service where we want to know everything about a given IP address that tries to access our secured network. The use case would be that we want to block all IP addresses that we know are potentially from known malicious users. Typically, to keep the example discussion simple, what we would do is the following:

  • Resolve the IP to the domain list pertaining to the IP address
  • Check the IP address against the list of IP addresses that are marked as malicious
  • Check the domain(s) against the set of domains that are marked as malicious
  • Return the answer as good or...