Book Image

Architecting Data-Intensive Applications

By : Anuj Kumar
Book Image

Architecting Data-Intensive Applications

By: Anuj Kumar

Overview of this book

<p>Are you an architect or a developer who looks at your own applications gingerly while browsing through Facebook and applauding it silently for its data-intensive, yet ?uent and efficient, behaviour? This book is your gateway to build smart data-intensive systems by incorporating the core data-intensive architectural principles, patterns, and techniques directly into your application architecture.</p> <p>This book starts by taking you through the primary design challenges involved with architecting data-intensive applications. You will learn how to implement data curation and data dissemination, depending on the volume of your data. You will then implement your application architecture one step at a time. You will get to grips with implementing the correct message delivery protocols and creating a data layer that doesn’t fail when running high traffic. This book will show you how you can divide your application into layers, each of which adheres to the single responsibility principle. By the end of this book, you will learn to streamline your thoughts and make the right choice in terms of technologies and architectural principles based on the problem at hand.</p>
Table of Contents (18 chapters)
Title Page
Packt Upsell
Contributors
Preface
Index

Information exchange policy


The information exchange policy framework (https://www.first.org/iep) was put together by FIRST for Computer Security Incident Response Teams (CSIRT), security communities, organizations, and vendors who may consider implementation with a view to supporting their information sharing and information exchange initiatives.

The IEP framework is composed of four different policy types: Handling, Action, Sharing, and Licensing (HASL).

Let's look at each of these briefly.

Handling policy statements

Policy statement

ENCRYPT IN TRANSIT

Type

HANDLING

Description

States whether the information received has to be encrypted when it is retransmitted by the recipient.

Enumerations

MUST recipients MUST encrypt the information received when it is retransmitted or redistributed. MAY recipients MAY encrypt the information received when it is retransmitted or redistributed.

Required

NO

 

Policy statement

ENCRYPT IN REST

Type

HANDLING

Description

States whether the information received has to be encrypted by the recipient when it is stored.

Enumerations

MUST recipients MUST encrypt the information received when it is stored. MAY recipients MAY encrypt the information received when it is stored.

Required

NO

 

Action policy statements

Policy statement

PERMITTED ACTIONS

Type

ACTION

Description

States the permitted actions that recipients can take upon receiving information.

Enumerations

NONE recipients MUST NOT act upon the information received. CONTACT FOR INSTRUCTION recipients MUST contact the providers before acting upon the information received. An example is where information redacted by the provider could be derived by the recipient and the affected parties identified. INTERNALLY VISIBLE ACTIONS recipients MAY conduct actions on the information received that are only visible on the recipient's internal networks and systems, and MUST NOT conduct actions that are visible outside of the recipient's networks and systems, or that are visible to third parties. EXTERNALLY VISIBLE INDIRECT ACTIONS recipients MAY conduct indirect, or passive, actions on the information received that are externally visible and MUST NOT conduct direct, or active, actions. EXTERNALLY VISIBLE DIRECT ACTIONS recipients MAY conduct direct, or active, actions on the information received that are externally visible.

Required

NO

 

Policy statement

AFFECTED PARTY NOTIFICATIONS

Type

ACTION

Description

Recipients are permitted to notify affected third parties of a potential compromise or threat. Examples include permitting National CSIRTs to send notifications to affected constituents, or a service provider contacting affected customers.

Enumerations

MAY recipients MAY notify affected parties of a potential compromise or threat. MUST NOT recipients MUST NOT notify affected parties of potential compromises or threats.

Required

NO

Sharing policy statements

Policy statement

TRAFFIC LIGHT PROTOCOL

Type

SHARING

Description

Recipients are permitted to redistribute the information received within the scope of redistribution, as defined by the enumerations. The enumerations "RED", "AMBER", "GREEN", and "WHITE" in this document are to be interpreted as described in the FIRST traffic light protocol policy.

Enumerations

RED Personal for identified recipients only. AMBER Limited sharing based on a need­-to-­know basis. GREEN Community-wide sharing. WHITE Unlimited sharing.

Required

NO

 

 

Policy statement

PROVIDER ATTRIBUTION

Type

SHARING

Description

Recipients could be required to attribute or anonymize the provider when redistributing the information received.

Enumerations

MAY recipients MAY attribute the provider when redistributing the information received. MUST recipients MUST attribute the provider when redistributing the information received. MUST NOT recipients MUST NOT attribute the provider when redistributing the information received.

Required

NO

 

Policy statement

OBFUSCATE AFFECTED PARTIES

Type

SHARING

Description

Recipients could be required to obfuscate or anonymize information that could be used to identify the affected parties before redistributing the information received. Examples include removing affected parties' IP addresses, or removing the affected parties names, but leaving the affected parties' industry vertical prior to sending a notification.

Enumerations

MAY recipients MAY obfuscate information concerning the specific parties affected. MUST recipients MUST obfuscate information concerning the specific parties affected. MUST NOT recipients MUST NOT obfuscate information concerning the specific parties affected.

Required

NO

Licensing policy statements

Policy statement

EXTERNAL REFERENCE

Type

LICENSING

Description

This statement can be used to convey a description or reference to any applicable licenses, agreements, or conditions between the producer and receiver, for example, specific terms of use, contractual language, agreement name, or a URL.

Enumerations

There are no EXTERNAL REFERENCE enumerations and this is a free-form text field.

Required

NO

 

Policy statement

UNMODIFIED RESALE

Type

LICENSING

Description

States whether the recipient MAY or MUST NOT resell the information received unmodified, or in a semantically equivalent format, for example, transposing the information from a .csv file format to a .json file format would be considered semantically equivalent.

Enumerations

MAY recipients MAY resell the information received. MUST NOT recipients MUST NOT resell the information received unmodified or in a semantically equivalent format.

Required

NO

 

Metadata policy statements

Policy statement

POLICY ID

Type

METADATA

Description

Provides a unique ID to identify a specific IEP implementation.

Required

YES

 

Policy statement

POLICY VERSION

Type

METADATA

Description

States the version of the IEP framework that has been used, for instance, 1.0.

Required

NO

 

Policy statement

POLICY NAME

Type

METADATA

Description

This statement can be used to provide a name for an IEP implementation, for instance, FIRST Mailing List IEP.

Required

NO

 

Policy statement

POLICY START DATE

Type

METADATA

Description

States the UTC date from when the IEP is effective.

Required

NO

 

Policy statement

POLICY END DATE

Type

METADATA

Description

States the UTC date that the IEP is effective until.

Required

NO

 

Policy statement

POLICY REFERENCE

Type

METADATA

Description

This statement can be used to provide a URL reference to the specific IEP implementation.

Required

NO

 

It is very important for any organization to understand where they are gathering the data from and what the obligations associated with the data are before using it for both internal purposes or sharing. A lack of clear understanding of these could lead to breaches of trust and may not be a desirable situation.

Now that data sharing is behind us, let's talk a little bit about the nature of the data itself. Data usually exhibits three characteristics, which are essential to understand when it comes to designing the data collection system (to be discussed in the next couple of chapters). Industry calls it the 3 V's of data. Let's briefly look at what the 3 V's stand for and why they are important to bear in mind when designing the system.