Book Image

Scalable Data Architecture with Java

By : Sinchan Banerjee
Book Image

Scalable Data Architecture with Java

By: Sinchan Banerjee

Overview of this book

Java architectural patterns and tools help architects to build reliable, scalable, and secure data engineering solutions that collect, manipulate, and publish data. This book will help you make the most of the architecting data solutions available with clear and actionable advice from an expert. You’ll start with an overview of data architecture, exploring responsibilities of a Java data architect, and learning about various data formats, data storage, databases, and data application platforms as well as how to choose them. Next, you’ll understand how to architect a batch and real-time data processing pipeline. You’ll also get to grips with the various Java data processing patterns, before progressing to data security and governance. The later chapters will show you how to publish Data as a Service and how you can architect it. Finally, you’ll focus on how to evaluate and recommend an architecture by developing performance benchmarks, estimations, and various decision metrics. By the end of this book, you’ll be able to successfully orchestrate data architecture solutions using Java and related technologies as well as to evaluate and present the most suitable solution to your clients.

Related Content you might be interested in

Current Title:

Small book image
Scalable Data Architecture with Java
Sinchan Banerjee
Sep, 2022 | 382 pages
Small book image
Building Data Streaming Applications with Apache Kafka
Manish Kumar | Chanchal Singh
Aug, 2017 | 278 pages
Small book image
Modern Data Architecture on AWS
Behram Irani
Aug, 2023 | 420 pages
Small book image
Java Memory Management
Maaike van Putten | Dr Seán Kennedy
Nov, 2022 | 146 pages
Table of Contents (19 chapters)
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Download the example code files
Download the color images
Conventions used
Get in touch
Share Your Thoughts
1
Section 1 – Foundation of Data Systems
Section 1 – Foundation of Data Systems
Free Chapter
2
Chapter 1: Basics of Modern Data Architecture
Chapter 1: Basics of Modern Data Architecture
Exploring the landscape of data engineering
Responsibilities and challenges of a Java data architect
Techniques to mitigate those challenges
Summary
3
Chapter 2: Data Storage and Databases
Chapter 2: Data Storage and Databases
Understanding data types, formats, and encodings
Understanding file, block, and object storage
The data lake, data warehouse, and data mart
Databases and their types
Data model design considerations
Summary
4
Chapter 3: Identifying the Right Data Platform
Chapter 3: Identifying the Right Data Platform
Technical requirements
Virtualization and containerization platforms
Hadoop platforms
Cloud platforms
Choosing the correct platform
Summary
5
Section 2 – Building Data Processing Pipelines
Section 2 – Building Data Processing Pipelines
6
Chapter 4: ETL Data Load – A Batch-Based Solution to Ingesting Data in a Data Warehouse
Chapter 4: ETL Data Load – A Batch-Based Solution to Ingesting Data in a Data Warehouse
Technical requirements
Understanding the problem and source data
Building an effective data model
Designing the solution
Implementing and unit testing the solution
Summary
7
Chapter 5: Architecting a Batch Processing Pipeline
Chapter 5: Architecting a Batch Processing Pipeline
Technical requirements
Developing the architecture and choosing the right tools
Implementing the solution
Querying the ODL using AWS Athena
Summary
8
Chapter 6: Architecting a Real-Time Processing Pipeline
Chapter 6: Architecting a Real-Time Processing Pipeline
Technical requirements
Understanding and analyzing the streaming problem
Architecting the solution
Implementing and verifying the design
Summary
9
Chapter 7: Core Architectural Design Patterns
Chapter 7: Core Architectural Design Patterns
Core batch processing patterns
Core stream processing patterns
Hybrid data processing patterns
Serverless patterns for data ingestion
Summary
10
Chapter 8: Enabling Data Security and Governance
Chapter 8: Enabling Data Security and Governance
Technical requirements
Introducing data governance – what and why
Practical data governance using DataHub and NiFi
Understanding the need for data security
Solution and tools available for data security
Summary
11
Section 3 – Enabling Data as a Service
Section 3 – Enabling Data as a Service
12
Chapter 9: Exposing MongoDB Data as a Service
Chapter 9: Exposing MongoDB Data as a Service
Technical requirements
Introducing DaaS – what and why
Creating a DaaS to expose data using Spring Boot
API management
Enabling API management over the DaaS API using AWS API Gateway
Summary
13
Chapter 10: Federated and Scalable DaaS with GraphQL
Chapter 10: Federated and Scalable DaaS with GraphQL
Technical requirements
Introducing GraphQL – what, when, and why
Core architectural patterns of GraphQL
A practical use case – exposing federated data models using GraphQL
Summary
14
Section 4 – Choosing Suitable Data Architecture
Section 4 – Choosing Suitable Data Architecture
15
Chapter 11: Measuring Performance and Benchmarking Your Applications
Chapter 11: Measuring Performance and Benchmarking Your Applications
Performance engineering and planning
Tools for performance engineering
Publishing performance benchmarks
Optimizing performance
Summary
16
Chapter 12: Evaluating, Recommending, and Presenting Your Solutions
Chapter 12: Evaluating, Recommending, and Presenting Your Solutions
Creating cost and resource estimations
Creating an architectural decision matrix
Data-driven architectural decisions to mitigate risk
Presenting the solution and recommendations
Summary
17
Index
Index
Why subscribe?
18
Other Books You May Enjoy
Other Books You May Enjoy
Packt is searching for authors like you
Share Your Thoughts

Basics of Modern Data Architecture

With the advent of the 21st century, due to more and more internet usage and more powerful data insight tools and technologies emerging, there has been a data explosion, and data has become the new gold. This has implied an increased demand for useful and actionable data, as well as the need for quality data engineering solutions. However, architecting and building scalable, reliable, and secure data engineering solutions is often complicated and challenging.

A poorly architected solution often fails to meet the needs of the business. Either the data quality is poor, it fails to meet the SLAs, or it’s not sustainable or scalable as the data grows in production. To help data engineers and architects build better solutions, every year, dozens of open source and preoperatory tools get released. Even a well-designed solution sometimes fails because of a poor choice or implementation of the tools.

This book discusses various architectural patterns, tools, and technologies with step-by-step hands-on explanations to help an architect choose the most suitable solution and technology stack to solve a data engineering problem. Specifically, it focuses on tips and tricks to make architectural decisions easier. It also covers other essential skills that a data architect requires such as data governance, data security, performance engineering, and effective architectural presentation to customers or upper management.

In this chapter, we will explore the landscape of data engineering and the basic features of data in modern business ecosystems. We will cover various categories of modern data engineering problems that a data architect tries to solve. Then, we will learn about the roles and responsibilities of a Java data architect. We will also discuss the challenges that a data architect faces while designing a data engineering solution. Finally, we will provide an overview of the techniques and tools that we’ll discuss in this book and how they will help an aspiring data architect do their job more efficiently and be more productive.

In this chapter, we’re going to cover the following main topics:

  • Exploring the landscape of data engineering
  • Responsibilities and challenges of a Java data architect
  • Techniques to mitigate those challenges
Previous Section
End of Section 1
Next Section