Book Image

Applied Architecture Patterns on the Microsoft Platform

Book Image

Applied Architecture Patterns on the Microsoft Platform

Overview of this book

Every day, architects and developers are asked to solve specific business problems in the most efficient way possible using a broad range of technologies. Packed with real-world examples of how to use the latest Microsoft technologies, this book tackles over a dozen specific use case patterns and provides an applied implementation with supporting code downloads for every chapter. In this book, we guide you through thirteen architectural patterns and provide detailed code samples for the following technologies: Windows Server AppFabric, Windows Azure Platform AppFabric, SQL Server (including Integration Services, Service Broker, and StreamInsight), BizTalk Server, Windows Communication Foundation (WCF), and Windows Workflow Foundation (WF). This book brings together – and simplifies – the information and methodology you need to make the right architectural decisions and use a broad range of the Microsoft platform to meet your requirements. Throughout the book, we will follow a consistent architectural decision framework which considers key business, organizational, and technology factors. The book is broken up into four sections. First, we define the techniques and methodologies used to make architectural decisions throughout the book. In Part I, we provide a set of primers designed to get you up to speed with each of the technologies demonstrated in the book. Part II looks at messaging patterns and includes use cases which highlight content-based routing, workflow, publish/subscribe, and distributed messaging. Part III digs into data processing patterns and looks at bulk data processing, complex events, multi-master synchronization, and more. Finally, Part IV covers performance-related patterns including low latency, failover to the cloud, and reference data caching.
Table of Contents (26 chapters)
Applied Architecture Patterns on the Microsoft Platform
Credits
Foreword
About the Authors
About the Reviewer
Preface

Chapter 12. Debatching Bulk Data

Debatching data is the process of turning one huge pile of data into many small piles of data.

Why is it better to shovel one ton of data using two thousand, one pound shovels instead of one big load from a huge power shovel? After all, large commercial databases and the attendant bulk loader or SQL Loader programs are designed to do just that: insert huge loads of data in a single shot.

The bulk load approach works under certain tightly constrained circumstances. They are as follows:

  1. 1. The "bulk" data comes to you already matching the table structure of the destination system. Of course, this may mean that it was debatched before it gets to your system.

  2. 2. The destination system can accept some, potentially significant, error rate when individual rows fail to load.

  3. 3. There are no updates or deletes, just inserts.

  1. 4. Your destination system can handle bulk loads. Certain systems (for example, some legacy medical systems or other proprietary systems) cannot handle...