Book Image

SQL Server 2017 Integration Services Cookbook

By : Christian Cote, Dejan Sarka, David Peter Hansen, Matija Lah, Samuel Lester, Christo Olivier
Book Image

SQL Server 2017 Integration Services Cookbook

By: Christian Cote, Dejan Sarka, David Peter Hansen, Matija Lah, Samuel Lester, Christo Olivier

Overview of this book

SQL Server Integration Services is a tool that facilitates data extraction, consolidation, and loading options (ETL), SQL Server coding enhancements, data warehousing, and customizations. With the help of the recipes in this book, you’ll gain complete hands-on experience of SSIS 2017 as well as the 2016 new features, design and development improvements including SCD, Tuning, and Customizations. At the start, you’ll learn to install and set up SSIS as well other SQL Server resources to make optimal use of this Business Intelligence tools. We’ll begin by taking you through the new features in SSIS 2016/2017 and implementing the necessary features to get a modern scalable ETL solution that fits the modern data warehouse. Through the course of chapters, you will learn how to design and build SSIS data warehouses packages using SQL Server Data Tools. Additionally, you’ll learn to develop SSIS packages designed to maintain a data warehouse using the Data Flow and other control flow tasks. You’ll also be demonstrated many recipes on cleansing data and how to get the end result after applying different transformations. Some real-world scenarios that you might face are also covered and how to handle various issues that you might face when designing your packages. At the end of this book, you’ll get to know all the key concepts to perform data integration and transformation. You’ll have explored on-premises Big Data integration processes to create a classic data warehouse, and will know how to extend the toolbox with custom tasks and transforms.

Related Content you might be interested in

Current Title:

Small book image
SQL Server 2017 Integration Services Cookbook
Christian Cote | Dejan Sarka | David Peter Hansen | Matija Lah | Samuel Lester | Christo Olivier
Jun, 2017 | 558 pages
Small book image
ETL with Azure Cookbook
Madina Saitakhmetova | Christian Cote | Matija Lah
Sep, 2020 | 446 pages
Small book image
Hands-On Data Warehousing with Azure Data Factory
Michelle Gutzait | Christian Cote | Giuseppe Ciaburro
May, 2018 | 284 pages
Small book image
Hands-On Data Science with SQL Server 2017
Vladimir Muzny | Marek Chmel
Nov, 2018 | 506 pages
Table of Contents (18 chapters)
Title Page
Title Page
Credits
Credits
About the Authors
About the Authors
About the Reviewers
About the Reviewers
www.PacktPub.com
www.PacktPub.com
Customer Feedback
Customer Feedback
Preface
Preface
Free Chapter
1
SSIS Setup
SSIS Setup
Introduction
SQL Server 2016 download
Installing JRE for PolyBase
Installing SQL Server 2016
SQL Server Management Studio installation
SQL Server Data Tools installation
Testing SQL Server connectivity
2
What Is New in SSIS 2016
What Is New in SSIS 2016
Introduction
Creating SSIS Catalog
Custom logging
Azure tasks and transforms
Incremental package deployment
Multiple version support
Error column name
Control Flow templates
3
Key Components of a Modern ETL Solution
Key Components of a Modern ETL Solution
Introduction
Installing the sample solution
Deploying the source database with its data
Deploying the target database
SSIS projects
Framework calls in EP_Staging.dtsx
4
Data Warehouse Loading Techniques
Data Warehouse Loading Techniques
Introduction
Designing patterns to load dimensions of a data warehouse
Loading the data warehouse using the framework
Near real-time and on-demand loads
Using parallelism
5
Dealing with Data Quality
Dealing with Data Quality
Introduction
Profiling data with SSIS
Creating a DQS knowledge base
Data cleansing with DQS
Creating a MDS model
Matching with DQS
Using SSIS fuzzy components
6
SSIS Performance and Scalability
SSIS Performance and Scalability
Introduction
Using SQL Server Management Studio to execute an SSIS package
Using T-SQL to execute an SSIS package
Using the DTExec command-line utility to execute an SSIS package
Scheduling an SSIS package execution
Using the cascading lookup pattern
Using the lookup cache
Using lookup expressions
Determining the maximum number of worker threads in a data flow
Using the master package concept
Requesting an execution tree in SSDT
Monitoring SSIS performance
Establishing a performance monitor session
Configuring a performance monitor data collector set
7
Unleash the Power of SSIS Script Task and Component
Unleash the Power of SSIS Script Task and Component
Introduction
Using variables in SSIS Script task
Execute complex filesystem operations with the Script task
Reading data profiling XML results with the Script task
Correcting data with the Script component
Validating data using regular expressions in a Script component
Using the Script component as a source
Using the Script component as a destination
8
SSIS and Advanced Analytics
SSIS and Advanced Analytics
Introduction
Splitting a dataset into a training and test set
Testing the randomness of the split with a SSAS decision trees model
Preparing a Naive Bayes SSAS data mining model
Querying the SSAS data mining model with the data mining query transformation
Creating an R data mining model
Using the R data mining model in SSIS
Text mining with term extraction and term lookup transformations
9
On-Premises and Azure Big Data Integration
On-Premises and Azure Big Data Integration
Introduction
Azure Blob storage data management
Installing a Hortonworks cluster
Copying data to an on-premises cluster
Using Hive – creating a database
Transforming the data with Hive
Transferring data between Hadoop and Azure
Leveraging a HDInsight big data cluster
Managing data with Pig Latin
Importing Azure Blob storage data
10
Extending SSIS Custom Tasks and Transformations
Extending SSIS Custom Tasks and Transformations
Introduction
Designing a custom task
Designing a custom transformation
Managing custom component versions
11
Scale Out with SSIS 2017
Scale Out with SSIS 2017
Introduction
SQL Server 2017 download and setup
SQL Server client tools setup
Configuring SSIS for scale out executions
Executing a package using scale out functionality

Framework calls in EP_Staging.dtsx

This section of the book introduces an ETL framework. The purposes of such a framework are the following:

  • Store execution statistics outside the SSIS catalog: although the Microsoft SSIS team always enhances catalog performance, keeping all execution data in the catalog will degrade catalog performance.
  • Align the data in such a way to be able to do some analytics on the execution data. In that case, think of the framework as an SSIS package execution data warehouse.

The following diagram is the ER diagram of the framework tables. These tables are stored in the SystemLog schema:

Here's the table list and their purposes:

  • LoadApplications: This table contains one value: SSISCookBook sample solution. The purpose of this table, coupled with the Loads table that we'll describe next, is to group many SSIS executions together into a common solution. For example, in this book, we'll have an entry-point package that will call the execution of the entry-point packages...
Previous Section
End of Chapter 3
Next Chapter