SQL Server 2017 Integration Services Cookbook

SQL Server 2017 Integration Services Cookbook

By : Christian Cote, Dejan Sarka, David Peter Hansen, Matija Lah, Samuel Lester, Christo Olivier

Buy this Book

SQL Server 2017 Integration Services Cookbook

By: Christian Cote, Dejan Sarka, David Peter Hansen, Matija Lah, Samuel Lester, Christo Olivier

Buy this Book

Overview of this book

SQL Server Integration Services is a tool that facilitates data extraction, consolidation, and loading options (ETL), SQL Server coding enhancements, data warehousing, and customizations. With the help of the recipes in this book, you’ll gain complete hands-on experience of SSIS 2017 as well as the 2016 new features, design and development improvements including SCD, Tuning, and Customizations. At the start, you’ll learn to install and set up SSIS as well other SQL Server resources to make optimal use of this Business Intelligence tools. We’ll begin by taking you through the new features in SSIS 2016/2017 and implementing the necessary features to get a modern scalable ETL solution that fits the modern data warehouse. Through the course of chapters, you will learn how to design and build SSIS data warehouses packages using SQL Server Data Tools. Additionally, you’ll learn to develop SSIS packages designed to maintain a data warehouse using the Data Flow and other control flow tasks. You’ll also be demonstrated many recipes on cleansing data and how to get the end result after applying different transformations. Some real-world scenarios that you might face are also covered and how to handle various issues that you might face when designing your packages. At the end of this book, you’ll get to know all the key concepts to perform data integration and transformation. You’ll have explored on-premises Big Data integration processes to create a classic data warehouse, and will know how to extend the toolbox with custom tasks and transforms.

Title Page

Credits

About the Authors

About the Reviewers

www.PacktPub.com

Customer Feedback

Preface

Free Chapter

SSIS Setup

Introduction

SQL Server 2016 download

Installing JRE for PolyBase

Installing SQL Server 2016

SQL Server Management Studio installation

SQL Server Data Tools installation

Testing SQL Server connectivity

What Is New in SSIS 2016

Introduction

Creating SSIS Catalog

Custom logging

Azure tasks and transforms

Incremental package deployment

Multiple version support

Error column name

Control Flow templates

Key Components of a Modern ETL Solution

Introduction

Installing the sample solution

Deploying the source database with its data

Deploying the target database

SSIS projects

Framework calls in EP_Staging.dtsx

Data Warehouse Loading Techniques

Introduction

Designing patterns to load dimensions of a data warehouse

Loading the data warehouse using the framework

Near real-time and on-demand loads

Using parallelism

Dealing with Data Quality

Introduction

Profiling data with SSIS

Creating a DQS knowledge base

Data cleansing with DQS

Creating a MDS model

Matching with DQS

Using SSIS fuzzy components

SSIS Performance and Scalability

Introduction

Using SQL Server Management Studio to execute an SSIS package

Using T-SQL to execute an SSIS package

Using the DTExec command-line utility to execute an SSIS package

Scheduling an SSIS package execution

Using the cascading lookup pattern

Using the lookup cache

Using lookup expressions

Determining the maximum number of worker threads in a data flow

Using the master package concept

Requesting an execution tree in SSDT

Monitoring SSIS performance

Establishing a performance monitor session

Configuring a performance monitor data collector set

Unleash the Power of SSIS Script Task and Component

Introduction

Using variables in SSIS Script task

Execute complex filesystem operations with the Script task

Reading data profiling XML results with the Script task

Correcting data with the Script component

Validating data using regular expressions in a Script component

Using the Script component as a source

Using the Script component as a destination

SSIS and Advanced Analytics

Introduction

Splitting a dataset into a training and test set

Testing the randomness of the split with a SSAS decision trees model

Preparing a Naive Bayes SSAS data mining model

Querying the SSAS data mining model with the data mining query transformation

Creating an R data mining model

Using the R data mining model in SSIS

Text mining with term extraction and term lookup transformations

On-Premises and Azure Big Data Integration

Introduction

Azure Blob storage data management

Installing a Hortonworks cluster

Copying data to an on-premises cluster

Using Hive – creating a database

Transforming the data with Hive

Transferring data between Hadoop and Azure

Leveraging a HDInsight big data cluster

Managing data with Pig Latin

Importing Azure Blob storage data

Extending SSIS Custom Tasks and Transformations

Introduction

Designing a custom task

Designing a custom transformation

Managing custom component versions

Scale Out with SSIS 2017

Introduction

SQL Server 2017 download and setup

SQL Server client tools setup

Configuring SSIS for scale out executions

Executing a package using scale out functionality

Customer Reviews

5 star

4 star

3 star

2 star

1 star

Managing data with Pig Latin

Pig Latin is one of the programs available in big data clusters. The purpose of this program is to run scripts that can accept any type of data. "Pig can eat everything," as the mantra of the creators states.

This recipe is just meant to show you how to call a simple Pig script. No transformations are done. The purpose of the script is to show you how we can use an Azure Pig task with SSIS.

Getting ready

This recipe assumes that you have created a HDInsight cluster successfully.

How to do it...

In the StgAggregatedSales.dtsx SSIS package, drag and drop an Azure Pig Task onto the control flow. Rename it apt_AggregateData.

Double-click on it to open the Azure HDInsight Pig Task Editor and set the properties as shown in the following screenshot:

In the script property, insert the following code:

SalesExtractsSource = LOAD 'wasbs:///Import/FactOrdersAggregated.txt'; 
rmf wasbs:///Export/; 
STORE SalesExtractsSource INTO 'wasbs:///Export/' USING PigStorage('|');

The first...

SQL Server 2017 Integration Services Cookbook

By : Christian Cote, Dejan Sarka, David Peter Hansen, Matija Lah, Samuel Lester, Christo Olivier

SQL Server 2017 Integration Services Cookbook

By: Christian Cote, Dejan Sarka, David Peter Hansen, Matija Lah, Samuel Lester, Christo Olivier

Overview of this book

Related Content you might be interested in

Current Title:

SQL Server 2017 Integration Services Cookbook

ETL with Azure Cookbook

Hands-On Data Warehousing with Azure Data Factory

Hands-On Data Science with SQL Server 2017

Managing data with Pig Latin

Getting ready

How to do it...