Book Image

AWS Administration - The Definitive Guide - Second Edition

By : Yohan Wadia
Book Image

AWS Administration - The Definitive Guide - Second Edition

By: Yohan Wadia

Overview of this book

Many businesses are moving from traditional data centers to AWS because of its reliability, vast service offerings, lower costs, and high rate of innovation. AWS can be used to accomplish a variety of both simple and tedious tasks. Whether you are a seasoned system admin or a rookie, this book will help you to learn all the skills you need to work with the AWS cloud. This book guides you through some of the most popular AWS services, such as EC2, Elastic Beanstalk, EFS, CloudTrail, Redshift, EMR, Data Pipeline, and IoT using a simple, real-world, application-hosting example. This book will also enhance your application delivery skills with the latest AWS services, such as CodeCommit, CodeDeploy, and CodePipeline, to provide continuous delivery and deployment, while also securing and monitoring your environment's workflow. Each chapter is designed to provide you with maximal information about each AWS service, coupled with easy to follow, hands-on steps, best practices, tips, and recommendations. By the end of the book, you will be able to create a highly secure, fault-tolerant, and scalable environment for your applications to run on.
Table of Contents (17 chapters)
Title Page
Packt Upsell

Chapter 10. Orchestrating Data using AWS Data Pipeline

In the previous chapter, we explored the AWS analytics suite of services by deep diving into Amazon EMR and Amazon Redshift services.

In this chapter, we will be continuing the trend and learning about an extremely versatile and powerful data orchestration and transformation service called AWS Data Pipeline.

Let's have a quick look at the various topics that we will be covering in this chapter:

  • Introducing AWS Data Pipeline along with a quick look at some of its concepts and terminologies
  • Getting started with Data Pipeline using a simple Hello World example
  • Working with the Data Pipeline definition file
  • Executing scripts and commands on remote EC2 instances using a data pipeline
  • Backing up data from one S3 bucket to another using a simple, parameterized data pipeline
  • Building pipelines using the AWS CLI

So without any further ado, let's get started right away!