Mastering Cloud Development using Microsoft Azure

CloudMakers is a newly founded company in Milano, Italy. The founders are as follows:

In addition to these three key people, who have been getting along for many years before founding CM, there are a couple of additional, IT-related people:

CM has been founded with a very limited budget, and the board has taken the decision to minimize the IT investment in order to prioritize their marketing and advertisement efforts until they break even. Under this constraint, every initial investment should be evaluated and approved while pay-per-use services are welcomed.

Luke planned to build the new core infrastructure on Microsoft Azure, using the cloud to quickly deliver new services as well as the development environments required for the entire company and the technical team.

He planned to implement a BYOD (Bring Your Own Device) policy to reduce the governance of physical assets; so, the employees are invited to use their personal laptops and mobile phones for the purpose of work while they're at home or in the office.

Luke's plan is to build a consistent development environment entirely on the cloud, where employees can connect through Remote Desktop Protocol (RDP) to do their work (refer to the Setting up the development environment section).

Since the majority of the code is going to be written using Visual Studio and Luke has had strong experience in using Team Foundation Server (TFS) for Application Lifecycle Management (ALM), CM is going to use Visual Studio Online to store the company's code base as well as to help Leila keep track of the actual activities of the technical team (refer to the Implementing the development process section).

Since Luke is often out of the office due to several business events with Han and due to his participation in international meetings about start-ups and technical conferences and as a consequence of the ability of the entire staff to work from home, CM planned to introduce a way to let employees have a complete workplace even from their mobile devices: smartphones, tablets, and non-Windows PCs (refer to the Provisioning development environments for mobility section).

After a demand made by Owen, overwhelmed by growing requests from Flint to prepare a complex Test/Prod Deployment Environment, Han asks Luke to find an appropriate solution to simplify and automate the process of the creation and maintenance of these environments since they are used to provide clients with demo products. In addition, they should be online in minutes with minimal administrative effort (refer to the Automating repetitive operations section)

In Chapter 2, Enhancing the Hybrid Process, we'll learn how CM enhances the process it's just been created with by paying particular attention to security, disaster recovery, and the control of resources.

In the public cloud, there is often no private data center, so resources are almost always shared between different customers from around the world. This points to several potential issues about security, isolation, and more, but it is up to the cloud vendor to solve these issues.

Anyway, there comes a time when we need to decide where to place our data, our code, and our VMs. If the business is primarily located in the US and we choose a cloud vendor that has a single data center in Australia, this is probably not the better choice in terms of latency.

Again, if we decided for a cloud vendor that has a data center near us (the company) but far from the customers (that is, a company located in the UK with the data center in the UK but users in the US), this is, again, not a good choice. The best idea is a cloud vendor that provides different locations for a service, where companies can choose appropriately where to place (or replicate) services.

Content Delivery Networks (CDNs), covered later on in the book, represent a solution to shorten the distance between content and the end users. While Data Centers (DCs) are few, a CDN has hundreds of nodes distributed in regions even outside the Azure perimeter. For instance, a web application serving photos for users throughout the whole planet can have this topology:

The intention of the user is to gain access to resources faster, since the content is served from the nearest CDN location while the page is served by the central DC.

A good point to start building the company's development environment is to build a development machine for employees. In the last few years, developing through a well-configured virtual machine (either on-premise or in the cloud) has been a growing trend that has had consistent advantages:

In Microsoft Azure, VMs are basically provided in two main operating systems: Windows and Linux. The requirements a VM has are as follows:

Since VNs, NSGs, and IPs will be covered in the next chapters, we'll now introduce the concept of a storage account. For this account, the other requirements discussed earlier can be met even during VM provisioning, but it is better to plan this in advance according to requirement analysis:

These questions should help you understand how it is very important to stop and think before creating a new resource in the cloud. It is true that one of the main pillars of cloud computing is to be "elastic" (and in fact you can create/destroy almost everything in a few seconds/clicks); however, since different services are often interdependent, it is very important to avoid the use of a blind Next | Next approach.

Let's start with creating a new storage account:

During the creation process, some inputs are required:

In the next few steps, we provide a fresh Windows Server environment to later configure the applications required for the reference workstation:

Following the creation steps, you will be prompted with some input requests, such as the following:

The creation of a VM also implies the creation of one (or more) Virtual Hard Disks (VHDs) in order to store the operating system and the data of the running instance. It is possible to look at the contents of a storage account (containing the VHDs of the VMs) in the vhds container of Blob service directly from Portal, as shown in the following figures:

This is the storage account blade, where it is possible to inspect storage services and perform basic operations

When the VM is ready for use, it is possible to connect through RDP to customize it by deploying local software, such as SDKs, development tools, productivity suites, and so on. This is not in the scope of the book, so we assume it has already been done. What we want to do, instead, is create a generalized image of the VM in order to enable further image-based VM creation with minimal administrative effort.

In Windows-based operating systems, the sysprep command is used to generalize the operating system instance, removing specialized details, such as users, keys, and profiles. In Linux, we need to use the Azure Agent with the waagent command.

You can just launch the sysprep command to open this window. Then, specify the Generalize and Shutdown options.

Earlier in this chapter, we used a portal to perform various administrative activities. However, Azure has a fully featured REST management API. In addition, Microsoft provides a .NET management library wrapped by PowerShell in order to perform almost every operation from the command line. We will properly use PowerShell later in the book.

But now, we use a third option to operate on Azure, the ARM Explorer. During the creation of the storage account, we briefly went over the concept of resource groups, the logical container for a group of service instances. The ARM Explorer (available here in preview: https://resources.azure.com/) is a lightweight tool used to inspect and administer Microsoft Azure using the Resource Manager deployment model (which is actually JSON-based).

The ARM Explorer tool is a great tool to learn what's under the hood of the ARM model. For example, for the given VM created previously, this is what the explorer shows:

The output shown by ARM Explorer while selecting a resource from the resources tree

As we can see in the preceding image, ARM Explorer lets us explore the various objects in the Azure subscriptions, also providing a PowerShell sample code to perform basic contextual operations.

In the previous figure, we selected the virtualMachines node. However, if we select the specific VM (in the preceding case, the CMDevReference tree node), the Action tab becomes available with a series of options. To perform the capture of the VM's image, we can proceed as follows:

We can check out the image that has been properly created by navigating the storage account, looking for the VHD we just created. Along with the VHD, Azure created a JSON representing the command to be used with ARM in order to create a fresh VM based on the captured image.

The location where Azure stored the generalized images with the corresponding ARM-ready JSON template to create new VMs

CloudMakers now has a generalized Windows-based image for development machines. In the Automating repetitive operations section, we'll learn how to automate many of these related activities.

CloudMakers developers (which are actually two in number, Luke and Flint, at the time of beginning the start-up) come with different kinds of experience. Among the differences between them, Luke has always used TFS in his project, and he is accustomed to its native, centralized source control engine. On the other hand, Flint has been a full stack developer, using distributed source control engines, such as Mercurial and Git.

As we see later, since every team project has its own source control engine, we can create personal workspaces (workspaces under the control of a single developer to test, play, and prototype) according to the developers' preference, while the company's workspaces (those containing the actual code base) can use a single, shared, and different engine.

We can create a brand new VSO account by directly going to https://go.microsoft.com/fwlink/?LinkId=307137&clcid=0x409. Or, we can use the Azure Portal by creating our first team project.

In the TFS/VSO ecosystem, a single deployment can have this tree structure:

In VSO, there is just one collection (DefaultCollection) with many (to be created by users) Team Projects.

From within the Azure Portal, we can find the New Team Project wizard to open this blade:

This is where we can create new Team Projects

Contextually, if not already created/owned, we need to create a new VSO account.

The creation of the CM account within DefaultCollection

The first team project we create can be a test, so play with the configuration settings:

VSO can integrate users with personal identities (Microsoft accounts) or organizations' identities (Azure AD). We'll learn how Azure AD works in the next sections, but remember that it is a good practice to bind a VSO account with an Azure AD tenant in order to implement better control of who has access to resources.

After provisioning a VSO account, when the billing and some minor configuration details are summarized in the Azure Portal, its management console is available here:

https://<<name_of_account>>.visualstudio.com

Each team project is available at this link, as shown here:

https://<<name_of_account>>.visualstudio.com/DefaultCollection/<<name_of_project>>

The management console of the portal looks similar to the one shown in the following screenshot:

This is the Kanban board created as a consequence of the Process Template selection while creating the team project. We can define tasks, assign people, create items (such as Stories, Iterations, and Features) and also manage other aspects of the development process (using the Code/Build/Test menus).

Visual Studio automatically recognizes VSO/TFS based on the endpoint URL we give to it. In the case of VSO, a web-based sign-in is asked from the user in order to perform a claims-based authentication for the correct provider (a Microsoft account or Azure AD). If a standard TFS is detected, even basic authentication will be available.

This is the DefaultCollection collection of the Team Project of the VSO account along with three Team Projects

Visual Studio 2013/2015 natively supports both Git and TFS as backend source control engines, but only one at time can be the default, as shown in the following screenshot:

This is where we can choose the default source control engine

In the case of TFVC-based Team Projects, we can map and get the entire project collection in a base local folder, as follows:

This is the source control explorer, the place where we can manage the TFVC-based projects of the collection

In the case of a Git-based team project, we can start by just cloning the repository.

We are now cloning the Git-based repo into a local folder

Finally, after writing some code, we can save our work as follows:

In this example, we both Commit (save locally) and Push (align the remote Git repo) in a single action. For further information regarding Git, refer to the Git documentation

As mentioned at the beginning of this section, VSO is a complete solution for ALM, and it's not just another source control environment. It can automatically deploy the build result to connected services, and more generally, it can react to specific triggers with custom actions even against third-party services.

A webhook is a method to notify decoupled systems in a standard way based on HTTP and public endpoints. In VSO, there are several predefined service hooks, whose purpose is to trigger events against third-party services in the case of a specific VSO event that has occurred. At the time of writing, the events supported by VSO (which VSO can listen to) are as follows:

In fact, among the built-in supported third-party services, we have the following:

In fact, there are a lot more built-in integrations, and there is also a provider, Zapier, that acts as a simple orchestrator itself. On the Zapier side, we can react to webhooks by triggering a lot (hundreds) of different services, even the ones mentioned earlier.

Finally, if you want to trigger a custom HTTP listener, there is a simple webhooks provider, letting you decide what to notify and how to do it as long as it's understood that the target should be an HTTP endpoint.

In this hook, we listen to the Work item updated event in case the event occurred in the Core area for an Epic work item type and when someone change the State field

These are some of the invocation settings in custom webhooks. We can also specify an optional basic authentication and the details of what to send in the body of the POST request.

Let's introduce Azure AD before explaining RemoteApp, and let's start identifying what Azure AD is not. It is not the managed version of the Windows Server Active Directory. Despite its resemblance in name, Azure AD is just an IAM service managed and hosted by Microsoft in Azure. We should not even try to make a comparison because they have different scopes and different features. It is true that we can link Azure AD with an on-premise Active Directory, but that will be just for the purpose of extending the on-premise AD functionalities to work with Internet-based applications.

Azure Active Directory is a growing Identity and Access Management (IAM) service (now with B2C features), with great integration options for custom applications; different applications (and the various services of Azure itself) can use Azure AD as their IAM solution in order to provide authentication/authorization.

To create a new Azure AD tenant, we need to move to the old portal (https://manage.windowsazure.com) and proceed as follows:

We create the Azure AD tenant for the CM company.

This directory will host the users of the company in order to provide them with authentication/authorization of services.

If we plan to centralize the access rules for online resources, Azure AD can help us by providing a single identity for our users, created one by one (or through REST APIs), as follows:

This creates a new user

Microsoft tries to solve the issues of the scenario mentioned earlier, implementing a layer of abstraction between the user and the actual VM performing the work. RemoteApp is this layer, completely based on existing technologies (such as RDP), and enables users to connect to applications instead of VMs.

This is the architecture overview of how RemoteApp works

To start from the beginning, we should:

As for the Azure AD tenant, even the RemoteApp collection can be created only from the old portal (https://manage.windowsazure.com).

Note

Classic versus Resource Manager: the hidden issue

Earlier in this chapter, we recommended that you create new resources by always using the Resource Manager mode and not the classic one. This is because the Resource Manager mode is the new one, and hosts new features and investments. However, there is a strong separation between these two areas: so strong that, for example in the old portal (which is based on the classic model), there is no way to reuse components created in the new way.

As an example of this, try to look at storage accounts or custom VM images in the old portal:

This screenshot has been taken in the same subscription where the storage account and the custom VM image were created previously. Remember this behavior and plan your deployments accordingly.

As a quick fix for the lack of the VM image in the old portal, proceed as follows:

Finally, to complete the creation process, go to REMOTEAPP COLLECTIONS and create a new collection based on the custom image you've just registered.

If all went well, you can now do the following:

Azure Automation is a comprehensive set of features that provide automation over Azure resources as well as on-premise ones using PowerShell in conjunction with Azure APIs and other services. Think about Azure Automation as PowerShell-as-a-Service, since it provides an isolated execution engine for arbitrary PowerShell code.

After creating an automation account, it is possible to import existing PowerShell modules by entering the Assets tile in the Automation Account blade and then selecting Modules.

During the creation process, you can choose to create a sample hello world

A custom module can be added to the built-in ones by uploading the corresponding ZIP archive, as follows:

A custom module added to the built-in ones

To be able to write code against actual Azure resources, a valid credential or management certificate is required (in fact, there is no trust between the automation account and the subscription that hosts it).

It is strongly recommended that you create an Azure AD credential with specific permissions in order to place it in the automation code. Azure Automation provides an efficient mechanism to hide the actual credentials for the various Runbooks (scripts) by storing them in advance in the Assets folder.

The Assets folder actually contains resources, as follows:

When creating a new automation account, if specified during creation, a sample runbook (named Get-AzureVMTutorial) is added to the collection. The purpose of this runbook is to list the first 10 Azure VMs in the default subscription.

In this sample runbook, we edit $CredentialAssetName to reflect the Credential asset that we have in the Assets folder

With automation, we can write PowerShell code directly into browser windows and test it using the managed execution engine of Azure. After some tests using the Test pane window (also shown in the preceding screenshot), a runbook can be published, which means that it is available in the Runbooks collection and can be used by other runbooks as a child activity.

In the next few pages, we set up the scenario mentioned earlier, where a well-known VM must be shut down after 6 P.M. and brought back online at 9 AM. To do this, we use a graphical runbook, which is, as the name suggests, a runbook created and edited completely with a GUI composer without touching a line of PowerShell code directly.

We start by creating the runbook, as follows:

We specify the name and the Graphical type

We can now search for the Add-AzureAccount CmdLet and add it in the central canvas.

In the left-hand side pane, we search for CmdLets and add them in the right canvas

Each PowerShell CmdLet has its own set of parameters. The GUI of a Graphical runbook provides a complete and strongly typed mapping between these parameters and the user interface. In fact, for the Add-AzureAccount command, it recognizes the mandatory parameter as follows:

The GUI lets you choose between different parameter sets, and for each set, it provides assistance in filling the expected values

In the preceding example, we just need to select Credential and specify that we need to pick it from the Credentials assets. We can find (if previously set up) the Default credential and use it for the sample.

Following this pattern, we can now add Get-AzureVM, Start-AzureVM and Stop-AzureVM to the canvas.

The Get-AzureVM activity required at least the ServiceName parameter, but VMName can also be specified to get a specific VM instance. We provide this information without hardcoding it into the runbook in two ways:

Using the second method, we define two inputs with default values, as follows:

We specify both input parameters to keep the runbook implementation generic. Default values are used in case no values are provided by the runbook executor.

We can now connect the dots to obtain the following structure:

We linked the three activities in a single workflow. The connections between Get-AzureVM and the lower nodes are different because they apply conditions

The connections from Get-AzureVM to the lower nodes are dashed because we specify a condition to navigate them. By clicking on the single connection, we can specify a set of parameters.

The respective conditions for the two tasks are as follows:

Each activity can have an output: in the preceding case, we check the Status property of the output of the Get-AzureVM activity, and in conjunction with a temporal condition, we activate the appropriate node.

For the Start-AzureVM and Stop-AzureVM activities, the VM can be specified by name using the runbook input, as mentioned earlier or using the activity output data source with one of the available sources for parameter values.

The runbook can now be saved, published, or scheduled on an hourly basis. We can run a runbook in these three ways:

There are a lot of other advanced features in Azure Automation. It provides complete support in order to help you avoid mistakes during repetitive actions, while encouraging the IT staff to use Platform-as-a-service as the execution engine for their scripts.

Azure Scheduler lets us create scheduled actions against an HTTP(s) endpoint as a storage queue. By implementing complex infrastructures, we often need to schedule some actions with specific recurrence: also, Windows Task Scheduler helps us run custom processes in a scheduled fashion. However, having a task scheduler running on a user-managed virtual machine involves maintenance and skills, not to mention that it is not a highly available solution, as the underlying VM represents a single point of failure.

Azure Scheduler, via the Azure Portal or the Management API, lets us submit jobs of two types:

In the former case, we can customize the request by appending custom headers and choosing the appropriate method (GET, POST, and so on); in the latter, we can place a message in a storage queue, assuming there is someone on the other side ready to process it before or later.

Azure Scheduler does not run custom code in .NET or other runtimes: triggered by Azure itself and following the recurrence rules, its purpose is to start a user process. It can be used, for example, in conjunction with Azure Automation webhooks to start specific runbook jobs on an advanced schedule basis.

From the old portal, in the Scheduler section, we can create a new job like this. In this case, we call a plain HTTP endpoint with the GET method

Jobs are created in Job collections and stored in a specific region (the call will happen from this region to the target endpoint specified in the action).

As mentioned earlier, Azure Scheduler is more powerful than the Azure Automation built-in scheduler. In the following figure, we see an example of this:

In this example, we set up a schedule every 1 minute with accurate starting and ending times

We can even specify a body for the HTTP method that allows it, in addition to specific HTTP headers and basic authorization