DevSecOps Drivers and Challenges

Due to the rapid release of cloud services, law enforcement, security incidents, and tenants' data protection, the security is indispensable to cloud/internet services. Moving security activities from right to left during the development lifecycle and having built-in security practices in the continuous integration pipeline are the goals of DevSecOps.

The business environment, culture, law compliance, and external market drive relate to how the DevSecOps security assurance program rolls out in an organization. The DevSecOps or security assurance program management involved with the whole organization across all business units and the key success to DevSecOps will require all stakeholders to agree with the goal and approaches.

We will cover the following topics in this chapter:

Security compliance (ISO 2700x, FIPS, CSA-CCM)
Legal/law compliance—General Data Protection Regulation (GDPR)
New technology (third-party, cloud, containers, and virtualization)
Cloud service hacks/abuse
Rapid release

As shown in the following diagram, this is how external drivers and challenges impact on a team when delivering secure cloud services:

Security compliance

For cloud services, it's very important to have security compliance-ready. Security compliance not only shows how the security controls of the cloud service meet security standards but also demonstrates security trustworthiness for customers and partners. Security compliance provides an overview of a security assurance program, but it won't specifically tell us which security technical approach it should apply. For frequent cloud service releases, constantly monitoring and auditing to meet security compliance can be a big challenge.

Although most cloud service providers are security compliance ready (ISO, PCI, FedRAMP, SOC, and so on), it's still the cloud service customer's responsibility to secure data and manage their own application compliance assessment. Both cloud service customers and providers need to maintain system or application audit logs, configuration lists, and change histories for compliance assessment. The compliance assessment should be considered a continuous activity—not a one-time audit check.

In this chapter, we will introduce key cloud services security compliance as a reference to building a security assurance program, and how these security compliance standards relate to DevSecOps.

ISO 27001

ISO 27001 is an information security management system (ISMS). It provides an overview of organization-level security assurance programs. ISO 27001 won't specify a technical security approach, but it provides a complete set of a security management programs. As the diagram shows, the segments in the upper parts may be more directly related to DevOps security practices, such as compliance, business continuity, operation security, access control, software development, cryptography, incident management, and communication. This will serve as a guideline to further developing our own DevOps security program:

We won't introduce ISO 27001 details, but the following table summarizes how ISO 27001 relates to each role and the DevOps team:

Role

Company/organization security policy

Operation or DevOps team

Development team

ISO 27001 chapters

5 Information security policies

6 Organization of information security

7 Human resource security

8 Assess management

15 Supplier relationships

11 Physical and environmental security

9 Access Control

10 Cryptography

12 Operation security

13 Communication security

17 Information security aspects of business continuity management

16 Information security incident management

18 Compliance; with internal requirements, such as policies, and external requirements, such as laws

19 Cloud services control

14 System development

10 Cryptography

9 Access control

ISO 27017 and ISO 27018

ISO 27018 is mainly for the protection of personally identifiable information (PII) in the cloud. It's an extended security compliance based on ISO 27001 and ISO 27002. On top of ISO 27001/27002, ISO 27018 additionally defines PII protection security requirements

ISO 27017 provides both service providers and cloud service consumers with the ability to implement security controls for cloud services. ISO 27017 is an extension to ISO 27002 to address cloud-specific security issues.

Cloud Security Alliance (CSA)

As there are many cloud security compliance methods out there, we may get frustrated trying to follow each of them. The CSA (Cloud Security Alliance) Cloud Controls Matrix (CCM) consolidated most security compliance methods into one matrix called CCM. Take application and interface application security as an example—CCM includes all security compliance controls such as ISO, FedRAMP, and NIST 800-53 related to this area, and defines the control ID. The key benefit of referring to CCM is that we can simply focus on CCM and know all other security compliance regulations will be met as well.

In addition, CSA provides a Consensus Assessments Initiative Questionnaire (CAIQ). It's a yes/no questionnaire for cloud consumers or cloud provides to do security self-assessment and to understand the requirements of security controls. Google Vendor Security Assessment Questionnaires (VSAQ) also provide a security assessment questionnaire in terms of Web Application Security, Security and Privacy Program, Infrastructure Security and Physical and Datacenter Security.

Furthermore, if you are looking for the top cloud threats and security control mitigations, Cloud Security Alliance (CSA) cloud top threats provide guidelines. At the time of writing, it defines the top 12 cloud threats, mappings to threat analysis, CCM/Control ID, and the domains of CSA Security Guidance reference. The following table shows related CSA security guides and how to apply security practices in your organization:

CSA security guides	What it is?	When to apply?
CSA Security Guidance reference	Cloud security white paper	If your organization needs a cloud service security guideline or white paper, this can be a good reference.
Cloud top threats	Top 12 cloud threats and mappings to threat analysis, CCM/Control ID, and domains of CSA Security Guidance reference	It can be the basis for cloud threat modeling.
CAIQ	Yes/no questionnaire	A list of yes/no questions for self-assessment to understand existing security control requirements.
CSA CCM	One consolidated worldwide security standard mapping	It's a great consolidated reference and includes most security compliance standards (ISO 27001, PCI, NIST, and so on). It's the only matrix you need to review security standards compliance.

Federal Information Processing Standards (FIPS)

The FIPS mainly defines minimum security requirements for the use of cryptographic modules. Every organization that is not going to get a FIPS certificate should also refer to it. It's highly recommended that you refer to Security Requirements for Cryptographic Modules to understand what cryptographic modules may be considered safe, legacy, or weak.

For developers who would like to learn how to implement cryptographic modules correctly, the following resources are recommended.

OWASP Cryptographic Storage Cheat Sheet.
OWASP Guide to Cryptography
OWASP Key Management Cheat Sheet

Here is a summary of the minimum security requirements for each cryptography algorithm and its usage:

Usage scenario	Unsafe cryptography algorithm (key length)	Legacy Systems Only	Recommended cryptography algorithm
Symmetric encryption	Blowfish, DES, Skipjack, RC4	3 DES only when (key 1 != key 2 != key 3)	AES > 128 bits
Asymmetric encryption	RSA (< 1024 bits)	RSA (1024 bits)	RSA (> 1024 bits)
Hash	MD5	SHA1 (1024 bits)	SHA256
Digital signature	RSA (< 1024 bits) DSA (< 1024 bits) ECDSA (<= 160 bits)	DSA (1024 bits) RSA (1024 bits)	RSA (>=2048 bits) DSA (>=2048 bits) ECDSA (>=256 bits)
Hellman key exchange (DH)	DH ( < 1024 bits)	DH (1024-2047 bits)	DH (>=2048 bits) ECDH(>-256 bits)

Center for Internet Security (CIS) and OpenSCAP – securing your infrastructure

The CIS defines security benchmarks and the National Checklist Program (NCP), defined by the NIST SP 800-70, provides guidance on the security configurations of the operating system, database, virtualization, framework, and applications.

The IT and operation team are primarily responsible for ensuring the security of the infrastructure. However, the development team may also share some responsibilities for securing the infrastructure. For example, the development team may decide to deliver the application package in the form of a container or to apply Infrastructure as Code frameworks, such as Puppet or Chef. These technologies allow development teams to define a secure configuration, even in the development stage, and the operation team just needs to apply the secure configuration definition during application deployment.

In addition, it's also the development team's job to provide a list of configuration changes for every release's deployment. This will allow the operation team to review if the configuration changes are secure and appropriate. Due to the complexity and the amount of configuration that needs to be reviewed, the adoption of scanning tools to check if all the configurations are secure and comply with industry best practices is necessary. Cloud service providers may provide such scanning services or tools. Here, we recommend open source tools such as CIS-CAT Lite provided by CIS and OpenSCAP.

The journey to secure the infrastructure and platform can be completed in three stages. The first stage is to define a secure configuration baseline by referring to industry practices such as CIS or NIST NCP. Then, we may apply tools such as Chef or Puppet to ensure every deployment includes a secure configuration as well. The final stage is to do constant monitoring of frequent configuration changes and security compliance assessment.

Typical infrastructure components are listed in the following table. CIS provides secure configuration suggestions to each system component and also tools to do the scanning against the security best practice baseline.

CIS provides the CIS Benchmark, which defines the secure configuration of operating systems, server software, cloud services, networking devices, and so on. It helps operation teams to understand how to secure and configure an infrastructure and platform.

Infrastructure layers	System
Web services	Apache, Nginx, IIS
Database	MS SQL, MySQL, Oracle, MongoDB
Virtualization/container	VMware, Docker, Kubernetes
Networking	Cisco devices
Operating systems	Windows, Linux, Ubuntu, CentOS, SUSE

In addition to CIS Benchmark documents, CIS also provides tools to infrastructure or operation teams for secure configuration scanning. The CIS Security website provides related security configuration scanning tools to download.

Source: https://www.cisecurity.org/cybersecurity-tools/

National Checklist Program (NCP) repository

The NCP repository provides secure configuration for specific software components. For example, if you are looking for Apache security configuration or the CIS of Apache, you may use the NCP to do the search. The screenshot is from the NIST NCP (National Checklist Program).

Source: https://nvd.nist.gov/ncp/repository

OpenSCAP tools

OpenSCAP is similar to CIS security benchmarks; it also provides a secure configuration baseline. In addition, OpenSCAP also provides different kinds of tool for operation or infrastructure teams to do secure configuration evaluation and scanning. Depending on the requirements, there are four kinds of tool provided, as shown in the following screenshot:

Source: https://www.open-scap.org/tools/

Legal and security compliance

The EU GDPR, which came into force in May 2018, protects all EU citizens from privacy and data breaches. According to the GDPR FAQ:

"The GDPR not only applies to organizations located within the EU but it also applies to all companies processing and holding the personal data of data subjects residing in the European Union, regardless of the company's location."

In other words, if a company is providing services to customers in the European Union, its data handling will need to comply entirely with GDPR. From a DevSecOps point of view, it's related to data collection, handling, storage, backup, modification, transport, and removal—in a secure manner. According to GDPR Article 5, there are six privacy principles:

Lawfulness, fairness, and transparency
Purpose limitations
Data minimization
Accuracy
Storage limitations
Integrity and confidentiality

GDPR, like other security compliance policies, doesn't define the technical approach to achieve it. GDPR can still be too high-level for an engineering team. It needs to translate into software security requirements, design, threat modeling, tools, and so on. The following table summarizes typical security practices for the engineering team:

Stage	Common security practices for privacy or sensitive info handing
Design	Privacy Impact Assessment (PIA)
Coding	Data masking library Anonymous toolbox RAPPOR—privacy-preserving reporting algorithms Encryption storage (RSA, ASE) Secure erasure Secure communication protocol (such as TLS v1.2, SSH v2, SFTP, SNMP v3) Cookie consent Data Vault Key management
Testing	OWASP testing for weak cryptography, testing for error handling, testing for configuration, and so on
Deployment	OWASP configuration and deployment management testing CIS secure environment configuration Sensitive information in Git
Monitoring	ELK for log analysis Integrity monitoring (IDS/IPS) to monitor any unauthorized changes CIS secure configuration monitoring Sensitive information leakage in Git

New technology (third-party, cloud, containers, and virtualization)

New technologies such as virtualization, Docker, and microservices introduce new methods of software delivery and speeds up application deployment, but also brings new security threats and risks. We will briefly discuss how these new technologies change the practices of security and DevOps.

Virtualization

It's very common to install application services on top of a virtualized OS. Virtualization technology helps to make the most physical machine resources such as the CPU, memory, and disks. However, virtualization is a shared OS technology. It also introduces security risks such as VM escape, information leakage, and denial-of-service for applications running on top of virtualization.

Security practices in guest OS virtualization are normally involved with both OS and virtualization hardening. Here are some key security configurations related to virtualization. Refer to CIS Benchmarks for details:

Limit informative messages from the VM to the VMX file
Limit sharing console connections
Disconnect unauthorized devices (USB, DVD, serial devices, and so on)
Disable BIOS Boot Specification (BBS)
Disable guest-host interaction protocol handler
Disable host guest filesystem server
Disable VM console paste operations
Disable virtual disk shrinking
Do not send host information to guests

The following diagram shows the adoption of virtualization. Virtualization adds a hypervisor layer on top of the physical server so that the virtualized guest OS can run on top of it:

In addition to the secure configuration of virtualization, applying a security patch to virtualization is also a must for operation or IT teams.

In addition, the following resources may help you to search for Common Vulnerabilities and Exposures (CVE) in vulnerability databases:

Exploit Database https://www.exploit-db.com/
SecLists http://seclists.org/fulldisclosure/
Vulnerability Notes Database https://www.kb.cert.org/vuls/

To search for the vulnerabilities of a specific product or vendor, refer to the URL search for VMware as following:

Dockers

The introduction of Docker provides software package delivery and installation with new choices and can be one of the best ways to isolate different applications without using a whole separate OS virtual machine. Software can be packaged into a container by Docker. A container, like a VM image, includes everything needed to run application services such as runtime, system libraries, and settings. The key difference between a virtual machine image and a container is that the container doesn't actually include the whole OS. The container only includes key necessary system libraries and every container shares the same OS kernel during runtime. Therefore, Docker containers can boot up within seconds and use much less memory or far fewer disks than virtualization images.

The use of Docker can also greatly help operation teams to do deployment and secure configuration since a Docker container includes every configuration and the settings you need to run. To understand Docker security practices, check out the CIS Docker Benchmark and Docker security in the Further reading section.

Key secure practices and configurations of Docker are listed here:

Separate partition for containers
Updated Linux kernel
Only allow trusted users to control the Docker daemon
Audit the Docker daemon, files, and directories
Restrict network traffic between containers
TLS authentication for the Docker daemon
Do not bind Docker to another IP/port or a Unix socket
Docker daemon configuration files permissions
Container runtime (Linux Kernel capabilities, SSH, ports, memory, CPU, IPC)

The following diagram shows the key difference between virtualization and Docker. Virtualization is a guest OS level while Docker is actually an application-level isolation and shares the same guest OS:

Here is a summary of the known security vulnerabilities identified in Docker.

CVE ID	Related CWE ID	Description
CVE-2014-5282	20	Docker before 1.3 does not properly validate image IDs, which allows remote attackers to redirect to another image through the loading of untrusted images via Docker load.
CVE-2017-14992	20	Lack of content verification in Docker-CE (also known as Moby), and earlier allows a remote attacker to launch a Denial of Service attack via a crafted image layer payload; a.k.a Gzip bombing.
CVE-2017-7297	264	Rancher Labs rancher server 1.2.0+ is vulnerable to authenticated users disabling access control via an API call. This is fixed in versions rancher/server:v1.2.4, rancher/server:v1.3.5, rancher/server:v1.4.3, and rancher/server:v1.5.3.
CVE-2016-9962	362	RunC allowed additional container processes via runc exec to be ptraced by the pid 1 of the container. This allows the main processes of the container, if running as root, to gain access to file-descriptors of these new processes during initialization and can lead to container escapes or modification of runC state before the process is fully placed inside the container.
CVE-2014-0047	n/a	Docker before 1.5 allows local users to have an unspecified impact via vectors involving unsafe /tmp usage.

Here is a tip to query a specific vulnerability. Take 'CVE-2014-0047' as an example; just replace the CVE ID number at the end of the following URL.

Infrastructure as Code (IaC)

Puppet, Chef, Ansible, and SaltStack are tools to apply IaC. The key advantage of using these tools is that any system configuration can be defined in a text file at the design stage and changes can be managed by versions. This will help both operation or development teams to build security configuration baselines such as file permissions, firewall rules, web configurations, or MySQL connections. Once the security configuration baseline is defined, the operation team can monitor any unauthorized changes or roll back the configuration to previous specific versions.

For example, we may have baseline security firewall rules for a web services environment to only allow ports 80 and 443. All an operation team needs to do is to define the firewall rules by using one of the tools (Puppet, Chef, Ansible, SaltStack), and the framework will apply the rules, audit, and even correct changes if other ports are opened by mistake or by other service deployments.

The DevSec Hardening Framework project available at https://github.com/dev-sec provides Ansible, Chef, and Puppet secure configuration baseline template scripts.

The following diagram shows how IaC (for example, Puppet) works:

Cloud services hacks/abuse

A CSA survey on the top cloud security concerns has identified the following 12 issues:

Data breaches
Weak identity, credentials, and access management
Insecure APIs
System and application vulnerabilities
Account hijacking
Malicious insiders
Advanced Persistent Threats (APTs)
Data loss
Insufficient due diligence
Abuse and nefarious use of cloud services
Denial of service
Shared technology issues

In addition, service abuse has also become a headache for most e-commerce or shopping sites. Let's take one example to understand how hackers or misconduct users can benefit from it.

Case study – products on sale

Assume that one online shopping store is going to have a 50% discount on one new model phone for only the first 100 customers; it will be available at 12:00 on February 1.

What do hackers do?

For this kind of sale with 50 % profit is a great attraction for malicious users to do something. What underground users typically may do involves the massive registration of user accounts. There can be more than 10,000 users accounts registered in a short period of time just before the sales. At the moment of the sale, they will use automated scripts to trigger purchase behaviors and finish the orders within seconds. Once they have ordered or occupied all the phones, they may either sell them at higher prices or even not pay for the orders.

Is this illegal? These behaviors follow the business rules for registration and purchases. Although the behavior may not be against the law, it may be considered misconduct or service abuse. Therefore, this kind of on-sale activity may require additional business rules and regulations. After all, it's not purely hacking behavior. We will discuss this in later chapters. Here, we provide an overview of alleviating measures, which can be by means of business rules or technical approaches:

The sale is only limited to those customers with a certain period of purchase history
Apply CAPTCHA to distinguish humans from machines
Two-factor authentication and registration via phone SMS
Detection and correlation of IP, phone number, email, account ID, physical address, and GeoIP location
Unusual page browsing behavior such as skipping products and jumping to the purchase directly
Unusual massive logins or registration from the same IP or devices

Rapid release

Rapid, frequent, and iterative releases are very common for cloud services. This normally drives the need for DevOps practices. This can be both an opportunity and a challenge to security. The challenge is that a short period of frequent releases may not include enough time to do a full cycle of security testing. There are three maturity levels of DevOps practices:

Maturity level	Achieved	Technology adoption
Continuous integration	Source code repository and version control CI workflow with a daily build and unit testing	Jenkins Git Unit testing
Continuous delivery	Automated deploy to the staging environment Integration testing on the staging environment Deployment to production is done manually	IaC(Puppet) Docker
Continuous deployment	Automated deployment and acceptance testing on production Production changes or configuration management	IaC (puppet) Docker Automated acceptance testing Configuration monitoring

The adoption of DevOps practices means more collaboration between development, QA, IT, and operation teams, and more in-progress adoption of continuous integration or continuous delivery tools. This provides a good foundation to move to DevSecOps. Depending on the maturity level of the existing CI/CD, security practices or tools can be added on top of the existing CI/CD framework. It's the most effective and least learning curve to introduce security is don't change existing development, QA, IT, operation team the way they work. Building security tools around the existing CI/CD is still the best approach. We will explore this more in upcoming chapters.

The diagram below shows the security involved with development, QA, and operations through the whole CI/CD lifecycle.

Summary

In this chapter, we discussed external factors that drive the need for security such as security compliance, regulations, and the market. In addition, the adoption of new technologies also brings about new challenges such as Docker, virtualization, cloud services, and IaC.

For security compliance, we briefly discussed ISO 27001 and some security best practices/tools introduced by CSA such as CCM, cloud security guide, CAIQ, and Cloud top threats. FIPS was also discussed for the correct usage of cryptography. In terms of infrastructure security, CIS and OpenSCAP were introduced. Finally, the EU GDPR law regulates and drives the security requirements of data and privacy protection.

Based on all these security challenges and compliance rules, we introduced one small case study for cloud services, which could be hacked and abused. Moreover, what security technologies may apply to DevOps practices. In upcoming chapters, we will further discuss how security goals, metrics, and security assurance programs apply to different kinds of organization and practices.

Questions

Does FIPS define the security requirements for cryptography?
Which of the following defines the security compliance is primarily focused on personal data privacy?
1. ISO 27018
2. FIPS
3. GDPR
4. CIS
What can be considered cloud service abuse?
1. Account sharing
2. Brute-force logins
3. API abuse
4. All of the above

What is the CIS security benchmark used for?
1. Anti-virus
2. Defining secure configuration of the OS, platform, databases, and so on
3. Firewall
4. Integrity
Which role is involved with security practices during the DevOps cycle?
1. QA
2. RD
3. Operations
4. All of the above
How does the technology Infrastructure as Code help security operation teams?
1. Virus detection
2. Secure configuration
3. Intrusion detection
4. Encryption
Which of the following is not a privacy principle?
1. Spoofing
2. Purpose limitations
3. Storage limitations
4. Accuracy

Threat Modeling tools/templates	Rationale and purpose
Knowledge-base of threats and mitigation	Threat and mitigation knowledge can help the team...

Hands-On Security in DevOps

By : Tony Hsiang-Chih Hsu

Hands-On Security in DevOps

By: Tony Hsiang-Chih Hsu

Overview of this book

Related Content you might be interested in

Current Title:

Hands-On Security in DevOps

Practical Security Automation and Testing

Implementing DevSecOps Practices

Cloud Native Software Security Handbook