Book Image

DevOps: Puppet, Docker, and Kubernetes

By : Neependra Khare, Ke-Jou Carol Hsu, Hideto Saito, John Arundel, Hui-Chuan Chloe Lee, Thomas Uphill
Book Image

DevOps: Puppet, Docker, and Kubernetes

By: Neependra Khare, Ke-Jou Carol Hsu, Hideto Saito, John Arundel, Hui-Chuan Chloe Lee, Thomas Uphill

Overview of this book

With so many IT management and DevOps tools on the market, both open source and commercial, it’s difficult to know where to start. DevOps is incredibly powerful when implemented correctly, and here’s how to get it done.This Learning Path covers three broad areas: Puppet, Docker, and Kubernetes. This Learning Path is a large resource of recipes to ease your daily DevOps tasks. We begin with recipes that help you develop a complete and expert understanding of Puppet’s latest and most advanced features. Then we provide recipes that help you efficiently work with the Docker environment. Finally, we show you how to better manage containers in different scenarios in production using Kubernetes. This course is based on these books: 1. Puppet Cookbook, Third Edition 2. Docker Cookbook 3. Kubernetes Cookbook
Table of Contents (6 chapters)

While Facter's built-in facts are useful, it's actually quite easy to add your own facts. For example, if you have machines in different data centers or hosting providers, you could add a custom fact for this so that Puppet can determine whether any local settings need to be applied (for example, local DNS servers or network routes).

The name of the Ruby file that holds the fact definition is irrelevant. You can name this file whatever you wish; the name of the fact comes from the Facter.add() function call. You may also call this function several times within a single Ruby file to define multiple facts as necessary. For instance, you could grep the /proc/meminfo file and return several facts based on memory information as shown in the meminfo.rb file in the following code snippet:

After synchronizing this file to a node, the memory_active and memory_inactive facts would be available as follows:

You can extend the use of facts to build a completely nodeless Puppet configuration; in other words, Puppet can decide what resources to apply to a machine, based solely on the results of facts. Jordan Sissel has written about this approach at http://www.semicomplete.com/blog/geekery/puppet-nodeless-configuration.html.

You can find out more about custom facts, including how to make sure that OS-specific facts work only on the relevant systems, and how to weigh facts so that they're evaluated in a specific order at the puppetlabs website:

http://docs.puppetlabs.com/guides/custom_facts.html

How to do it...

Here's an example of a simple custom fact:

Create the directory modules/facts/lib/facter and then create the file modules/facts/lib/facter/hello.rb with the following contents:
Facter.add(:hello) do
  setcode do
    "Hello, world"
  end
end
Modify your site.pp file as follows:
node 'cookbook' {
  notify { $::hello: }
}
Run Puppet:
[root@cookbook ~]# puppet agent -t
Notice: /File[/var/lib/puppet/lib/facter/hello.rb]/ensure: defined content as '{md5}f66d5e290459388c5ffb3694dd22388b'
Info: Loading facts
Info: Caching catalog for cookbook.example.com
Info: Applying configuration version '1416205745'
Notice: Hello, world
Notice: /Stage[main]/Main/Node[cookbook]/Notify[Hello, world]/message: defined 'message' as 'Hello, world'
Notice: Finished catalog run in 0.53 seconds

The name of the Ruby file that holds the fact definition is irrelevant. You can name this file whatever you wish; the name of the fact comes from the Facter.add() function call. You may also call this function several times within a single Ruby file to define multiple facts as necessary. For instance, you could grep the /proc/meminfo file and return several facts based on memory information as shown in the meminfo.rb file in the following code snippet:

After synchronizing this file to a node, the memory_active and memory_inactive facts would be available as follows:

You can extend the use of facts to build a completely nodeless Puppet configuration; in other words, Puppet can decide what resources to apply to a machine, based solely on the results of facts. Jordan Sissel has written about this approach at http://www.semicomplete.com/blog/geekery/puppet-nodeless-configuration.html.

You can find out more about custom facts, including how to make sure that OS-specific facts work only on the relevant systems, and how to weigh facts so that they're evaluated in a specific order at the puppetlabs website:

http://docs.puppetlabs.com/guides/custom_facts.html

How it works...

Facter facts are defined in

The name of the Ruby file that holds the fact definition is irrelevant. You can name this file whatever you wish; the name of the fact comes from the Facter.add() function call. You may also call this function several times within a single Ruby file to define multiple facts as necessary. For instance, you could grep the /proc/meminfo file and return several facts based on memory information as shown in the meminfo.rb file in the following code snippet:

After synchronizing this file to a node, the memory_active and memory_inactive facts would be available as follows:

You can extend the use of facts to build a completely nodeless Puppet configuration; in other words, Puppet can decide what resources to apply to a machine, based solely on the results of facts. Jordan Sissel has written about this approach at http://www.semicomplete.com/blog/geekery/puppet-nodeless-configuration.html.

You can find out more about custom facts, including how to make sure that OS-specific facts work only on the relevant systems, and how to weigh facts so that they're evaluated in a specific order at the puppetlabs website:

http://docs.puppetlabs.com/guides/custom_facts.html

There's more...

The name of the Ruby file that holds the fact definition is irrelevant. You can name this file whatever you wish; the name of the fact comes from the Facter.add() function call. You may also call this function several times within a single Ruby file to define multiple facts as necessary. For instance, you could grep the /proc/meminfo file and return several facts based on memory information as shown in the meminfo.rb file See also

The Importing dynamic information recipe in
  • Chapter 3, Writing Better Manifests
  • The Configuring Hiera recipe in Chapter 2, Puppet Infrastructure

The Creating custom facts recipe describes how to add extra facts written in Ruby. You can also create facts from simple text files or scripts with external facts instead.

External facts live in the /etc/facter/facts.d directory and have a simple key=value format like this:

In this example, we'll create an external fact by creating files on the node. We'll also show how to override a previously defined fact.

  1. Current versions of Facter will look into /etc/facter/facts.d for files of type .txt, .json, or .yaml. If facter finds a text file, it will parse the file for key=value pairs and add the key as a new fact:
    [root@cookbook ~]# facter model
    ED-209
    
  2. If the file is a YAML or JSON file, then facter will parse the file for key=value pairs in the respective format. For YAML, for instance:
    ---
    registry: NCC-68814
    class: Andromeda
    shipname: USS Prokofiev
  3. The resulting output will be as follows:
    [root@cookbook ~]# facter registry class shipname
    class => Andromeda
    registry => NCC-68814
    shipname => USS Prokofiev
    
  4. In the case of executable files, Facter will assume that their output is a list of key=value pairs. It will execute all the files in the facts.d directory and add their output to the internal fact hash.
  5. In the users example, Facter will execute the users.sh script, which results in the following output:
    users=2
    
  6. It will then search this output for users and return the matching value:
    [root@cookbook ~]# facter users
    2
    
  7. If there are multiple matches for the key you specified, Facter determines which fact to return based on a weight property. In my version of facter, the weight of external facts is 10,000 (defined in facter/util/directory_loader.rb as EXTERNAL_FACT_WEIGHT). This high value is to ensure that the facts you define can override the supplied facts. For example:
    [root@cookbook ~]# facter architecture
    x86_64
    [root@cookbook ~]# echo "architecture=ppc64">>/etc/facter/facts.d/myfacts.txt
    [root@cookbook ~]# facter architecture
    ppc64
    

Since all external facts have a weight of 10,000, the order in which they are parsed within the /etc/facter/facts.d directory sets their precedence (with the last one encountered having the highest precedence). To create a fact that will be favored over another, you'll need to have it created in a file that comes last alphabetically:

Getting ready

Here's what you need to do to prepare your system to add external facts:

You'll need Facter Version 1.7 or higher to use external facts, so look up the value of facterversion or use facter -v:
[root@cookbook ~]# facter facterversion
2.3.0
[root@cookbook ~]# facter -v
2.3.0
You'll also need to create the external facts directory, using the following command:
[root@cookbook ~]# mkdir -p /etc/facter/facts.d

In this example, we'll create an external fact by creating files on the node. We'll also show how to override a previously defined fact.

  1. Current versions of Facter will look into /etc/facter/facts.d for files of type .txt, .json, or .yaml. If facter finds a text file, it will parse the file for key=value pairs and add the key as a new fact:
    [root@cookbook ~]# facter model
    ED-209
    
  2. If the file is a YAML or JSON file, then facter will parse the file for key=value pairs in the respective format. For YAML, for instance:
    ---
    registry: NCC-68814
    class: Andromeda
    shipname: USS Prokofiev
  3. The resulting output will be as follows:
    [root@cookbook ~]# facter registry class shipname
    class => Andromeda
    registry => NCC-68814
    shipname => USS Prokofiev
    
  4. In the case of executable files, Facter will assume that their output is a list of key=value pairs. It will execute all the files in the facts.d directory and add their output to the internal fact hash.
  5. In the users example, Facter will execute the users.sh script, which results in the following output:
    users=2
    
  6. It will then search this output for users and return the matching value:
    [root@cookbook ~]# facter users
    2
    
  7. If there are multiple matches for the key you specified, Facter determines which fact to return based on a weight property. In my version of facter, the weight of external facts is 10,000 (defined in facter/util/directory_loader.rb as EXTERNAL_FACT_WEIGHT). This high value is to ensure that the facts you define can override the supplied facts. For example:
    [root@cookbook ~]# facter architecture
    x86_64
    [root@cookbook ~]# echo "architecture=ppc64">>/etc/facter/facts.d/myfacts.txt
    [root@cookbook ~]# facter architecture
    ppc64
    

Since all external facts have a weight of 10,000, the order in which they are parsed within the /etc/facter/facts.d directory sets their precedence (with the last one encountered having the highest precedence). To create a fact that will be favored over another, you'll need to have it created in a file that comes last alphabetically:

How to do it...

In this example, we'll create a simple external fact that returns a message, as shown in the Creating custom facts recipe:

Create the file /etc/facter/facts.d/local.txt with the following contents:
model=ED-209
Run the following command:
[root@cookbook ~]# facter model
ED-209

Well, that was easy! You can add more facts to the same file, or other files, of course, as follows:

model=ED-209
builder=OCP
directives=4

However, what if you need to compute a fact in some way, for example, the number of logged-in users? You can create executable facts to do this.

Create the file /etc/facter/facts.d/users.sh with the following contents:
#!/bin/sh
echo users=`who |wc -l`
Make this file executable with the following command:
[root@cookbook ~]# chmod a+x /etc/facter/facts.d/users.sh
Now check the users value with the following command:
[root@cookbook ~]# facter users
2

In this example, we'll create an external fact by creating files on the node. We'll also show how to override a previously defined fact.

  1. Current versions of Facter will look into /etc/facter/facts.d for files of type .txt, .json, or .yaml. If facter finds a text file, it will parse the file for key=value pairs and add the key as a new fact:
    [root@cookbook ~]# facter model
    ED-209
    
  2. If the file is a YAML or JSON file, then facter will parse the file for key=value pairs in the respective format. For YAML, for instance:
    ---
    registry: NCC-68814
    class: Andromeda
    shipname: USS Prokofiev
  3. The resulting output will be as follows:
    [root@cookbook ~]# facter registry class shipname
    class => Andromeda
    registry => NCC-68814
    shipname => USS Prokofiev
    
  4. In the case of executable files, Facter will assume that their output is a list of key=value pairs. It will execute all the files in the facts.d directory and add their output to the internal fact hash.
  5. In the users example, Facter will execute the users.sh script, which results in the following output:
    users=2
    
  6. It will then search this output for users and return the matching value:
    [root@cookbook ~]# facter users
    2
    
  7. If there are multiple matches for the key you specified, Facter determines which fact to return based on a weight property. In my version of facter, the weight of external facts is 10,000 (defined in facter/util/directory_loader.rb as EXTERNAL_FACT_WEIGHT). This high value is to ensure that the facts you define can override the supplied facts. For example:
    [root@cookbook ~]# facter architecture
    x86_64
    [root@cookbook ~]# echo "architecture=ppc64">>/etc/facter/facts.d/myfacts.txt
    [root@cookbook ~]# facter architecture
    ppc64
    

Since all external facts have a weight of 10,000, the order in which they are parsed within the /etc/facter/facts.d directory sets their precedence (with the last one encountered having the highest precedence). To create a fact that will be favored over another, you'll need to have it created in a file that comes last alphabetically:

How it works...

In this example, we'll

create an external fact by creating files on the node. We'll also show how to override a previously defined fact.

  1. Current versions of Facter will look into /etc/facter/facts.d for files of type .txt, .json, or .yaml. If facter finds a text file, it will parse the file for key=value pairs and add the key as a new fact:
    [root@cookbook ~]# facter model
    ED-209
    
  2. If the file is a YAML or JSON file, then facter will parse the file for key=value pairs in the respective format. For YAML, for instance:
    ---
    registry: NCC-68814
    class: Andromeda
    shipname: USS Prokofiev
  3. The resulting output will be as follows:
    [root@cookbook ~]# facter registry class shipname
    class => Andromeda
    registry => NCC-68814
    shipname => USS Prokofiev
    
  4. In the case of executable files, Facter will assume that their output is a list of key=value pairs. It will execute all the files in the facts.d directory and add their output to the internal fact hash.
  5. In the users example, Facter will execute the users.sh script, which results in the following output:
    users=2
    
  6. It will then search this output for users and return the matching value:
    [root@cookbook ~]# facter users
    2
    
  7. If there are multiple matches for the key you specified, Facter determines which fact to return based on a weight property. In my version of facter, the weight of external facts is 10,000 (defined in facter/util/directory_loader.rb as EXTERNAL_FACT_WEIGHT). This high value is to ensure that the facts you define can override the supplied facts. For example:
    [root@cookbook ~]# facter architecture
    x86_64
    [root@cookbook ~]# echo "architecture=ppc64">>/etc/facter/facts.d/myfacts.txt
    [root@cookbook ~]# facter architecture
    ppc64
    

Since all external facts have a weight of 10,000, the order in which they are parsed within the /etc/facter/facts.d directory sets their precedence (with the last one encountered having the highest precedence). To create a fact that will be favored over another, you'll need to have it created in a file that comes last alphabetically:

There's more...

Since all external facts have a weight of 10,000, the order in which they are parsed within the /etc/facter/facts.d directory sets their precedence (with the last one encountered having the highest precedence). To create a fact that will be favored over another, you'll need to have it created in a file that comes last alphabetically:

[root@cookbook ~]# facter architecture ppc64 [root@cookbook ~]# echo "architecture=r10000" >>/etc/facter/facts.d/z-architecture.txt [root@cookbook ~]# facter architecture r10000
Debugging external facts

If you're having
Using external facts in Puppet

Any external facts you create See also

The Importing dynamic information recipe in
  • Chapter 3, Writing Better Manifests
  • The Configuring Hiera recipe in Chapter 2, Puppet Infrastructure
  • The Creating custom facts recipe in this chapter
How to do it...

In this example we'll

If you have a server that is already configured as it needs to be, or nearly so, you can capture that configuration as a Puppet manifest. The Puppet resource command generates Puppet manifests from the existing configuration of a system. For example, you can have puppet resource generate a manifest that creates all the users found on the system. This is very useful to take a snapshot of a working system and get its configuration quickly into Puppet.

How to do it...

Here are some examples of using puppet resource to get data from a running system:

To generate the manifest for a particular user, run the following command:
[root@cookbook ~]# puppet resource user thomas
user { 'thomas':
  ensure           => 'present',
  comment          => 'thomas Admin User',
  gid              => '1001',
  groups           => ['bin', 'wheel'],
  home             => '/home/thomas',
  password         => '!!',
  password_max_age => '99999',
  password_min_age => '0',
  shell            => '/bin/bash',
  uid              => '1001',
}
For a
There's more...

You can use puppet resource to examine each of the

If you want to quickly capture the complete configuration of a running system as a Puppet manifest, there are a couple of tools available to help. In this example, we'll look at Blueprint, which is designed to examine a machine and dump its state as Puppet code.

These steps will show you how to run Blueprint:

  1. Run the following commands:
    [root@cookbook ~]# mkdir blueprint && cd blueprint
    [root@cookbook blueprint]# blueprint create -P blueprint_test
    # [blueprint] searching for APT packages to exclude
    # [blueprint] searching for Yum packages to exclude
    # [blueprint] caching excluded Yum packages
    # [blueprint] parsing blueprintignore(5) rules
    # [blueprint] searching for npm packages
    # [blueprint] searching for configuration files
    # [blueprint] searching for APT packages
    # [blueprint] searching for PEAR/PECL packages
    # [blueprint] searching for Python packages
    # [blueprint] searching for Ruby gems
    # [blueprint] searching for software built from source
    # [blueprint] searching for Yum packages
    # [blueprint] searching for service dependencies
    blueprint_test/manifests/init.pp
    
  2. Read the blueprint_test/manifests/init.pp file to see the generated code:
    #
    # Automatically generated by blueprint(7).  Edit at your own risk.
    #
    class blueprint_test {
      Exec {
        path => '/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin',
      }
      Class['sources'] -> Class['files'] -> Class['packages']
        class files {
          file {
            '/etc':
              ensure => directory;
            '/etc/aliases.db':
    content => template('blueprint_test/etc/aliases.db'),
              ensure  => file,
    group   => root,
              mode    => 0644,
              owner   => root;
    '/etc/audit':
              ensure => directory;
    '/etc/audit/audit.rules':
              content => template('blueprint_test/etc/audit/audit.rules'),
              ensure  => file,
              group   => root,
              mode    => 0640,
              owner   => root;
            '/etc/blkid':
              ensure => directory;
    '/etc/cron.hourly':
              ensure => directory;
    '/etc/cron.hourly/run-backup':
              content => template('blueprint_test/etc/cron.hourly/run-backup'),
              ensure  => file,
              group   => root,
              mode    => 0755,
    owner   => root;
    '/etc/crypttab':
              content => template('blueprint_test/etc/crypttab'),
              ensure  => file,
              group   => root,
              mode    => 0644,
              owner   => root;

Blueprint just takes a snapshot of the system as it stands; it makes no intelligent decisions, and Blueprint captures all the files on the system and all the packages. It will generate a configuration much larger than you may actually require. For instance, when configuring a server, you may specify that you want the Apache package installed. The dependencies for the Apache package will be installed automatically and you need to specify them. When generating the configuration with a tool such as Blueprint, you will capture all those dependencies and lock the versions that are installed on your system currently. Looking at our generated Blueprint code, we can see that this is the case:

If you were creating this manifest yourself, you would likely specify ensure => installed instead of a specific version.

Packages install default versions of files. Blueprint has no notion of this and will add all the files to the manifest, even those that have not changed. By default, Blueprint will indiscriminately capture all the files in /etc as file resources.

Blueprint and similar tools have a very small use case generally, but may help you to get familiar with the Puppet syntax and give you some ideas on how to specify your own manifests. I would not recommend blindly using this tool to create a system, however.

There's no shortcut to good configuration management, those who hope to save time and effort by cutting and pasting someone else's code as a whole (as with public modules) are likely to find that it saves neither.

Getting ready

Here's what you need to do to prepare your system to use Blueprint.

Run the following command to install Blueprint; we'll use puppet resource here to change the state of the python-pip package:

[root@cookbook ~]# puppet resource package python-pip ensure=installed Notice: /Package[python-pip]/ensure: created package { 'python-pip': ensure => '1.3.1-4.el6', } [root@cookbook ~]# pip install blueprint Downloading/unpacking blueprint Downloading blueprint-3.4.2.tar.gz (59kB): 59kB downloaded Running setup.py egg_info for package blueprint Installing collected packages: blueprint Running setup.py install for blueprint changing mode of build/scripts-2.6/blueprint from 644 to 755 ... Successfully installed blueprint Cleaning up...

These steps will show you how to run Blueprint:

  1. Run the following commands:
    [root@cookbook ~]# mkdir blueprint && cd blueprint
    [root@cookbook blueprint]# blueprint create -P blueprint_test
    # [blueprint] searching for APT packages to exclude
    # [blueprint] searching for Yum packages to exclude
    # [blueprint] caching excluded Yum packages
    # [blueprint] parsing blueprintignore(5) rules
    # [blueprint] searching for npm packages
    # [blueprint] searching for configuration files
    # [blueprint] searching for APT packages
    # [blueprint] searching for PEAR/PECL packages
    # [blueprint] searching for Python packages
    # [blueprint] searching for Ruby gems
    # [blueprint] searching for software built from source
    # [blueprint] searching for Yum packages
    # [blueprint] searching for service dependencies
    blueprint_test/manifests/init.pp
    
  2. Read the blueprint_test/manifests/init.pp file to see the generated code:
    #
    # Automatically generated by blueprint(7).  Edit at your own risk.
    #
    class blueprint_test {
      Exec {
        path => '/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin',
      }
      Class['sources'] -> Class['files'] -> Class['packages']
        class files {
          file {
            '/etc':
              ensure => directory;
            '/etc/aliases.db':
    content => template('blueprint_test/etc/aliases.db'),
              ensure  => file,
    group   => root,
              mode    => 0644,
              owner   => root;
    '/etc/audit':
              ensure => directory;
    '/etc/audit/audit.rules':
              content => template('blueprint_test/etc/audit/audit.rules'),
              ensure  => file,
              group   => root,
              mode    => 0640,
              owner   => root;
            '/etc/blkid':
              ensure => directory;
    '/etc/cron.hourly':
              ensure => directory;
    '/etc/cron.hourly/run-backup':
              content => template('blueprint_test/etc/cron.hourly/run-backup'),
              ensure  => file,
              group   => root,
              mode    => 0755,
    owner   => root;
    '/etc/crypttab':
              content => template('blueprint_test/etc/crypttab'),
              ensure  => file,
              group   => root,
              mode    => 0644,
              owner   => root;

Blueprint just takes a snapshot of the system as it stands; it makes no intelligent decisions, and Blueprint captures all the files on the system and all the packages. It will generate a configuration much larger than you may actually require. For instance, when configuring a server, you may specify that you want the Apache package installed. The dependencies for the Apache package will be installed automatically and you need to specify them. When generating the configuration with a tool such as Blueprint, you will capture all those dependencies and lock the versions that are installed on your system currently. Looking at our generated Blueprint code, we can see that this is the case:

If you were creating this manifest yourself, you would likely specify ensure => installed instead of a specific version.

Packages install default versions of files. Blueprint has no notion of this and will add all the files to the manifest, even those that have not changed. By default, Blueprint will indiscriminately capture all the files in /etc as file resources.

Blueprint and similar tools have a very small use case generally, but may help you to get familiar with the Puppet syntax and give you some ideas on how to specify your own manifests. I would not recommend blindly using this tool to create a system, however.

There's no shortcut to good configuration management, those who hope to save time and effort by cutting and pasting someone else's code as a whole (as with public modules) are likely to find that it saves neither.

How to do it...

These steps will

show you how to run Blueprint:

  1. Run the following commands:
    [root@cookbook ~]# mkdir blueprint && cd blueprint
    [root@cookbook blueprint]# blueprint create -P blueprint_test
    # [blueprint] searching for APT packages to exclude
    # [blueprint] searching for Yum packages to exclude
    # [blueprint] caching excluded Yum packages
    # [blueprint] parsing blueprintignore(5) rules
    # [blueprint] searching for npm packages
    # [blueprint] searching for configuration files
    # [blueprint] searching for APT packages
    # [blueprint] searching for PEAR/PECL packages
    # [blueprint] searching for Python packages
    # [blueprint] searching for Ruby gems
    # [blueprint] searching for software built from source
    # [blueprint] searching for Yum packages
    # [blueprint] searching for service dependencies
    blueprint_test/manifests/init.pp
    
  2. Read the blueprint_test/manifests/init.pp file to see the generated code:
    #
    # Automatically generated by blueprint(7).  Edit at your own risk.
    #
    class blueprint_test {
      Exec {
        path => '/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin',
      }
      Class['sources'] -> Class['files'] -> Class['packages']
        class files {
          file {
            '/etc':
              ensure => directory;
            '/etc/aliases.db':
    content => template('blueprint_test/etc/aliases.db'),
              ensure  => file,
    group   => root,
              mode    => 0644,
              owner   => root;
    '/etc/audit':
              ensure => directory;
    '/etc/audit/audit.rules':
              content => template('blueprint_test/etc/audit/audit.rules'),
              ensure  => file,
              group   => root,
              mode    => 0640,
              owner   => root;
            '/etc/blkid':
              ensure => directory;
    '/etc/cron.hourly':
              ensure => directory;
    '/etc/cron.hourly/run-backup':
              content => template('blueprint_test/etc/cron.hourly/run-backup'),
              ensure  => file,
              group   => root,
              mode    => 0755,
    owner   => root;
    '/etc/crypttab':
              content => template('blueprint_test/etc/crypttab'),
              ensure  => file,
              group   => root,
              mode    => 0644,
              owner   => root;

Blueprint just takes a snapshot of the system as it stands; it makes no intelligent decisions, and Blueprint captures all the files on the system and all the packages. It will generate a configuration much larger than you may actually require. For instance, when configuring a server, you may specify that you want the Apache package installed. The dependencies for the Apache package will be installed automatically and you need to specify them. When generating the configuration with a tool such as Blueprint, you will capture all those dependencies and lock the versions that are installed on your system currently. Looking at our generated Blueprint code, we can see that this is the case:

If you were creating this manifest yourself, you would likely specify ensure => installed instead of a specific version.

Packages install default versions of files. Blueprint has no notion of this and will add all the files to the manifest, even those that have not changed. By default, Blueprint will indiscriminately capture all the files in /etc as file resources.

Blueprint and similar tools have a very small use case generally, but may help you to get familiar with the Puppet syntax and give you some ideas on how to specify your own manifests. I would not recommend blindly using this tool to create a system, however.

There's no shortcut to good configuration management, those who hope to save time and effort by cutting and pasting someone else's code as a whole (as with public modules) are likely to find that it saves neither.

There's more...

Blueprint

just takes a snapshot of the system as it stands; it makes no intelligent decisions, and Blueprint captures all the files on the system and all the packages. It will generate a configuration much larger than you may actually require. For instance, when configuring a server, you may specify that you want the Apache package installed. The dependencies for the Apache package will be installed automatically and you need to specify them. When generating the configuration with a tool such as Blueprint, you will capture all those dependencies and lock the versions that are installed on your system currently. Looking at our generated Blueprint code, we can see that this is the case:

If you were creating this manifest yourself, you would likely specify ensure => installed instead of a specific version.

Packages install default versions of files. Blueprint has no notion of this and will add all the files to the manifest, even those that have not changed. By default, Blueprint will indiscriminately capture all the files in /etc as file resources.

Blueprint and similar tools have a very small use case generally, but may help you to get familiar with the Puppet syntax and give you some ideas on how to specify your own manifests. I would not recommend blindly using this tool to create a system, however.

There's no shortcut to good configuration management, those who hope to save time and effort by cutting and pasting someone else's code as a whole (as with public modules) are likely to find that it saves neither.

When Puppet runs on a node, it needs to know which classes should be applied to that node. For example, if it is a web server node, it might need to include an apache class. The normal way to map nodes to classes is in the Puppet manifest itself, for example, in your site.pp file:

Alternatively, you can use an External Node Classifier (ENC) to do this job. An ENC is any executable program that can accept the fully-qualified domain name (FQDN) as the first command-line argument ($1). The script is expected to return a list of classes, parameters, and an optional environment to apply to the node. The output is expected to be in the standard YAML format. When using an ENC, you should keep in mind that the classes applied through the standard site.pp manifest are merged with those provided by the ENC.

An ENC could be a simple shell script, for example, or a wrapper around a more complicated program or API that can decide how to map nodes to classes. The ENC provided by Puppet enterprise and The Foreman (http://theforeman.org/) are both simple scripts, which connect to the web API of their respective systems.

In this example, we'll build the most simple of ENCs, a shell script that simply prints a list of classes to include. We'll start by including an enc class, which defines notify that will print a top-scope variable $enc.

Getting ready

We'll start by creating our enc class to include with the enc script:

Run the following command:
t@mylaptop ~/puppet $ mkdir -p modules/enc/manifests
Create the file modules/enc/manifests/init.pp with the following contents:
class enc {
  notify {"We defined this from $enc": }
}
How to do it...

Here's how to build a simple external node classifier. We'll perform all these steps on our Puppet master server. If you are running masterless, then do these steps on a node:

Create the file /etc/puppet/cookbook.sh with the following contents:
#!/bin/bash
cat <<EOF
---
classes:
enc:
parameters:
  enc: $0
EOF
Run the
How it works...

When an ENC is set in puppet.conf, Puppet will call the specified program with the node's fqdn (technically, the certname variable) as the first command-line argument. In our example script, this argument is ignored, and it just outputs a fixed list of classes (actually, just one class).

Obviously this script is not terribly useful; a more sophisticated script might check a database to find the class list, or look up the node in a hash, or an external text file or database (often an organization's There's more...

An ENC can supply a See also

See the puppetlabs ENC page

As you know, Puppet has a bunch of useful built-in resource types: packages, files, users, and so on. Usually, you can do everything you need to do by using either combinations of these built-in resources, or define, which you can use more or less in the same way as a resource (see Chapter 3, Writing Better Manifests for information on definitions).

In the early days of Puppet, creating your own resource type was more common as the list of core resources was shorter than it is today. Before you consider creating your own resource type, I suggest searching the Forge for alternative solutions. Even if you can find a project that only partially solves your problem, you will be better served by extending and helping out that project, rather than trying to create your own. However, if you need to create your own resource type, Puppet makes it quite easy. The native types are written in Ruby, and you will need a basic familiarity with Ruby in order to create your own.

Let's refresh our memory on the distinction between types and providers. A type describes a resource and the parameters it can have (for example, the package type). A provider tells Puppet how to implement a resource type for a particular platform or situation (for example, the apt/dpkg providers implement the package type for Debian-like systems).

A single type (package) can have many providers (APT, YUM, Fink, and so on). If you don't specify a provider when declaring a resource, Puppet will choose the most appropriate one given the environment.

We'll use Ruby in this section; if you are not familiar with Ruby try visiting http://www.ruby-doc.org/docs/Tutorial/ or http://www.codecademy.com/tracks/ruby/.

When deciding whether or not you should create a custom type, you should ask a few questions about the resource you are trying to describe such as:

How to do it...

In this section, we'll see how to create a custom type that we can use to manage Git repositories, and in the next section, we'll write a provider to implement this type.

Create the file modules/cookbook/lib/puppet/type/gitrepo.rb with the following contents:

Puppet::Type.newtype(:gitrepo) do ensurable newparam(:source) do isnamevar end newparam(:path) end

When deciding whether or not you should create a custom type, you should ask a few questions about the resource you are trying to describe such as:

How it works...

Custom types can

When deciding whether or not you should create a custom type, you should ask a few questions about the resource you are trying to describe such as:

There's more...

When deciding whether or not you should create a custom type, you should ask a few questions about the Documentation

Our example is deliberately simple, but when you move on to developing real custom types for your production environment, you should add documentation strings Validation

You can use parameter validation to generate useful error messages

In the previous section, we created a new custom type called gitrepo and told Puppet that it takes two parameters, source and path. However, so far, we haven't told Puppet how to actually check out the repo; in other words, how to create a specific instance of this type. That's where the provider comes in.

We saw that a type will often have several possible providers. In our example, there is only one sensible way to instantiate a Git repo, so we'll only supply one provider: git. If you were to generalize this type—to just repo, say—it's not hard to imagine creating several different providers depending on the type of repo, for example, git, svn, cvs, and so on.

We'll add the git provider, and create an instance of a gitrepo resource to check that it all works. You'll need Git installed for this to work, but if you're using the Git-based manifest management setup described in Chapter 2, Puppet Infrastructure, we can safely assume that Git is available.

  1. Create the file modules/cookbook/lib/puppet/provider/gitrepo/git.rb with the following contents:
    require 'fileutils'
    
    Puppet::Type.type(:gitrepo).provide(:git) do
      commands :git => "git"
    
      def create
        git "clone", resource[:source], resource[:path]
      end
    
      def exists?
        File.directory? resource[:path]
      end
    end
  2. Modify your site.pp file as follows:
    node 'cookbook' {
      gitrepo { 'https://github.com/puppetlabs/puppetlabs-git':
        ensure => present,
        path   => '/tmp/puppet',
      }
    }
  3. Run Puppet:
    [root@cookbook ~]# puppet agent -t
    Notice: /File[/var/lib/puppet/lib/puppet/type/gitrepo.rb]/ensure: defined content as '{md5}6471793fe2b4372d40289ad4b614fe0b'
    Notice: /File[/var/lib/puppet/lib/puppet/provider/gitrepo]/ensure: created
    Notice: /File[/var/lib/puppet/lib/puppet/provider/gitrepo/git.rb]/ensure: defined content as '{md5}f860388234d3d0bdb3b3ec98bbf5115b'
    Info: Caching catalog for cookbook.example.com
    Info: Applying configuration version '1416378876'
    Notice: /Stage[main]/Main/Node[cookbook]/Gitrepo[https://github.com/puppetlabs/puppetlabs-git]/ensure: created
    Notice: Finished catalog run in 2.59 seconds
    

Custom providers can live in any module, in a lib/puppet/provider/TYPE_NAME subdirectory in a file named after the provider. (The provider is the actual program that is run on the system; in our example, the program is Git and the provider is in modules/cookbook/lib/puppet/provider/gitrepo/git.rb. Note that the name of the module is irrelevant.)

After an ntitial require line in git.rb, we tell Puppet to register a new provider for the gitrepo type with the following line:

When you declare an instance of the gitrepo type in your manifest, Puppet will first of all check whether the instance already exists, by calling the exists? method on the provider. So we need to supply this method, complete with code to check whether an instance of the gitrepo type already exists:

This is not the most sophisticated implementation; it simply returns true if a directory exists matching the path parameter of the instance. A better implementation of exists? might check, for example, whether there is a .git subdirectory and that it contains valid Git metadata. But this will do for now.

If exists? returns true, then Puppet will take no further action because the specified resource exists (as far as Puppet knows). If it returns false, Puppet assumes the resource doesn't yet exist, and will try to create it by calling the provider's create method.

Accordingly, we supply some code for the create method that calls the git clone command to create the repo:

The method has access to the instance's parameters, which we need to know where to check out the repo from, and which directory to create it in. We get this by looking at resource[:source] and resource[:path].

How to do it...

We'll add the git provider, and create an instance of a gitrepo resource to check that it all works. You'll need Git installed for this to work, but if you're using the Git-based manifest management setup described in

Chapter 2, Puppet Infrastructure, we can safely assume that Git is available.

  1. Create the file modules/cookbook/lib/puppet/provider/gitrepo/git.rb with the following contents:
    require 'fileutils'
    
    Puppet::Type.type(:gitrepo).provide(:git) do
      commands :git => "git"
    
      def create
        git "clone", resource[:source], resource[:path]
      end
    
      def exists?
        File.directory? resource[:path]
      end
    end
  2. Modify your site.pp file as follows:
    node 'cookbook' {
      gitrepo { 'https://github.com/puppetlabs/puppetlabs-git':
        ensure => present,
        path   => '/tmp/puppet',
      }
    }
  3. Run Puppet:
    [root@cookbook ~]# puppet agent -t
    Notice: /File[/var/lib/puppet/lib/puppet/type/gitrepo.rb]/ensure: defined content as '{md5}6471793fe2b4372d40289ad4b614fe0b'
    Notice: /File[/var/lib/puppet/lib/puppet/provider/gitrepo]/ensure: created
    Notice: /File[/var/lib/puppet/lib/puppet/provider/gitrepo/git.rb]/ensure: defined content as '{md5}f860388234d3d0bdb3b3ec98bbf5115b'
    Info: Caching catalog for cookbook.example.com
    Info: Applying configuration version '1416378876'
    Notice: /Stage[main]/Main/Node[cookbook]/Gitrepo[https://github.com/puppetlabs/puppetlabs-git]/ensure: created
    Notice: Finished catalog run in 2.59 seconds
    

Custom providers can live in any module, in a lib/puppet/provider/TYPE_NAME subdirectory in a file named after the provider. (The provider is the actual program that is run on the system; in our example, the program is Git and the provider is in modules/cookbook/lib/puppet/provider/gitrepo/git.rb. Note that the name of the module is irrelevant.)

After an ntitial require line in git.rb, we tell Puppet to register a new provider for the gitrepo type with the following line:

When you declare an instance of the gitrepo type in your manifest, Puppet will first of all check whether the instance already exists, by calling the exists? method on the provider. So we need to supply this method, complete with code to check whether an instance of the gitrepo type already exists:

This is not the most sophisticated implementation; it simply returns true if a directory exists matching the path parameter of the instance. A better implementation of exists? might check, for example, whether there is a .git subdirectory and that it contains valid Git metadata. But this will do for now.

If exists? returns true, then Puppet will take no further action because the specified resource exists (as far as Puppet knows). If it returns false, Puppet assumes the resource doesn't yet exist, and will try to create it by calling the provider's create method.

Accordingly, we supply some code for the create method that calls the git clone command to create the repo:

The method has access to the instance's parameters, which we need to know where to check out the repo from, and which directory to create it in. We get this by looking at resource[:source] and resource[:path].

How it works...

Custom providers

can live in any module, in a lib/puppet/provider/TYPE_NAME subdirectory in a file named after the provider. (The provider is the actual program that is run on the system; in our example, the program is Git and the provider is in modules/cookbook/lib/puppet/provider/gitrepo/git.rb. Note that the name of the module is irrelevant.)

After an ntitial require line in git.rb, we tell Puppet to register a new provider for the gitrepo type with the following line:

When you declare an instance of the gitrepo type in your manifest, Puppet will first of all check whether the instance already exists, by calling the exists? method on the provider. So we need to supply this method, complete with code to check whether an instance of the gitrepo type already exists:

This is not the most sophisticated implementation; it simply returns true if a directory exists matching the path parameter of the instance. A better implementation of exists? might check, for example, whether there is a .git subdirectory and that it contains valid Git metadata. But this will do for now.

If exists? returns true, then Puppet will take no further action because the specified resource exists (as far as Puppet knows). If it returns false, Puppet assumes the resource doesn't yet exist, and will try to create it by calling the provider's create method.

Accordingly, we supply some code for the create method that calls the git clone command to create the repo:

The method has access to the instance's parameters, which we need to know where to check out the repo from, and which directory to create it in. We get this by looking at resource[:source] and resource[:path].

There's more...

You can see that custom types and providers in Puppet are very powerful. In fact, they can do anything—at least, anything that Ruby can do. If you are managing some parts of your infrastructure with complicated define statements and exec resources, you may want to consider replacing these with a custom type. However, as stated previously, it's worth looking around to see if someone else has already done this before implementing your own.

Our example was very simple, and there is much more to learn about writing your own types. If you're going to distribute your code for others to use, or even if you aren't, it's a good idea to include tests with it. puppetlabs has a useful page on the interface between custom types

If you've read the recipe Using GnuPG to encrypt secrets in Chapter 4, Working with Files and Packages, then you've already seen an example of a custom function (in that example, we created a secret function, which shelled out to GnuPG). Let's look at custom functions in a little more detail now and build an example.

If you've read the recipe Distributing cron jobs efficiently in Chapter 6, Managing Resources and Files, you might remember that we used the inline_template function to set a random time for cron jobs to run, based on the hostname of the node. In this example, we'll take that idea and turn it into a custom function called random_minute:

  1. Create the file modules/cookbook/lib/puppet/parser/functions/random_minute.rb with the following contents:
    module Puppet::Parser::Functions
      newfunction(:random_minute, :type => :rvalue) do |args|
        lookupvar('hostname').sum % 60
      end
    end
  2. Modify your site.pp file as follows:
    node 'cookbook' {
      cron { 'randomised cron job':
        command => '/bin/echo Hello, world >>/tmp/hello.txt',
        hour    => '*',
        minute  => random_minute(),
      }
    }
  3. Run Puppet:
    [root@cookbook ~]# puppet agent -t
    Info: Retrieving pluginfacts
    Info: Retrieving plugin
    Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/random_minute.rb]/ensure: defined content as '{md5}e6ff40165e74677e5837027bb5610744'
    Info: Loading facts
    Info: Caching catalog for cookbook.example.com
    Info: Applying configuration version '1416379652'
    Notice: /Stage[main]/Main/Node[cookbook]/Cron[custom fuction example job]/ensure: created
    Notice: Finished catalog run in 0.41 seconds
    
  4. Check crontab with the following command:
    [root@cookbook ~]# crontab -l
    # HEADER: This file was autogenerated at Wed Nov 19 01:48:11 -0500 2014 by puppet.
    # HEADER: While it can still be managed manually, it is definitely not recommended.
    # HEADER: Note particularly that the comments starting with 'Puppet Name' should
    # HEADER: not be deleted, as doing so could cause duplicate cron jobs.
    # Puppet Name: run-backup
    0 15 * * * /usr/local/bin/backup
    # Puppet Name: custom fuction example job
    15 * * * * /bin/echo Hallo, welt >>/tmp/hallo.txt
    

You can, of course, do a lot more with custom functions. In fact, anything you can do in Ruby, you can do in a custom function. You also have access to all the facts and variables that are in scope at the point in the Puppet manifest where the function is called, by calling lookupvar as shown in the example. You can also work on arguments, for example, a general purpose hashing function that takes two arguments: the size of the hash table and optionally the thing to hash. Create modules/cookbook/lib/puppet/parser/functions/hashtable.rb with the following contents:

Now we'll create a test for our hashtable function and alter site.pp as follows:

Now, run Puppet and observe the values that are returned:

Our simple definition quickly grew when we added the ability to add arguments. As with all programming, care should be taken when working with arguments to ensure that you do not have any error conditions. In the preceding code, we specifically looked for the situation where the size variable was 0, to avoid a divide by zero error.

To find out more about what you can do with custom functions, see the puppetlabs website:

http://docs.puppetlabs.com/guides/custom_functions.html

How to do it...

If you've read the

recipe Distributing cron jobs efficiently in Chapter 6, Managing Resources and Files, you might remember that we used the inline_template function to set a random time for cron jobs to run, based on the hostname of the node. In this example, we'll take that idea and turn it into a custom function called random_minute:

  1. Create the file modules/cookbook/lib/puppet/parser/functions/random_minute.rb with the following contents:
    module Puppet::Parser::Functions
      newfunction(:random_minute, :type => :rvalue) do |args|
        lookupvar('hostname').sum % 60
      end
    end
  2. Modify your site.pp file as follows:
    node 'cookbook' {
      cron { 'randomised cron job':
        command => '/bin/echo Hello, world >>/tmp/hello.txt',
        hour    => '*',
        minute  => random_minute(),
      }
    }
  3. Run Puppet:
    [root@cookbook ~]# puppet agent -t
    Info: Retrieving pluginfacts
    Info: Retrieving plugin
    Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/random_minute.rb]/ensure: defined content as '{md5}e6ff40165e74677e5837027bb5610744'
    Info: Loading facts
    Info: Caching catalog for cookbook.example.com
    Info: Applying configuration version '1416379652'
    Notice: /Stage[main]/Main/Node[cookbook]/Cron[custom fuction example job]/ensure: created
    Notice: Finished catalog run in 0.41 seconds
    
  4. Check crontab with the following command:
    [root@cookbook ~]# crontab -l
    # HEADER: This file was autogenerated at Wed Nov 19 01:48:11 -0500 2014 by puppet.
    # HEADER: While it can still be managed manually, it is definitely not recommended.
    # HEADER: Note particularly that the comments starting with 'Puppet Name' should
    # HEADER: not be deleted, as doing so could cause duplicate cron jobs.
    # Puppet Name: run-backup
    0 15 * * * /usr/local/bin/backup
    # Puppet Name: custom fuction example job
    15 * * * * /bin/echo Hallo, welt >>/tmp/hallo.txt
    

You can, of course, do a lot more with custom functions. In fact, anything you can do in Ruby, you can do in a custom function. You also have access to all the facts and variables that are in scope at the point in the Puppet manifest where the function is called, by calling lookupvar as shown in the example. You can also work on arguments, for example, a general purpose hashing function that takes two arguments: the size of the hash table and optionally the thing to hash. Create modules/cookbook/lib/puppet/parser/functions/hashtable.rb with the following contents:

Now we'll create a test for our hashtable function and alter site.pp as follows:

Now, run Puppet and observe the values that are returned:

Our simple definition quickly grew when we added the ability to add arguments. As with all programming, care should be taken when working with arguments to ensure that you do not have any error conditions. In the preceding code, we specifically looked for the situation where the size variable was 0, to avoid a divide by zero error.

To find out more about what you can do with custom functions, see the puppetlabs website:

http://docs.puppetlabs.com/guides/custom_functions.html

How it works...

Custom

You can, of course, do a lot more with custom functions. In fact, anything you can do in Ruby, you can do in a custom function. You also have access to all the facts and variables that are in scope at the point in the Puppet manifest where the function is called, by calling lookupvar as shown in the example. You can also work on arguments, for example, a general purpose hashing function that takes two arguments: the size of the hash table and optionally the thing to hash. Create modules/cookbook/lib/puppet/parser/functions/hashtable.rb with the following contents:

Now we'll create a test for our hashtable function and alter site.pp as follows:

Now, run Puppet and observe the values that are returned:

Our simple definition quickly grew when we added the ability to add arguments. As with all programming, care should be taken when working with arguments to ensure that you do not have any error conditions. In the preceding code, we specifically looked for the situation where the size variable was 0, to avoid a divide by zero error.

To find out more about what you can do with custom functions, see the puppetlabs website:

http://docs.puppetlabs.com/guides/custom_functions.html

There's more...

You can, of course, do a lot more with custom functions. In fact, anything you can do in Ruby, you can do in a custom function. You also have access to all the facts and variables that are in scope at the point in the Puppet manifest where the function is called, by calling lookupvar as shown in the example. You can also work on arguments, for example, a general purpose hashing function that takes two arguments: the size of the hash table and optionally the thing to hash. Create modules/cookbook/lib/puppet/parser/functions/hashtable.rb with the following contents:

module Puppet::Parser::Functions newfunction(:hashtable, :type => :rvalue) do |args| if args.length == 2 hashtable=lookupvar(args[1]).sum else hashtable=lookupvar('hostname').sum end if args.length > 0 size = args[0].to_i else size = 60 end unless size == 0 hashtable % size else 0 end end end

Now we'll create a

It would be great if we could verify that our Puppet manifests satisfy certain expectations without even having to run Puppet. The rspec-puppet tool is a nifty tool to do this. Based on RSpec, a testing framework for Ruby programs, rspec-puppet lets you write test cases for your Puppet manifests that are especially useful to catch regressions (bugs introduced when fixing another bug), and refactoring problems (bugs introduced when reorganizing your code).

Let's create an example class, thing, and write some tests for it.

  1. Define the thing class:
    class thing {
      service {'thing':
        ensure  => 'running',
        enable  => true,
        require => Package['thing'],
      }
      package {'thing':
        ensure => 'installed'
      }
      file {'/etc/thing.conf':
        content => 'fubar\n',
        mode    => 0644,
        require => Package['thing'],
        notify  => Service['thing'],
      }
    }
  2. Run the following commands:
    t@mylaptop ~/puppet]$cd modules/thing
    t@mylaptop~/puppet/modules/thing $ rspec-puppet-init
     + spec/
     + spec/classes/
     + spec/defines/
     + spec/functions/
     + spec/hosts/
     + spec/fixtures/
     + spec/fixtures/manifests/
     + spec/fixtures/modules/
     + spec/fixtures/modules/heartbeat/
     + spec/fixtures/manifests/site.pp
     + spec/fixtures/modules/heartbeat/manifests
     + spec/fixtures/modules/heartbeat/templates
     + spec/spec_helper.rb
     + Rakefile
    
  3. Create the file spec/classes/thing_spec.rb with the following contents:
    require 'spec_helper'
    
    describe 'thing' do
      it { should create_class('thing') }
      it { should contain_package('thing') }
      it { should contain_service('thing').with(
        'ensure' => 'running'
      ) }
      it { should contain_file('/etc/things.conf') }
    end
  4. Run the following commands:
    t@mylaptop ~/.puppet/modules/thing $ rspec
    ...F
    
    Failures:
    
      1) thing should contain File[/etc/things.conf]
         Failure/Error: it { should contain_file('/etc/things.conf') }
           expected that the catalogue would contain File[/etc/things.conf]
         # ./spec/classes/thing_spec.rb:9:in `block (2 levels) in <top (required)>'
    
    Finished in 1.66 seconds
    4 examples, 1 failure
    
    Failed examples:
    
    rspec ./spec/classes/thing_spec.rb:9 # thing should contain File[/etc/things.conf]
    

The rspec-puppet-init command creates a framework of directories for you to put your specs (test programs) in. At the moment, we're just interested in the spec/classes directory. This is where you'll put your class specs, one per class, named after the class it tests, for example, thing_spec.rb.

The spec code itself begins with the following statement, which sets up the RSpec environment to run the specs:

Then, a describe block follows:

The describe identifies the class we're going to test (thing) and wraps the list of assertions about the class inside a do .. end block.

Assertions are our stated expectations of the thing class. For example, the first assertion is the following:

The create_class assertion is used to ensure that the named class is actually created. The next line:

The contain_package assertion means what it says: the class should contain a package resource named thing.

Next, we test for the existence of the thing service:

The preceding code actually contains two assertions. First, that the class contains a thing service:

Second, that the service has an ensure attribute with the value running:

You can specify any attributes and values you want using the with method, as a comma-separated list. For example, the following code asserts several attributes of a file resource:

In our thing example, we need to only test that the file thing.conf is present, using the following code:

When you run the rake spec command, rspec-puppet will compile the relevant Puppet classes, run all the specs it finds, and display the results:

As you can see, we defined the file in our test as /etc/things.conf but the file in the manifests is /etc/thing.conf, so the test fails. Edit thing_spec.rb and change /etc/things.conf to /etc/thing.conf:

Now run rspec again:

Getting ready

Here's what you'll need to do to install rspec-puppet.

Run the following commands:

t@mylaptop~ $ sudo puppet resource package rspec-puppet ensure=installed provider=gem Notice: /Package[rspec-puppet]/ensure: created package { 'rspec-puppet': ensure => ['1.0.1'], } t@mylaptop ~ $ sudo puppet resource package puppetlabs_spec_helper ensure=installed provider=gem Notice: /Package[puppetlabs_spec_helper]/ensure: created package { 'puppetlabs_spec_helper': ensure => ['0.8.2'], }

Let's create an example class, thing, and write some tests for it.

  1. Define the thing class:
    class thing {
      service {'thing':
        ensure  => 'running',
        enable  => true,
        require => Package['thing'],
      }
      package {'thing':
        ensure => 'installed'
      }
      file {'/etc/thing.conf':
        content => 'fubar\n',
        mode    => 0644,
        require => Package['thing'],
        notify  => Service['thing'],
      }
    }
  2. Run the following commands:
    t@mylaptop ~/puppet]$cd modules/thing
    t@mylaptop~/puppet/modules/thing $ rspec-puppet-init
     + spec/
     + spec/classes/
     + spec/defines/
     + spec/functions/
     + spec/hosts/
     + spec/fixtures/
     + spec/fixtures/manifests/
     + spec/fixtures/modules/
     + spec/fixtures/modules/heartbeat/
     + spec/fixtures/manifests/site.pp
     + spec/fixtures/modules/heartbeat/manifests
     + spec/fixtures/modules/heartbeat/templates
     + spec/spec_helper.rb
     + Rakefile
    
  3. Create the file spec/classes/thing_spec.rb with the following contents:
    require 'spec_helper'
    
    describe 'thing' do
      it { should create_class('thing') }
      it { should contain_package('thing') }
      it { should contain_service('thing').with(
        'ensure' => 'running'
      ) }
      it { should contain_file('/etc/things.conf') }
    end
  4. Run the following commands:
    t@mylaptop ~/.puppet/modules/thing $ rspec
    ...F
    
    Failures:
    
      1) thing should contain File[/etc/things.conf]
         Failure/Error: it { should contain_file('/etc/things.conf') }
           expected that the catalogue would contain File[/etc/things.conf]
         # ./spec/classes/thing_spec.rb:9:in `block (2 levels) in <top (required)>'
    
    Finished in 1.66 seconds
    4 examples, 1 failure
    
    Failed examples:
    
    rspec ./spec/classes/thing_spec.rb:9 # thing should contain File[/etc/things.conf]
    

The rspec-puppet-init command creates a framework of directories for you to put your specs (test programs) in. At the moment, we're just interested in the spec/classes directory. This is where you'll put your class specs, one per class, named after the class it tests, for example, thing_spec.rb.

The spec code itself begins with the following statement, which sets up the RSpec environment to run the specs:

Then, a describe block follows:

The describe identifies the class we're going to test (thing) and wraps the list of assertions about the class inside a do .. end block.

Assertions are our stated expectations of the thing class. For example, the first assertion is the following:

The create_class assertion is used to ensure that the named class is actually created. The next line:

The contain_package assertion means what it says: the class should contain a package resource named thing.

Next, we test for the existence of the thing service:

The preceding code actually contains two assertions. First, that the class contains a thing service:

Second, that the service has an ensure attribute with the value running:

You can specify any attributes and values you want using the with method, as a comma-separated list. For example, the following code asserts several attributes of a file resource:

In our thing example, we need to only test that the file thing.conf is present, using the following code:

When you run the rake spec command, rspec-puppet will compile the relevant Puppet classes, run all the specs it finds, and display the results:

As you can see, we defined the file in our test as /etc/things.conf but the file in the manifests is /etc/thing.conf, so the test fails. Edit thing_spec.rb and change /etc/things.conf to /etc/thing.conf:

Now run rspec again:

How to do it...

Let's create an example class, thing, and write some tests for it.

Define the thing class:
class thing {
  service {'thing':
    ensure  => 'running',
    enable  => true,
    require => Package['thing'],
  }
  package {'thing':
    ensure => 'installed'
  }
  file {'/etc/thing.conf':
    content => 'fubar\n',
    mode    => 0644,
    require => Package['thing'],
    notify  => Service['thing'],
  }
}
Run the

The rspec-puppet-init command creates a framework of directories for you to put your specs (test programs) in. At the moment, we're just interested in the spec/classes directory. This is where you'll put your class specs, one per class, named after the class it tests, for example, thing_spec.rb.

The spec code itself begins with the following statement, which sets up the RSpec environment to run the specs:

Then, a describe block follows:

The describe identifies the class we're going to test (thing) and wraps the list of assertions about the class inside a do .. end block.

Assertions are our stated expectations of the thing class. For example, the first assertion is the following:

The create_class assertion is used to ensure that the named class is actually created. The next line:

The contain_package assertion means what it says: the class should contain a package resource named thing.

Next, we test for the existence of the thing service:

The preceding code actually contains two assertions. First, that the class contains a thing service:

Second, that the service has an ensure attribute with the value running:

You can specify any attributes and values you want using the with method, as a comma-separated list. For example, the following code asserts several attributes of a file resource:

In our thing example, we need to only test that the file thing.conf is present, using the following code:

When you run the rake spec command, rspec-puppet will compile the relevant Puppet classes, run all the specs it finds, and display the results:

As you can see, we defined the file in our test as /etc/things.conf but the file in the manifests is /etc/thing.conf, so the test fails. Edit thing_spec.rb and change /etc/things.conf to /etc/thing.conf:

Now run rspec again:

How it works...

The rspec-puppet-init command creates

a framework of directories for you to put your specs (test programs) in. At the moment, we're just interested in the spec/classes directory. This is where you'll put your class specs, one per class, named after the class it tests, for example, thing_spec.rb.

The spec code itself begins with the following statement, which sets up the RSpec environment to run the specs:

Then, a describe block follows:

The describe identifies the class we're going to test (thing) and wraps the list of assertions about the class inside a do .. end block.

Assertions are our stated expectations of the thing class. For example, the first assertion is the following:

The create_class assertion is used to ensure that the named class is actually created. The next line:

The contain_package assertion means what it says: the class should contain a package resource named thing.

Next, we test for the existence of the thing service:

The preceding code actually contains two assertions. First, that the class contains a thing service:

Second, that the service has an ensure attribute with the value running:

You can specify any attributes and values you want using the with method, as a comma-separated list. For example, the following code asserts several attributes of a file resource:

In our thing example, we need to only test that the file thing.conf is present, using the following code:

When you run the rake spec command, rspec-puppet will compile the relevant Puppet classes, run all the specs it finds, and display the results:

As you can see, we defined the file in our test as /etc/things.conf but the file in the manifests is /etc/thing.conf, so the test fails. Edit thing_spec.rb and change /etc/things.conf to /etc/thing.conf:

Now run rspec again:

There's more...

There are many conditions you can verify with rspec. Any resource type can be verified with contain_<resource type>(title). In addition to verifying your classes will apply correctly, you can also test functions and definitions by using the appropriate subdirectories within the spec directory (classes, defines, or functions).

You can find more information about rspec-puppet, including complete documentation for the assertions available

and a tutorial, at http://rspec-puppet.com/.

When you want to start testing how your code applies to nodes, you'll need to look at another tool, beaker. Beaker works with various virtualization platforms to create temporary virtual machines to which Puppet code is applied. The results are then used for acceptance testing of the Puppet code. This method of testing and developing at the same time is known as Test-driven development (TDD). More information about beaker is available on the GitHub site at https://github.com/puppetlabs/beaker.

See also

The Checking your manifests with puppet-lint recipe in

When you begin to include modules from the forge in your Puppet infrastructure, keeping track of which versions you installed and ensuring consistency between all your testing areas can become a bit of a problem. Luckily, the tools we will discuss in the next two sections can bring order to your system. We will first begin with librarian-puppet, which uses a special configuration file named Puppetfile to specify the source location of your various modules.

The Puppetfile allows you to pull in modules from sources other than the forge. You may use a local Git url or even a GitHub url to download modules that are not on the Forge. More information on librarian-puppet can be found on the GitHub website at https://github.com/rodjek/librarian-puppet.

Note that librarian-puppet will create the modules directory and remove any modules you placed in there by default. Most installations using librarian-puppet opt to place their local modules in a /local subdirectory (/dist or /companyname are also used).

In the next section, we'll talk about r10k, which goes one step further than librarian and manages your entire environment directory.

Getting ready

We'll install librarian-puppet to work through the example.

Install librarian-puppet on your Puppet master, using Puppet of course:

root@puppet:~# puppet resource package librarian-puppet ensure=installed provider=gem Notice: /Package[librarian-puppet]/ensure: created package { 'librarian-puppet': ensure => ['2.0.0'], }

The Puppetfile allows you to pull in modules from sources other than the forge. You may use a local Git url or even a GitHub url to download modules that are not on the Forge. More information on librarian-puppet can be found on the GitHub website at https://github.com/rodjek/librarian-puppet.

Note that librarian-puppet will create the modules directory and remove any modules you placed in there by default. Most installations using librarian-puppet opt to place their local modules in a /local subdirectory (/dist or /companyname are also used).

In the next section, we'll talk about r10k, which goes one step further than librarian and manages your entire environment directory.

How to do it...

We'll use librarian-puppet to download and install a module in this example:

Create a working directory for yourself; librarian-puppet will overwrite your modules directory by default, so we'll work in a temporary location for now:
root@puppet:~# mkdir librarian
root@puppet:~# cd librarian
Create a new Puppetfile with the following contents:
#!/usr/bin/env ruby
#^syntax detection

forge "https://forgeapi.puppetlabs.com"

# A module from the Puppet Forge
mod 'puppetlabs-stdlib'

The Puppetfile allows you to pull in modules from sources other than the forge. You may use a local Git url or even a GitHub url to download modules that are not on the Forge. More information on librarian-puppet can be found on the GitHub website at https://github.com/rodjek/librarian-puppet.

Note that librarian-puppet will create the modules directory and remove any modules you placed in there by default. Most installations using librarian-puppet opt to place their local modules in a /local subdirectory (/dist or /companyname are also used).

In the next section, we'll talk about r10k, which goes one step further than librarian and manages your entire environment directory.

How it works...

The first line of

The Puppetfile allows you to pull in modules from sources other than the forge. You may use a local Git url or even a GitHub url to download modules that are not on the Forge. More information on librarian-puppet can be found on the GitHub website at https://github.com/rodjek/librarian-puppet.

Note that librarian-puppet will create the modules directory and remove any modules you placed in there by default. Most installations using librarian-puppet opt to place their local modules in a /local subdirectory (/dist or /companyname are also used).

In the next section, we'll talk about r10k, which goes one step further than librarian and manages your entire environment directory.

There's more...

The Puppetfile allows

you to pull in modules from sources other than the forge. You may use a local Git url or even a GitHub url to download modules that are not on the Forge. More information on librarian-puppet can be found on the GitHub website at https://github.com/rodjek/librarian-puppet.

Note that librarian-puppet will create the modules directory and remove any modules you placed in there by default. Most installations using librarian-puppet opt to place their local modules in a /local subdirectory (/dist or /companyname are also used).

In the next section, we'll talk about r10k, which goes one step further than librarian and manages your entire environment directory.

The Puppetfile is a very good format to describe which modules you wish to include in your environment. Building upon the Puppetfile is another tool, r10k. r10k is a total environment management tool. You can use r10k to clone a local Git repository into your environmentpath and then place the modules specified in your Puppetfile into that directory. The local Git repository is known as the master repository; it is where r10k expects to find your Puppetfile. r10k also understands Puppet environments and will clone Git branches into subdirectories of your environmentpath, simplifying the deployment of multiple environments. What makes r10k particularly useful is its use of a local cache directory to speed up deployments. Using a configuration file, r10k.yaml, you can specify where to store this cache and also where your master repository is held.

We'll install r10k on our controlling machine (usually the master). This is where we will control all the modules downloaded and installed.

  1. Install r10k on your puppet master, or on whichever machine you wish to manage your environmentpath directory:
    root@puppet:~# puppet resource package r10k ensure=installed provider=gem
    Notice: /Package[r10k]/ensure: created
    package { 'r10k':
      ensure => ['1.3.5'],
    }
    
  2. Make a new copy of your Git repository (optional, do this on your Git server):
    [git@git repos]$ git clone --bare puppet.git puppet-r10k.git
    Initialized empty Git repository in /home/git/repos/puppet-r10k.git/
    
  3. Check out the new Git repository (on your local machine) and move the existing modules directory to a new location. We'll use /local in this example:
    t@mylaptop ~ $ git clone [email protected]:repos/puppet-r10k.git
    Cloning into 'puppet-r10k'...
    remote: Counting objects: 2660, done.
    remote: Compressing objects: 100% (2136/2136), done.
    remote: Total 2660 (delta 913), reused 1049 (delta 238)
    Receiving objects: 100% (2660/2660), 738.20 KiB | 0 bytes/s, done.
    Resolving deltas: 100% (913/913), done.
    Checking connectivity... done.
    t@mylaptop ~ $ cd puppet-r10k/
    t@mylaptop ~/puppet-r10k $ git checkout production
    Branch production set up to track remote branch production from origin.
    Switched to a new branch 'production'
    t@mylaptop ~/puppet-r10k $ git mv modules local
    t@mylaptop ~/puppet-r10k $ git commit -m "moving modules in preparation for r10k"
    [master c96d0dc] moving modules in preparation for r10k
     9 files changed, 0 insertions(+), 0 deletions(-)
     rename {modules => local}/base (100%)
     rename {modules => local}/puppet/files/papply.sh (100%)
     rename {modules => local}/puppet/files/pull-updates.sh (100%)
     rename {modules => local}/puppet/manifests/init.pp (100%)
    

We'll create a Puppetfile to control r10k and install modules on our master.

  1. Create a Puppetfile into the new Git repository with the following contents:
    forge "http://forge.puppetlabs.com"
    mod 'puppetlabs/puppetdb', '3.0.0'
    mod 'puppetlabs/stdlib', '3.2.0'
    mod 'puppetlabs/concat'
    mod 'puppetlabs/firewall'
  2. Add the Puppetfile to your new repository:
    t@mylaptop ~/puppet-r10k $ git add Puppetfile
    t@mylaptop ~/puppet-r10k $ git commit -m "adding Puppetfile"
    [production d42481f] adding Puppetfile
     1 file changed, 7 insertions(+)
     create mode 100644 Puppetfile
    t@mylaptop ~/puppet-r10k $ git push
    Counting objects: 7, done.
    Delta compression using up to 4 threads.
    Compressing objects: 100% (5/5), done.
    Writing objects: 100% (5/5), 589 bytes | 0 bytes/s, done.
    Total 5 (delta 2), reused 0 (delta 0)
    To [email protected]:repos/puppet-r10k.git
       cf8dfb9..d42481f  production -> production
    
  3. Back to your master, create /etc/r10k.yaml with the following contents:
    ---
    :cachedir: '/var/cache/r10k'
    :sources:
     :plops:
      remote: '[email protected]:repos/puppet-r10k.git'
      basedir: '/etc/puppet/environments'
    
  4. Run r10k to have the /etc/puppet/environments directory populated (hint: create a backup of your /etc/puppet/environments directory first):
    root@puppet:~# r10k deploy environment -p
    
  5. Verify that your /etc/puppet/environments directory has a production subdirectory. Within that directory, the /local directory will exist and the modules directory will have all the modules listed in the Puppetfile:
    root@puppet:/etc/puppet/environments# tree -L 2
    .
    ├── master
    │   ├── manifests
    │   ├── modules
    │   └── README
    └── production
        ├── environment.conf
        ├── local
        ├── manifests
        ├── modules
        ├── Puppetfile
        └── README
    
Getting ready

We'll install r10k

on our controlling machine (usually the master). This is where we will control all the modules downloaded and installed.

  1. Install r10k on your puppet master, or on whichever machine you wish to manage your environmentpath directory:
    root@puppet:~# puppet resource package r10k ensure=installed provider=gem
    Notice: /Package[r10k]/ensure: created
    package { 'r10k':
      ensure => ['1.3.5'],
    }
    
  2. Make a new copy of your Git repository (optional, do this on your Git server):
    [git@git repos]$ git clone --bare puppet.git puppet-r10k.git
    Initialized empty Git repository in /home/git/repos/puppet-r10k.git/
    
  3. Check out the new Git repository (on your local machine) and move the existing modules directory to a new location. We'll use /local in this example:
    t@mylaptop ~ $ git clone [email protected]:repos/puppet-r10k.git
    Cloning into 'puppet-r10k'...
    remote: Counting objects: 2660, done.
    remote: Compressing objects: 100% (2136/2136), done.
    remote: Total 2660 (delta 913), reused 1049 (delta 238)
    Receiving objects: 100% (2660/2660), 738.20 KiB | 0 bytes/s, done.
    Resolving deltas: 100% (913/913), done.
    Checking connectivity... done.
    t@mylaptop ~ $ cd puppet-r10k/
    t@mylaptop ~/puppet-r10k $ git checkout production
    Branch production set up to track remote branch production from origin.
    Switched to a new branch 'production'
    t@mylaptop ~/puppet-r10k $ git mv modules local
    t@mylaptop ~/puppet-r10k $ git commit -m "moving modules in preparation for r10k"
    [master c96d0dc] moving modules in preparation for r10k
     9 files changed, 0 insertions(+), 0 deletions(-)
     rename {modules => local}/base (100%)
     rename {modules => local}/puppet/files/papply.sh (100%)
     rename {modules => local}/puppet/files/pull-updates.sh (100%)
     rename {modules => local}/puppet/manifests/init.pp (100%)
    

We'll create a Puppetfile to control r10k and install modules on our master.

  1. Create a Puppetfile into the new Git repository with the following contents:
    forge "http://forge.puppetlabs.com"
    mod 'puppetlabs/puppetdb', '3.0.0'
    mod 'puppetlabs/stdlib', '3.2.0'
    mod 'puppetlabs/concat'
    mod 'puppetlabs/firewall'
  2. Add the Puppetfile to your new repository:
    t@mylaptop ~/puppet-r10k $ git add Puppetfile
    t@mylaptop ~/puppet-r10k $ git commit -m "adding Puppetfile"
    [production d42481f] adding Puppetfile
     1 file changed, 7 insertions(+)
     create mode 100644 Puppetfile
    t@mylaptop ~/puppet-r10k $ git push
    Counting objects: 7, done.
    Delta compression using up to 4 threads.
    Compressing objects: 100% (5/5), done.
    Writing objects: 100% (5/5), 589 bytes | 0 bytes/s, done.
    Total 5 (delta 2), reused 0 (delta 0)
    To [email protected]:repos/puppet-r10k.git
       cf8dfb9..d42481f  production -> production
    
  3. Back to your master, create /etc/r10k.yaml with the following contents:
    ---
    :cachedir: '/var/cache/r10k'
    :sources:
     :plops:
      remote: '[email protected]:repos/puppet-r10k.git'
      basedir: '/etc/puppet/environments'
    
  4. Run r10k to have the /etc/puppet/environments directory populated (hint: create a backup of your /etc/puppet/environments directory first):
    root@puppet:~# r10k deploy environment -p
    
  5. Verify that your /etc/puppet/environments directory has a production subdirectory. Within that directory, the /local directory will exist and the modules directory will have all the modules listed in the Puppetfile:
    root@puppet:/etc/puppet/environments# tree -L 2
    .
    ├── master
    │   ├── manifests
    │   ├── modules
    │   └── README
    └── production
        ├── environment.conf
        ├── local
        ├── manifests
        ├── modules
        ├── Puppetfile
        └── README
    
How to do it...

We'll create a Puppetfile to control r10k and install modules on our master.

Create a Puppetfile into the new Git repository with the following contents:
forge "http://forge.puppetlabs.com"
mod 'puppetlabs/puppetdb', '3.0.0'
mod 'puppetlabs/stdlib', '3.2.0'
mod 'puppetlabs/concat'
mod 'puppetlabs/firewall'
Add the Puppetfile to your new repository:
t@mylaptop ~/puppet-r10k $ git add Puppetfile
t@mylaptop ~/puppet-r10k $ git commit -m "adding Puppetfile"
[production d42481f] adding Puppetfile
 1 file changed, 7 insertions(+)
 create mode 100644 Puppetfile
t@mylaptop ~/puppet-r10k $ git push
Counting objects: 7, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (5/5), done.
Writing objects: 100% (5/5), 589 bytes | 0 bytes/s, done.
Total 5 (delta 2), reused 0 (delta 0)
To [email protected]:repos/puppet-r10k.git
   cf8dfb9..d42481f  production -> production
Back to
How it works...

We started by There's more...

You can automate the deployment of your environments using r10k. The command we used to run r10k and populate our environments directory can be easily placed inside a Git hook to automatically update your environment. There is also a marionette collective (mcollective) plugin

(https://github.com/acidprime/r10k), which can be used to have r10k run on an arbitrary set of servers.

Using either of these tools will help keep your site consistent, even if you are not taking advantage of the various modules available on the Forge.