Book Image

Instant Puppet 3 Starter

By : Jo Rhett
Book Image

Instant Puppet 3 Starter

By: Jo Rhett

Overview of this book

<p>Puppet is the tool that will save you time. This book teaches you how to do more with less using Puppet 3. This guide ensures the highest level of accuracy so everything is exactly the way you want it, every time. This Starter guide demonstrates the difference between deterministic and procedural results. Most importantly, it teaches you tricks for you to get better results every time, by thinking about and expressing your desired outcome in a deterministic fashion."Instant Puppet 3 Starter" provides you with all the information that you need, from startup to complete confidence in its use. This book will explore and teach the core components of Puppet, consisting of setting up a working client and server and building your first custom module. Become the Puppet master. Explore how it works and be in awe of the drastic improvement in consistency of your systems, with minimal effort in maintenance. Instant Puppet 3 Starter enables you to write your first policy using core methods to reduce the amount of manual work you would do to set up clients on new systems. In addition you will build a test environment for developing new modules, and source external data for use in the Puppet policy. Finally, you will learn to run the Puppet server under Phusion Passenger to improve performance and scalability. Instant Puppet 3 Starter won't just introduce you to an application; it will provide you with a working environment that saves you time and effort when deploying code or synchronizing files across systems.</p>
Table of Contents (6 chapters)

Top 5 features you need to know about


In this section we will discuss five ways to extend and enhance Puppet in your environment. We'll show you how to review the changes Puppet has made, get a list of backups of files, use environments to test the changes, improve the Puppet server's performance, and use data from other applications in the Puppet policy.

Reviewing system changes

The easiest way to review the history of changes made to the system is to examine the Puppet log. This is normally in the messages log from syslog, but you can also enable direct file logging in the puppet.conf configuration. If you are just looking for the recent changes on a client system, the following is likely all you need:

$ grep puppet-agent /var/log/messages

You can also have the Puppet agent send a report to the server. The server can store or process these reports. Let's see how to set that up now. The first step is to enable the sending of reports to the server. Add the following line to /etc/puppet/puppet.conf:

[agent]
  report = true

Now we need to tell the server what to do with the reports. Here is an example setting for the server:

[master]
  reports = tagmail, store

This tells the server to store the Puppet report and to give it to a report processor named tagmail. You can also create your own report processor and list it here.

File report store

If you enable the store method, the puppet reports from each client node will be stored in a YAML format on the disk. Assuming that you used the paths recommended in this book, the directory /var/lib/puppet-server/reports/hostname will contain one YAML file for each Puppet run. These will build up over time. If you enable this, you should also create a cron job to remove the older reports. Here's an example that Puppet can enforce:

cron { 'clean-up-reports':
  ensure  => present,
  command => 'find /var/lib/puppet-server/reports –type f –name \*.yaml –mtime +21 –delete', 
  user    => puppet,
  weekday => 0,
  hour    => 2,
  minute  => 11,
}

Tagmail report processor

It is useful when starting to send reports to the tagmail report processor. This allows you to send messages about changes to systems to people in e-mail. The message content can be based on either the log level of the message, or the tag applied to the resource. Here are some examples that I find useful. You define which messages should go to which e-mail addresses in the /etc/puppet/tagmail.conf file. Following are some useful examples:

# Send every log message at every level to me
all: [email protected]

# Send critical messages to my pager
alert,emergy,crit: [email protected]

# Send log messages from the apache module to the webmaster
apache,!info,!debug: [email protected]

# Send every change notification or error to the operations team
notice,warning,err,alert,emergy,crit: [email protected]

Note

The syntax may remind you of syslog.conf; however, log levels are not inclusive of higher levels. If you put just info as a tag, you will receive info messages, but nothing from notice, alert, critical, and so on.

Tagmail is documented at http://docs.puppetlabs.com/references/latest/report.html#tagmail.

Puppet Dashboard

Puppet Dashboard is a product provided by Puppet Labs to store reports in a database and provides a web UI to review those changes. After you have installed and configured the Puppet Dashboard, you can enable it as a report processor in /etc/puppet/puppet.conf:

[master]
reports = tagmail, store, http
reporturl = http://dashboard.example.net/reports/upload

At the time of printing of this book, the Puppet Labs documentation for installing Dashboard looks very intimidating. If you are using RHEL 6 or CentOS 6 as we suggested with the Puppet Labs repositories, you can safely ignore the Installing Dependancies section of the installation and just install the dashboard as follows:

$ sudo yum install mysql-server puppet-dashboard

Then perform the configuration steps documented at http://docs.puppetlabs.com/dashboard/manual/1.2/bootstrapping.html#configuring-dashboard.

At the time of printing, the page doesn't mention that the Puppet Labs-provided packages include the appropriate startup scripts for you. You don't need to create your own as the page suggests:

$ sudo /sbin/service puppet-dashboard start
$ sudo /sbin/service puppet-dashboard-workers start
$ sudo /sbin/chkconfig puppet-dashboard on
$ sudo /sbin/chkconfig puppet-dashboard-workers on 

Puppet Dashboard won't work very well unless you run it under Passenger. This is well documented in the manual. We will discuss this in the Passenger section.

Note

The documentation expects you to run Dashboard on TCP port 3000. I find it easier to run Puppet Dashboard on port 80, since the users will be connecting to it with their browsers.

Custom report processors

You can build your own custom report processors to do anything you want with the provided data. The guidelines for writing your own custom report processors are available at http://docs.puppetlabs.com/guides/reporting.html.

Comparing and restoring files

When Puppet changes a file, it makes a backup of the file and stores it on the disk in the directory specified by the clientbucket configuration option. You can pull the old versions of files out of this directory at any time. Puppet includes a command to restore the files, but it doesn't provide a command to find them. Given the odd directory structure with the file contents stored by their MD5 checksum, this isn't trivial. Here's a good script for finding the Puppet backups of a given file:

#!/bin/bash 
FILE=$1
FOUND=0
CLIENTBUCKET=$(puppet config print clientbucketdir) 

echo "Searching for local backups of $FILE..." 
for backup in $(find $CLIENTBUCKET -type f -name paths \
  -exec grep -l $FILE {} \; |xargs -r ls -t); 
do 
  hash=$( basename $(dirname $backup)) 
  filename=$(< $backup ) 
  modify_time=$(stat --format '%y' $backup) 
  echo -e "$filename\t$hash\t$modify_time" 
  FOUND=$((FOUND+1)) 
done 

if [ $FOUND -gt 0 ]; then 
  if [ $FOUND -eq 1 ]; then 
	echo "1 backup was found." 
  elif [ $FOUND -gt 1 ]; then
	echo "$FOUND backups were found."
  fi
   echo "" 
   echo "To view a file: puppet filebucket get -b $CLIENTBUCKET <hash>"
   echo "To restore a file: puppet filebucket restore -b $CLIENTBUCKET /new/path <hash>"

else 
    echo "No previous versions of the $FILE were found."
fi

An exercise for the reader: use the file resource example from the Quick start section and replicate this script to all client nodes.

Using environments to test changes

Environments provide you with the ability to isolate pieces of code and only invoke them for systems specifically configured for that environment, or only when the environment is enabled by hand. I'll show you how you can use environments to extend what Puppet can accomplish.

The following configuration option will provide flexibility when loading modules. This configuration will first check to see if a module exists in an environment-specific directory, then fall back to the main module directory. The rest of the following examples assume that you have added this option and restarted the Puppet master:

[master]
modulepath = $confdir/env/$environment/modules:$confdir/modules

Testing your code

Once you are using puppet in production, you will want to avoid testing new modules on your production systems. Assuming you set the modulepath as listed in previous configuration, you can put new the modules you want to test in /etc/puppet/env/testing/modules. If you have a dedicated test system, you can put the following in the puppet.conf:

[agent]
 environment = testing

You can also invoke the puppet agent in the testing environment on a case-by-case basis, if you need to test a new module on a particular system:

$ sudo puppet agent --test --environment testing

Using this approach, you can test out changes to your modules prior to putting the new version in the main module path and affecting hundreds of client systems.

Release engineering

Another way to use environments is for modules that you do not want to run every time, but only upon request. For example, you may have a module that pushes out a software release.

Assuming you set the modulepath as listed before, you can put the new modules you want to test in /etc/puppet/env/release/modules. To perform the software release, you would invoke Puppet on that system in the 'release' environment.

$ sudo puppet agent --test --environment release

Running the Puppet server under Passenger

The Puppet master process is a WEBrick ruby application server. It will handle only two concurrent Puppet clients. In any environment with more than 50 Puppet clients before it gets overloaded. If you intend to have more than 50 clients, you should disable the Puppet master process and enable it under a more powerful Ruby on the Rails application server. In this section we will document how to make it run under the popular Passenger Rails application server.

First, you will need to install the Apache web server and Phusion Passenger for your platform. The platform-specific instructions are available at http://www.modrails.com/documentation/Users%20guide%20Apache.html.

If you are using a Red Hat or CentOS system, it could be as simple as this:

$ sudo rpm –i http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
$ sudo rpm –i http://passenger.stealthymonkeys.com/rhel/6/passenger-release.noarch.rpm
$ sudo yum install httpd rubygems mod_ssl mod_passenger

Configure the puppetmaster service as a Passenger application. We do this by creating a configuration file for Apache, which defines the Puppet master application, and a config.ru file that contains the Phusion Passenger application environment. All of these steps must be done as root:

$ sudo bash
# sed –e 's/squigley.namespace.at/puppet.example.net/' \
  -e 's#/etc/puppet/ssl#/var/lib/puppet-server/ssl#' \
  /usr/share/puppet/ext/rack/files/apache2.conf\
  > /etc/httpd/conf.d/puppetmaster.conf

# mkdir –p /etc/puppet/rack/public
# sed –e 's#/var/lib/puppet#/var/lib/puppet-server#' \
/usr/share/puppet/ext/rack/files/config.ru \
>/etc/puppet/rack/config.ru
# chown puppet /etc/puppet/rack/config.ru
# exit

The chown command shown the previous code snippet is essential. Passenger runs the application specified in the config.ru file as the user who owns the file. Failing to set the owner of this file correctly is more than 50 percent of the issues we see on the mailing list.

Now, stop the Puppet master service (and Puppet Dashboard if you have it) and start Apache Passenger.

$ sudo /sbin/service puppetmaster stop
Stopping puppetmaster:                [OK]
$ sudo /sbin/chkconfigpuppetmaster off
$ sudo /sbin/chkconfighttpd on
$ sudo /sbin/service httpd start
Starting httpd:                [OK]

At this point you can test your clients as before. Absolutely nothing should be different from a client/agent perspective.

To run Puppet Dashboard under Passenger, you would create a configuration file named /etc/httpd/conf.d/dashboard.conf, similar to the example file /usr/share/puppet-dashboard/ext/passenger/dashboard-vhost.conf. However, be aware that the first 12 lines duplicate the settings already present in the other files in this directory. I would start with the following file, and add more options from the example as necessary:

<VirtualHost *:80>
  ServerNamedashboard.example.net
  DocumentRoot /usr/share/puppet-dashboard/public/
  RailsBaseURI /
  <Directory /usr/share/puppet-dashboard/public/>
    Options None
    Order allow,deny
    allow from all
  </Directory>
  ErrorLog /var/log/httpd/dashboard.example.net_error.log
  LogLevel warn
  CustomLog /var/log/httpd/dashboard.example.net_access.log combined
  ServerSignature On</VirtualHost>

Then perform the following steps to make the change active:

$ sudo /sbin/service puppet-dashboard stop
Stopping Puppet Dashboard:           [OK]
$ sudo /sbin/chkconfig puppet-dashboard off
$ sudo rm /etc/httpd/conf.d/welcome.conf
$ sudo /sbin/service httpd restart
Stopping httpd:                      [OK]
Starting httpd:                      [OK]

Using external data in your Puppet policy

Do you already have a source of data about your hosts or users? You really wouldn't want to maintain that same data in your Puppet policy now, would you? In this section we will talk about how to import external data and use it in your Puppet policy.

The best way to source external data is through a component called Hiera. Since you can import any kind of data, this topic is worthy of its own book. But we will set up a very small but powerful example that you can easily expand on: providing a list of user accounts.

All examples in this section use the YAML data format. You can install modules to provide other data backends, like JSON or MySQL. Puppet 3 includes only two Hiera backends by default: YAML and Puppet. The Puppet backend allows you to lookup data from Puppet and return it with the YAML data in a single result set.

Configuring Hiera data location

First, we need to tell Puppet where to find Hiera data for this class. Assuming we would like to store the Heira input data in /etc/puppet/hiera, you should create the file /etc/puppet/hiera.yaml with the following input. Note that YAML is very exact in its syntax, so you need to type in every dash and every space in this text:

---
:backends:
  - yaml
:yaml:
  :datadir: '/etc/puppet/hiera'
:hierarchy:
  - %{::hostname} 
  - common

This file tells Puppet that the Hiera data format will be YAML, and the files will be in the /etc/puppet/hiera directory. Then it tells Puppet to first look for a file with the name of the client host (from the hostname fact) with .yaml appended to the end, then to look for a file named common.yaml.

Creating Hiera data

Next we should create the actual YAML data. Here is an example we could use for /etc/puppet/hiera/common.yaml:

---
managehomedirs: true
users: 
  root:
    uid: 0
    gid: 0
  jack:
    uid: 1002
    gid: 1002
  jill:
    uid: 1001
    gid: 1001

If we wanted to add the user john to the system named web1, we would create another file /etc/puppet/hiera/web1.yaml:

---
users:
  john:
    uid: 1003
    gid: 1003

Note

You might be asking yourself: Isn't this just another file I have to maintain? Yes, in our example, this file is a static text file. In real life, you would have this data be output in the YAML or JSON format from your other application. You can also create a custom Hiera backend to query the application directly for the data.

Using Hiera data in a Puppet module

Next we use a function to import data from Hiera into the Puppet module. Back in our theoretical users module, the import functions would look as follows:

$managehomedirs = hiera('managehomedirs')
$users          = hiera_hash('users')

The first function there will find a single value for managehomedirs. With the hierarchy we named in the preceding example, it will look first for a YAML file named for the hostname, then it will look in common.yaml. Since this value is defined is in the common file, managehomedirs will be set to true.

The second Hiera function will return a merged hash of every value for that name in the hierarchy. For any system other than web1, the array will contain root, jack, and jill. For the web1 system it will contain john, root, jack, and jill.

Now, here is a complete module for managing the users on your system. You'll note that this module includes no data. All data is sourced from Hiera, thus allowing Puppet to use another application as a source for policy data:

class 'users' {
  $managehomedirs = hiera('managehomedirs',true)
  $defaults       = { managehome => $managehomedirs }
  $userlist       = hiera_hash('users')
  create_resources( user, $userlist, $defaults )
}

This module is pretty straightforward:

  • We load in the value for managehomedirs. If we cannot find the value, we default to true. We then use this as the sole key in a hash of default values.

  • We load in every user defined in every backend in the hierarchy.

  • We invoke the createresources() function to create a user resource for each user defined in our previous Hiera data files.

Note

This is a very powerful module. If you have a thousand user entries in your YAML data, it will create a thousand user resources in the Puppet catalog.

This section has provided you with a quick snapshot into the power and flexibility provided by Hiera data lookups in Puppet. You can create extensive hierarchies for data lookup, and you can use multiple data backends to provide policy data to Puppet. This will allow you to create Puppet policies which are independent of the data and source all of your configuration data from another application you use.

Learning more

You can find more information about using Hiera at http://puppetlabs.com/blog/first-look-installing-and-using-hiera/.

You can find more information about the YAML data format at http://www.yaml.org/.