Communication between the nodes and the master in a Puppet environment is verified with a system of X.509 SSL certificates. The master operates as a certificate authority (CA) for the system, although you may specify another server to act as the CA. When the agent first runs on a node, there are several steps taken to set up the trust relationship between the node and the master, which are outlined as follows:
The agent contacts the master and downloads the CA certificate.
The agent generates a certificate for itself based on the
certnameconfiguration option, which is usually equivalent to the hostname of the node.The agent issues a certificate signing request (CSR) to the master, asking the master to sign its certificate.
The master may choose to sign the certificate (if automatic signing is configured) or the operator of the master may sign the certificate.
The agent will check back every 2 minutes by default (configurable with the
waitforcertoption) to check whether its certificate has been signed.Once it has been signed, the agent will download its signed certificate.
At this point, the trust relationship between the agent and master has been established. The subsequent Puppet runs will not have to perform these steps. These runs will have a different series of steps, which are outlined in the following list:
The agent contacts and informs the master about its operating environment. Environments are used to separate nodes into logical groups.
The master looks through all the available modules in the given environment and begins sending all the
/libsubdirectories of the modules to the agent via thepluginsyncmethod.Once all the plugins have been downloaded to the node, the node runs the
facterutility to generate a list of facts about the node. The agent then ships the fact list back to the master.If you have PuppetDB configured in your environment, the facts are shipped to PuppetDB, which will decide to either create new entries for the node (if this is the first time, facts have arrived for this node), or update the existing fact values.
At this point, the master will use the fact list to generate values that are needed to compile the catalog for the node. A catalog is the living document of how a node is configured. It consists of all the Puppet modules, classes, and resources, that will be applied to a node.
The master will compile the catalog. This is the process of verifying that it is possible to apply resources to a node in a consistent manner. Puppet will generate a graph, with all the resources as vertices. If the graph has any cycles, then the compilation will fail. A cycle is also known as a circular dependency. It means that there are resources within the catalog that require each other, or that are mutually dependent. A failed compilation results in the agent exiting with an error code.
If the catalog compiles successfully, the catalog is then shipped to the agent.
At this point, the Puppet agent's run begins. All the resources in the catalog are applied sequentially to the node. When troubleshooting, it is important to remember that although the master is capable of running several catalog compilations at once, the agent on a node must be single-threaded by definition.
The catalog will either apply without error to the node, which is called a clean run, or fail and output errors. If the agent has a clean run and does not change any files on the node, the exit code of the process will be
0. If the agent has a clean run and does make any changes, the exit code will be2. If the agent fails to apply all the resources to the node, which is called a failed run, the agent will exit with the exit code as either4(with no file changes) or6(with file changes).Facter will be run at the end of a successful or clean run. The fact values that have changed or the new facts will be sent to PuppetDB. Note that if PuppetDB cannot be updated, the agent will mark the run as unclean.
Depending on your configuration, the agent will then generate a report, which will be shipped to the
report_server. If reports are configured, then they will be sent for both failed and clean runs.Several files on the node will also be updated at this point. When troubleshooting, it is often useful to examine the contents of the following files:
The
classes.txtfile contains the list of classes in the catalogThe
resources.txtfile contains a list of resources in the catalogThe
state.yamlfile contains timestamps for various files on the system as well as scheduling informationThe
last_run_report.yamlfile contains all the log messages that would be output to the console or during the Puppet run (this may be overridden by the--logdestcommand-line argument)The
last_run_summary.yamlfile contains a summary of the last run, with a count of the resources that were changed, the time taken to complete the run, and additional metrics
Beyond initial communication issues between the agents and the master, the bulk of Puppet troubleshooting revolves around failed catalog compilation and application.
With an idea of how the agent runs, we can now look closely at how one can configure Puppet as we examine the puppet.conf file in the next section.