MCollective is the Puppet Labs orchestration engine. It is not just specific to Puppet and can be used alongside other configuration management systems such as Chef, Salt, and CFEngine. MCollective uses a message broker to facilitate communication between nodes. The preferred broker is ActiveMQ.

The various components of MCollective are outlined in the following diagram:

The important aspect of this structure is that when you query the collective, you don't query the nodes directly. The nodes are subscribed to a message queue, and you submit your query to the queue. Every subscribed node will receive the message. Each node will reply if it matches the selection criteria of the message. Otherwise, the node ignores the message.

Relying on a message broker has been a barrier to entry for most of the small shops, but using a broker solves a lot of scale-related problems. When you are debugging issues with MCollective, always start at the broker level.

Briefly, in MCollective terminology...