Once you have a base set of virtual machines that are ready for application provisioning, you need to install and configure the appropriate packages on each node.
Create a new project named
storm-puppetwith the following folder structure:
The entry point into the Puppet execution on the provisioned node is
site.pp. Create it in themanifestsfolder:node 'storm.nimbus' { $cluster = 'storm1' include storm::nimbus include storm::ui } node /storm.supervisor[1-9]/ { $cluster = 'storm1' include storm::supervisor } node /storm.zookeeper[1-9]/ { include storm::zoo }Next, you need to define the storm module. A module exists in the
modulesfolder and has its ownmanifestsandtemplatefolder structure, much as with the structure found at the root level of the Puppet project. Within the storm module, create the required manifests (modules/storm/manifests), starting with theinit.ppfile:class storm { include storm::install include storm::config }The installation of the Storm application is the same on each
stormnode; only the configurations are adjusted where required, via templating. Next create theinstall.ppfile, which will download the required binaries and install them:class storm::install { $BASE_URL="https://bitbucket.org/qanderson/storm-deb-packaging/downloads/" $ZMQ_FILE="libzmq0_2.1.7_amd64.deb" $JZMQ_FILE="libjzmq_2.1.7_amd64.deb" $STORM_FILE="storm_0.8.1_all.deb" package { "wget": ensure => latest } # call fetch for each file exec { "wget_storm": command => "/usr/bin/wget ${BASE_URL}${STORM_FILE}" } exec {"wget_zmq": command => "/usr/bin/wget ${BASE_URL}${ZMQ_FILE}" } exec { "wget_jzmq": command => "/usr/bin/wget ${BASE_URL}${JZMQ_FILE}" } #call package for each file package { "libzmq0": provider => dpkg, ensure => installed, source => "${ZMQ_FILE}", require => Exec['wget_zmq'] } #call package for each file package { "libjzmq": provider => dpkg, ensure => installed, source => "${JZMQ_FILE}", require => [Exec['wget_jzmq'],Package['libzmq0']] } #call package for each file package { "storm": provider => dpkg, ensure => installed, source => "${STORM_FILE}", require => [Exec['wget_storm'], Package['libjzmq']] } }Tip
The
installmanifest here assumes the existence of package, Debian packages, for Ubuntu. These were built using scripts and can be tweaked based on your requirements. The binaries and creation scripts can be found at https://bitbucket.org/qanderson/storm-deb-packaging.The installation consists of the following packages:
Storm
ZeroMQ: http://www.zeromq.org/
Java-ZeroMQ
The configuration of each node is done through the template-based generation of the configuration files. In the
stormmanifests, createconfig.pp:class storm::config { require storm::install include storm::params file { '/etc/storm/storm.yaml': require => Package['storm'], content => template('storm/storm.yaml.erb'), owner => 'root', group => 'root', mode => '0644' } file { '/etc/default/storm': require => Package['storm'], content => template('storm/default.erb'), owner => 'root', group => 'root', mode => '0644' } }All the
stormparameters are defined using Hiera, with the Hiera configuration invoked fromparams.ppin thestormmanifests:class storm::params { #_ STORM DEFAULTS _# $java_library_path = hiera_array('java_library_path', ['/usr/local/lib', '/opt/local/lib', '/usr/lib']) }Tip
Due to the sheer number of properties, the file has been concatenated. For the complete file, please refer to the Git repository at https://bitbucket.org/qanderson/storm-puppet/src.
Each class of node is then specified; here we will specify the
nimbusclass:class storm::nimbus { require storm::install include storm::config include storm::params # Install nimbus /etc/default storm::service { 'nimbus': start => 'yes', jvm_memory => $storm::params::nimbus_mem } }class storm::supervisor { require storm::install include storm::config include storm::params # Install supervisor /etc/default storm::service { 'supervisor': start => 'yes', jvm_memory => $storm::params::supervisor_mem } }class storm::ui { require storm::install include storm::config include storm::params # Install ui /etc/default storm::service { 'ui': start => 'yes', jvm_memory => $storm::params::ui_mem } }And finally, specify the
zooclass (for azookeepernode):class storm::zoo { package {['zookeeper','zookeeper-bin','zookeeperd']: ensure => latest, } }Once all the files have been created, initialize the Git repository and push it to bitbucket.org.
In order to actually run the provisioning, navigate to the
vagrant-storm-clusterfolder and run the following command:vagrant upIf you would like to
sshinto any of the nodes, simply specify the following command:vagrant ssh nimbusReplace
nimbuswith your required node name.
There are various patterns that can be applied when using Puppet. The simplest one is using a distributed model, whereby nodes provision themselves as opposed to a centralized model using Puppet Master. In the distributed model, updating server configuration simply requires that you update your provisioning manifests and push them to your central Git repository. The various nodes will then pull and apply this configuration. This can either be achieved through cron jobs, triggers, or through the use of a Continuous Delivery tool such as Jenkins, Bamboo, or Go. Provisioning in the development environment is explicitly invoked by Vagrant through the following command:
config.vm.provision :shell, :inline => "puppet apply /tmp/storm-puppet/manifests/site.pp --verbose --modulepath=/tmp/storm-puppet/modules/ --debug"
The manifest is then applied declaratively by the Puppet. Puppet is declarative, in that each language element specifies the desired state together with methods for getting there. This means that, when the system is already in the required state, that particular provisioning step will be skipped, together with the adverse effects of duplicate provisioning.
The storm-puppet project is therefore cloned onto the node and then the manifest is applied locally. Each node only applies provisioning for itself, based on the hostname specified in the site.pp manifest, for example:
node 'storm.nimbus' {
$cluster = 'storm1'
include storm::nimbus
include storm::ui
}In this case, the nimbus node will include the Hiera configurations for cluster1, and the installation for the nimbus and ui nodes will be performed. Any combination of classes can be included in the node definition, thus allowing the complete environment to be succinctly defined.