There are many tools available that can create multiple virtual machines, install predefined software and even manage the state of that software.
Supervisord is a process control system. It is a client-server system that allows its users to monitor and control a number of processes on Unix-like operating systems. For details, visit http://supervisord.org/.
The server piece of the supervisor is known as supervisord. It is responsible for starting child programs upon its own invocation, responding to commands from clients, restarting crashed or exited subprocesses, logging its subprocess stdout and stderr output, and generating and handling "events" corresponding to points in subprocess lifetimes. The server process uses a configuration file. This is typically located in /etc/supervisord.conf. This configuration file is a Windows-INI style config file. It is important to keep this file secure via proper filesystem permissions because it might contain decrypted usernames and passwords:
supervisorctl: The command-line client piece of the supervisor is known as supervisorctl. It provides a shell-like interface for the features provided by supervisord. From supervisorctl, a user can connect to different supervisord processes. They can get the status on the subprocesses controlled by, stop and start subprocesses of, and get lists of running processes of a supervisord. The command-line client talks to the server across a Unix domain socket or an Internet (TCP) socket. The server can assert that the user of a client should present authentication credentials before it allows them to use commands. The client process typically uses the same configuration file as the server, but any configuration file with a
[supervisorctl]section in it will work.Web server: A (sparse) web user interface with functionality comparable to supervisorctl may be accessed via a browser if you start supervisord against an Internet socket. Visit the server URL (for example,
http://localhost:9001/) to view and control the process status through the web interface after activating the configuration file's[inet_http_server]section.XML-RPC interface: The same HTTP server that serves the web UI serves up an XML-RPC interface that can be used to interrogate and control the supervisor and the programs it runs. See XML-RPC API Documentation.
Machines: Let's assume that we have two EC2 machines of IP addresses
172-31-19-62and172.31.36.23. We will install supervisord on both machines and later configure to decide what services of Storm would be running on each machine.Storm and Zookeeper setup: Let's run Zookeeper, Nimbus, supervisor, and the UI on machine
172.31.36.23and only the supervisor on172-31-19-62.
Here is the process of the Zookeeper server setup and configuration:
Download Zookeeper's latest version and extract it:
tar –xvf zookeeper-3.4.6.tar.gzConfigure
zoo.cfgin theconfdirectory to start Zookeeper in cluster mode.Zookeeper conf:
server.1=172.31.36.23:2888:3888 tickTime=2000 initLimit=10 syncLimit=5 # the directory where the snapshot is stored. dataDir=/home/ec2-user/zookeeper-3.4.6/tmp/zookeeper clientPort=2181
Make sure that the directory specified in
dataDiris created and the user has read and write permissions on it.Then, go to the Zookeeper
bindirectory and start thezookeeperserver using the following command:[ec2-user@ip-172-31-36-23 bin~]$ zkServer.sh start
Storm server setup and configuration:
Download Storm's latest version from the Apache Storm website and extract it:
tar –xvf apache-storm-0.9.5.tar.gzHere is the configuration of the Storm Nimbus machine as well as the slave (added/changed configuration only):
storm.zookeeper.servers: - "172.31.36.23" nimbus.host: "172.31.36.23" nimbus.childopts: "-Xmx1024m -Djava.net.preferIPv4Stack=true" ui.childopts: "-Xmx768m -Djava.net.preferIPv4Stack=true" supervisor.childopts: "-Djava.net.preferIPv4Stack=true" worker.childopts: "-Xmx768m -Djava.net.preferIPv4Stack=true" storm.local.dir: "/home/ec2-user/apache-storm-0.9.5/local" supervisor.slots.ports: - 6700 - 6701 - 6702 - 6703
It is possible to install supervisord by the following two ways:
We will follow the second method of installation, the one in which Internet access is not required. We will download all dependencies and supervisord, and copy it to the servers.
Supervisord [supervisor-3.1.3.tar.gz] requires the following dependencies to be installed:
setuptools(latest) from http://pypi.python.org/pypi/setuptoolselementtree(latest) from http://effbot.org/downloads#elementtree.elementtree-1.2-20040618.tar.gzmeld3-0.6.5.tar.gz
Let's install supervisord and the necessary dependencies on both machines, 172.31.36.23 and 172-31-19-62.
The following are the steps for installing the dependencies:
setuptools:Unzip the
.zipfile using this command:[ec2-user@ip-172-31-19-62 ~]$ tar -xvf setuptools-17.1.1.zipGo to the
setuptools-17.1.1directory and run the installation command withsudo:[ec2-user@ip-172-31-19-62 setuptools-17.1.1]$ sudo python setup.py install
storm.zookeeper.servers: - "172.31.36.23" nimbus.host: "172.31.36.23" nimbus.childopts: "-Xmx1024m -Djava.net.preferIPv4Stack=true" ui.childopts: "-Xmx768m -Djava.net.preferIPv4Stack=true" supervisor.childopts: "-Djava.net.preferIPv4Stack=true" worker.childopts: "-Xmx768m -Djava.net.preferIPv4Stack=true" storm.local.dir: "/home/ec2-user/apache-storm-0.9.5/local" supervisor.slots.ports: - 6700 - 6701 - 6702 - 6703
meld3:Extract the
.ts.gzfile using the following command:[ec2-user@ip-172-31-19-62 ~]$ tar -xvf meld3-0.6.5.tar.gzGo to the
meld3.-0.6.5directory and run this command:[ec2-user@ip-172-31-19-62 meld3-0.6.5]$ sudo pyth setup.py install
elementtree:
The following are the supervisord installations:
Extract
supervisor-3.1.3using this command:[ec2-user@ip-172-31-19-62 ~]$ tar -xvf supervisor-3.1.3.tar.gzGo to the
supervisor-3.1.3directory and run the following command:[ec2-user@ip-172-31-19-62 supervisor-3.1.3]$ sudo python setup.py install
Lets configure services on the 172.31.36.23 machine and assume that the supervisord installation is done as explained previously. Once supervisor is installed, you can build the supervisord.conf file to start the supervisord and supervisorctl commands:
Make the
supervisor.conffile. Put it into the/etcdirectory.We can refer get sample
supervisord.confusing the following command:[ec2-user@ip-172-31-36-23 ~]$ echo_supervisord_conf
Take a look at the supervisord.conf file:
[unix_http_server] file = /home/ec2-user/supervisor.sock chmod = 0777 [inet_http_server] ; inet (TCP) server disabled by default port=172.31.36.23:9001 ; (ip_address:port specifier, *:port for all iface) username=user ; (default is no username (open server)) password=123 ; (default is no password (open server)) [rpcinterface:supervisor] supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface [supervisord] logfile_backups=10 ; (num of main logfile rotation backups;default 10) logfile=/home/ec2-user/supervisord.log ; (main log file;default $CWD/supervisord.log) logfile_maxbytes=50MB ; (max main logfile bytes b4 rotation;default 50MB) pidfile=/home/ec2-user/supervisord.pid ; (supervisord pidfile;default supervisord.pid) nodaemon=false ; (start in foreground if true;default false) minfds=1024 ; (min. avail startup file descriptors;default 1024) [supervisorctl] ;serverurl = unix:///home/ec2-user/supervisor.sock serverurl=http://172.31.36.23:9001 ; use an http:// url to specify an inet socket ;username=chris ; should be same as http_username if set ;password=123 ; should be same as http_password if set [program:storm-nimbus] command=/home/ec2-user/apache-storm-0.9.5/bin/storm nimbus user=ec2-user autostart=false autorestart=false startsecs=10 startretries=999 log_stdout=true log_stderr=true stdout_logfile=/home/ec2-user/storm/logs/nimbus.out logfile_maxbytes=20MB logfile_backups=10 [program:storm-ui] command=/home/ec2-user/apache-storm-0.9.5/bin/storm ui user=ec2-user autostart=false autorestart=false startsecs=10 startretries=999 log_stdout=true log_stderr=true stdout_logfile=/home/ec2-user/storm/logs/ui.out logfile_maxbytes=20MB logfile_backups=10 [program:storm-supervisor] command=/home/ec2-user/apache-storm-0.9.5/bin/storm supervisor user=ec2-user autostart=false autorestart=false startsecs=10 startretries=999 log_stdout=true log_stderr=true stdout_logfile=/home/ec2-user/storm/logs/supervisor.out logfile_maxbytes=20MB logfile_backups=10
Start the supervisor server first:
[ec2-user@ip-172-31-36-23 ~] sudo /usr/bin/supervisord -c /etc/supervisord.confThen, start all processes using supervisorctl:
[ec2-user@ip-172-31-36-23 ~] sudo /usr/bin/supervisorctl -c /etc/supervisord.conf status storm-nimbus STOPPED Not started storm-supervisor STOPPED Not started storm-ui STOPPED Not started [ec2-user@ip-172-31-36-23 ~]$ sudo /usr/bin/supervisorctl -c /etc/supervisord.conf start all storm-supervisor: started storm-ui: started storm-nimbus: started [ec2-user@ip-172-31-36-23 ~]$ jps 14452 Jps 13315 QuorumPeerMain 14255 nimbus 14233 supervisor 14234 core [ec2-user@ip-172-31-36-23 ~]$
We can view the supervisord web UI and control processes on the browser. 52.11.193.108 is the public IP address of the 172-31-36-23 machine (http://52.11.193.108:9001):
Keep only the following services in the configuration file:
[unix_http_server] [rpcinterface:supervisor] [supervisord] [supervisorctl] [program:storm-supervisor]
After that, you can start the supervisor server and all processes using supervisorctl on 172-31-19-62 machine.