Book Image

Apache Mesos Essentials

By : Dharmesh Kakadia
Book Image

Apache Mesos Essentials

By: Dharmesh Kakadia

Overview of this book

<p>Apache Mesos is a cluster manager that provides efficient resource isolation and sharing across distributed applications, or frameworks. It allows developers to concurrently run the likes of Hadoop, Spark, Storm, and other applications on a dynamically shared pool of nodes. With Mesos, you have the power to manage a wide range of resources in a multi-tenant environment.</p> <p>Starting with the basics, this book will give you an insight into all the features that Mesos has to offer. You will first learn how to set up Mesos in various environments from data centers to the cloud. You will then learn how to implement self-managed Platform as a Service environment with Mesos using various service schedulers, such as Chronos, Aurora, and Marathon. You will then delve into the depths of Mesos fundamentals and learn how to build distributed applications using Mesos primitives.</p> <p>Finally, you will round things off by covering the operational aspects of Mesos including logging, monitoring, high availability, and recovery.</p>
Table of Contents (15 chapters)
Apache Mesos Essentials
Credits
About the Author
About the Reviewers
www.PacktPub.com
Preface
Index

Running test frameworks


Mesos includes various example test frameworks written in C++, Java, and Python. They can be used to verify that the cluster is configured properly. The following test framework is written in C++, and it runs five sample applications. We will run it using the following command:

ubuntu@master:~/mesos/build/src $ ./test-framework --master=master:5050
I1228 08:53:13.303910  6044 sched.cpp:137] Version: 0.21.0
I1228 08:53:13.312556  6065 sched.cpp:234] New master detected at master@master
:5050
I1228 08:53:13.313287  6065 sched.cpp:242] No credentials provided. Attempting to register without authentication
I1228 08:53:13.316956  6061 sched.cpp:408] Framework registered with 20141228-085231-251789322-5050-5407-0001
Registered!
Received offer 20141228-085231-251789322-5050-5407-O3 with mem(*):1961; disk(*):35164; ports(*):[31000-32000]; cpus(*):2
Launching task 0 using offer 20141228-085231-251789322-5050-5407-O3
Launching task 1 using offer 20141228-085231-251789322-5050-5407-O3
Task 0 is in state TASK_RUNNING
Task 0 is in state TASK_FINISHED
Task 1 is in state TASK_RUNNING
Task 1 is in state TASK_FINISHED
Received offer 20141228-085231-251789322-5050-5407-O4 with mem(*):1961; disk(*):35164; ports(*):[31000-32000]; cpus(*):2
Launching task 2 using offer 20141228-085231-251789322-5050-5407-O4
Launching task 3 using offer 20141228-085231-251789322-5050-5407-O4
Task 2 is in state TASK_RUNNING
Task 2 is in state TASK_FINISHED
Task 3 is in state TASK_RUNNING
Task 3 is in state TASK_FINISHED
Received offer 20141228-085231-251789322-5050-5407-O5 with mem(*):1961; disk(*):35164; ports(*):[31000-32000]; cpus(*):2
Launching task 4 using offer 20141228-085231-251789322-5050-5407-O5
Task 4 is in state TASK_RUNNING
Task 4 is in state TASK_FINISHED
I1228 08:53:15.337805  6059 sched.cpp:1286] Asked to stop the driver
I1228 08:53:15.338147  6059 sched.cpp:752] Stopping framework '20141228-085231-251789322-5050-5407-0001'
I1228 08:53:15.338543  6044 sched.cpp:1286] Asked to stop the driver

Here the output shows the framework connected to the master and receives the resource offers from the master. It also shows the various states of the tasks it has launched. The Java example framework is included in the src/example/java folder:

ubuntu@master:~/mesos/build/src/examples/java $ ./test-framework master:5050
I1228 08:54:39.290570  7224 sched.cpp:137] Version: 0.21.0
I1228 08:54:39.302083  7250 sched.cpp:234] New master detected at master@master:5050
I1228 08:54:39.302613  7250 sched.cpp:242] No credentials provided. Attempting to register without authentication
I1228 08:54:39.307786  7250 sched.cpp:408] Framework registered with 20141228-085231-251789322-5050-5407-0002
Registered! ID = 20141228-085231-251789322-5050-5407-0002
Received offer 20141228-085231-251789322-5050-5407-O6 with cpus: 2.0 and mem: 1961.0
Launching task 0 using offer 20141228-085231-251789322-5050-5407-O6
Launching task 1 using offer 20141228-085231-251789322-5050-5407-O6
Status update: task 1 is in state TASK_RUNNING
Status update: task 0 is in state TASK_RUNNING
Status update: task 1 is in state TASK_FINISHED
Finished tasks: 1
Status update: task 0 is in state TASK_FINISHED
Finished tasks: 2
Received offer 20141228-085231-251789322-5050-5407-O7 with cpus: 2.0 and mem: 1961.0
Launching task 2 using offer 20141228-085231-251789322-5050-5407-O7
Launching task 3 using offer 20141228-085231-251789322-5050-5407-O7
Status update: task 2 is in state TASK_RUNNING
Status update: task 2 is in state TASK_FINISHED
Finished tasks: 3
Status update: task 3 is in state TASK_RUNNING
Status update: task 3 is in state TASK_FINISHED
Finished tasks: 4
Received offer 20141228-085231-251789322-5050-5407-O8 with cpus: 2.0 and mem: 1961.0
Launching task 4 using offer 20141228-085231-251789322-5050-5407-O8
Status update: task 4 is in state TASK_RUNNING
Status update: task 4 is in state TASK_FINISHED
Finished tasks: 5
I1228 08:54:41.788455  7248 sched.cpp:1286] Asked to stop the driver
I1228 08:54:41.788652  7248 sched.cpp:752] Stopping framework '20141228-085231-251789322-5050-5407-0002'
I1228 08:54:41.789008  7224 sched.cpp:1286] Asked to stop the driver

Similarly, the Python example framework is included in the src/example/python folder and shows frameworkId and the various tasks states:

ubuntu@master:~/mesos/build/src/examples/python $./test-framework master:5050
I1228 08:55:52.389428  8516 sched.cpp:137] Version: 0.21.0
I1228 08:55:52.422859  8562 sched.cpp:234] New master detected at master@master:5050
I1228 08:55:52.424178  8562 sched.cpp:242] No credentials provided. Attempting to register without authentication
I1228 08:55:52.428395  8562 sched.cpp:408] Framework registered with 20141228-085231-251789322-5050-5407-0003
Registered with framework ID 20141228-085231-251789322-5050-5407-0003
Received offer 20141228-085231-251789322-5050-5407-O9 with cpus: 2.0 and mem: 1961.0
Launching task 0 using offer 20141228-085231-251789322-5050-5407-O9
Launching task 1 using offer 20141228-085231-251789322-5050-5407-O9
Task 0 is in state TASK_RUNNING
Task 1 is in state TASK_RUNNING
Task 0 is in state TASK_FINISHED
Received message: 'data with a \x00 byte'
Task 1 is in state TASK_FINISHED
Received message: 'data with a \x00 byte'
Received offer 20141228-085231-251789322-5050-5407-O10 with cpus: 2.0 and mem: 1961.0
Launching task 2 using offer 20141228-085231-251789322-5050-5407-O10
Launching task 3 using offer 20141228-085231-251789322-5050-5407-O10
Task 2 is in state TASK_RUNNING
Task 2 is in state TASK_FINISHED
Task 3 is in state TASK_RUNNING
Task 3 is in state TASK_FINISHED
Received message: 'data with a \x00 byte'
Received message: 'data with a \x00 byte'
Received offer 20141228-085231-251789322-5050-5407-O11 with cpus: 2.0 and mem: 1961.0
Launching task 4 using offer 20141228-085231-251789322-5050-5407-O11
Task 4 is in state TASK_RUNNING
Task 4 is in state TASK_FINISHED
All tasks done, waiting for final framework message
Received message: 'data with a \x00 byte'
All tasks done, and all messages received, exiting
I1228 08:55:54.136085  8561 sched.cpp:1286] Asked to stop the driver
I1228 08:55:54.136147  8561 sched.cpp:752] Stopping framework '20141228-085231-251789322-5050-5407-0003'
I1228 08:55:54.136261  8516 sched.cpp:1286] Asked to stop the driver