Book Image

Apache Mesos Essentials

By : Dharmesh Kakadia
Book Image

Apache Mesos Essentials

By: Dharmesh Kakadia

Overview of this book

<p>Apache Mesos is a cluster manager that provides efficient resource isolation and sharing across distributed applications, or frameworks. It allows developers to concurrently run the likes of Hadoop, Spark, Storm, and other applications on a dynamically shared pool of nodes. With Mesos, you have the power to manage a wide range of resources in a multi-tenant environment.</p> <p>Starting with the basics, this book will give you an insight into all the features that Mesos has to offer. You will first learn how to set up Mesos in various environments from data centers to the cloud. You will then learn how to implement self-managed Platform as a Service environment with Mesos using various service schedulers, such as Chronos, Aurora, and Marathon. You will then delve into the depths of Mesos fundamentals and learn how to build distributed applications using Mesos primitives.</p> <p>Finally, you will round things off by covering the operational aspects of Mesos including logging, monitoring, high availability, and recovery.</p>
Table of Contents (15 chapters)
Apache Mesos Essentials
Credits
About the Author
About the Reviewers
www.PacktPub.com
Preface
Index

Tuning Spark on Mesos


Spark running on Mesos shares resources with other frameworks running on Mesos. It is important to understand the trade-offs of resource sharing and tuning Spark on Mesos for realizing the maximum benefits without sacrificing the application requirements. While running Spark on the Mesos cluster, resource sharing happens in two modes:

  • Fine-grain: By default, Spark will run each task as a separate Mesos task. This means that cluster resources are shared with the other frameworks running alongside Spark (or other instances of Spark itself). In this fine-grained mode, Mesos can share cluster resources among all applications in an elastic manner, but this elasticity comes at the cost of a higher overhead when launching tasks. This overhead is not significant for most applications but can be noticeable for applications with stringent timing requirements.

    Tip

    Mesos version below 0.21 had a bug in which the Spark application would not make progress while running in fine-grained...