Index

A

• Access Control List (ACL)
  • about / NodeManager (NM), The capacity scheduler
• administrative tools
  • about / Administrative tools
  • commands / Administrative tools
  • generic options, supporting / Administrative tools
  / Administrative tools
• anagrams / Practical examples of MRv1 and MRv2
• Apache Giraph
  • about / Apache Giraph
  • URL / Apache Giraph
• Apache Hadoop 2.2.0
  • about / Journey – present and future
• Apache Samza
  • about / Apache Samza
  • Kafka / Apache Samza
  • Apache YARN / Apache Samza
  • ZooKeeper / Apache Samza
  • Kafka producer, writing / Writing a Kafka producer
  • hello-samza project, writing / Writing the hello-samza project
• Apache Samza, layers
  • processing layer / Apache Samza
  • streaming layer / Apache Samza
  • execution layer / Apache Samza
• Apache Slider
  • about / Journey – present and future
• Apache Software Foundation
  • about / Mesos
• Apache Spark
  • about / Apache Spark
  • features / Apache Spark
  • running, on YARN / Why run on YARN?
• Apache Tez
  • about / Apache Tez
  • URL / Apache Tez
• Application Context (AppContext) / The MapReduce ApplicationMaster
• ApplicationMaster
  • about / The MapReduce ApplicationMaster
• ApplicationMaster (AM) / ApplicationMaster (AM)
  • restarting / The MapReduce ApplicationMaster
  • writing / Writing the YARN ApplicationMaster
  • resposibilities / Responsibilities of the ApplicationMaster
  • failures / ApplicationMaster failures
• ApplicationMaster Launcher service
  • about / ResourceManager
• ApplicationMasterService
  • about / ResourceManager
• ApplicationsManager
  • about / ResourceManager

B

• backward compatibility, MRv2 APIs
  • about / Backward compatibility of MRv2 APIs
  • binary compatibility, of org.apache.hadoop.mapred APIs / Binary compatibility of org.apache.hadoop.mapred APIs
  • source compatibility, of org.apache.hadoop.mapred APIs / Source compatibility of org.apache.hadoop.mapred APIs
• Bulk Synchronous Parallel (BSP)
  • about / Apache Giraph

C

• capacity scheduler
  • about / The capacity scheduler, The capacity scheduler
  • benefits / The capacity scheduler
  • features / The capacity scheduler
  • configurations / Capacity scheduler configurations
• cluster scheduling architecture
  • about / Omega
• configuration parameters
  • about / The fully-distributed mode
• container
  • failures / Container failures
• container allocation
  • about / Container allocation
  • to application / Container allocation to the application
• container configurations
  • about / Container configurations
  • parameters / Container configurations
• ContainerExecutor
  • about / NodeManager (NM)
• ContainerManager
  • about / NodeManager (NM)
• Context Objects / Old and new MapReduce APIs
• Corona
  • about / Corona
  • and Facebook, differences / Corona
  • URL / Corona

D

• data-processing graphs (DAGs)
  • about / Apache Tez
• DataNodes (DN) / The fully-distributed mode
  • configuring / The fully-distributed mode
• Docker
  • about / Future features

E

• EcoSystem
  • web interfaces / Web interfaces of the Ecosystem

F

• Facebook
  • about / Corona
  • and Corona, differences / Corona
• Fair scheduler / The fair scheduler
  • about / The fair scheduler
  • configurations / Fair scheduler configurations
• FIFO scheduler / The FIFO (First In First Out) scheduler
  • about / The FIFO (First In First Out) scheduler
  • configurations / The FIFO (First In First Out) scheduler
• fully-distributed mode
  • about / The fully-distributed mode
  • HistoryServer / HistoryServer
  • slave files / Slave files

G

• Google Kubernetes
  • about / Future features
• grid
  • starting / Starting a grid

H

• Hadoop
  • URL / Software
  • YARN, using in / Understanding where YARN fits into Hadoop
• Hadoop 0.23
  • about / Journey – present and future
• Hadoop 1.x
  • about / A short introduction to Hadoop 1.x and MRv1
  • components / A short introduction to Hadoop 1.x and MRv1
• Hadoop 2 release
  • about / The Hadoop 2 release
• Hadoop and YARN cluster
  • operating / Operating Hadoop and YARN clusters
  • starting / Starting Hadoop and YARN clusters
  • stopping / Stopping Hadoop and YARN clusters
• Hadoop cluster
  • HDFS / A short introduction to Hadoop 1.x and MRv1
  • MapReduce / A short introduction to Hadoop 1.x and MRv1
• Hadoop On Demand (HOD) / Omega
• hello-samza project
  • writing / Writing the hello-samza project
  • properties / Writing the hello-samza project
  • grid, starting / Starting a grid
• HistoryServer / HistoryServer
• HOYA (HBase on YARN)
  • about / HOYA (HBase on YARN)
  • URL / HOYA (HBase on YARN)

K

• Kafka producer
  • writing / Writing a Kafka producer
• KOYA (Kafka on YARN)
  • about / KOYA (Kafka on YARN)
  • URL / KOYA (Kafka on YARN)

M

• MapReduce, YARN
  • about / YARN's MapReduce support
  • ApplicationMaster / The MapReduce ApplicationMaster
  • settings, example / Example YARN MapReduce settings
  • YARN applications, developing / Developing YARN applications
• MapReduce applications
  • YARN, compatible with / YARN's compatibility with MapReduce applications
• MapReduce job
  • configurations / MapReduce job configurations
  • properties / MapReduce job configurations
• MapReduce JobHistoryServer
  • settings / HistoryServer
• MapReduce project
  • End-user MapReduce API / MRv1 versus MRv2
  • MapReduce framework / MRv1 versus MRv2
  • MapReduce system / MRv1 versus MRv2
• Mesos
  • about / Mesos
  • and YARN, difference between / Mesos
  • URL / Mesos
• modern operating system, of Hadoop
  • YARN, used as / YARN as the modern operating system of Hadoop
• monolithic schedulers / Omega
• MRv1
  • about / A short introduction to Hadoop 1.x and MRv1
  • versus MRv2 / MRv1 versus MRv2
  • examples / Practical examples of MRv1 and MRv2, Running the job
• MRv2
  • versus MRv1 / MRv1 versus MRv2
  • examples / Practical examples of MRv1 and MRv2, Preparing the input file(s)

N

• NameNode (NN) / The fully-distributed mode
  • configuring / The fully-distributed mode
• new MapReduce API
  • about / Old and new MapReduce APIs
  • versus old MapReduce API / Old and new MapReduce APIs
• NodeHealthCheckerService
  • about / NodeManager (NM)
• NodeManager (NM) / NodeManager (NM)
  • configuring / The fully-distributed mode
  • parameters / The fully-distributed mode
• NodeManagers (NM) / The fully-distributed mode
• NodeStatusUpdater
  • about / NodeManager (NM)

O

• old MapReduce API
  • about / Old and new MapReduce APIs
  • versus new MapReduce API / Old and new MapReduce APIs
• Omega
  • about / Omega

P

• Pi example
  • running / Running a sample Pi example
• prerequisites, single-node installation
  • platform / Platform
  • softwares / Software
• prerequisites, Storm-YARN
  • Hadoop YARN, installing / Hadoop YARN should be installed
  • Apache ZooKeeper, installing / Apache ZooKeeper should be installed
• program names
  • aggregatewordcount / Running sample examples on YARN
  • aggregatewordhist / Running sample examples on YARN
  • bbp / Running sample examples on YARN
  • dbcount / Running sample examples on YARN
  • distbbp / Running sample examples on YARN
  • grep / Running sample examples on YARN
  • join / Running sample examples on YARN
  • multifilewc / Running sample examples on YARN
  • pentomino / Running sample examples on YARN
  • pi / Running sample examples on YARN
  • randomtextwriter / Running sample examples on YARN
  • randomwriter / Running sample examples on YARN
  • secondarysort / Running sample examples on YARN
  • sort / Running sample examples on YARN
  • sudoku / Running sample examples on YARN
  • teragen / Running sample examples on YARN
  • terasort / Running sample examples on YARN
  • teravalidate / Running sample examples on YARN
  • wordcount / Running sample examples on YARN
  • wordmean / Running sample examples on YARN
  • wordmedian / Running sample examples on YARN
  • wordstandarddeviation / Running sample examples on YARN
• pseudo-distributed mode / The pseudo-distributed mode
• push-based scheduling / Corona

R

• redesign idea
  • about / The redesign idea
  • MapReduce, limitations / Limitations of the classical MapReduce or Hadoop 1.x
  • Hadoop 1.x, limitations / Limitations of the classical MapReduce or Hadoop 1.x
• Red Hat OpenShift
  • about / Future features
• Red Hat Package Managers (RPMs) / The fully-distributed mode
• ResourceManager / ResourceManager
• ResourceManager (RM)
  • scheduler / ResourceManager
  • security / ResourceManager
  • RM Restart Phase I / Recent developments in YARN architecture
  • RM Restart Phase II / Recent developments in YARN architecture
  • about / The fully-distributed mode
  • configuring / The fully-distributed mode
  • parameters / The fully-distributed mode
  • failures / ResourceManager failures
• ResourceManager (RM), components
  • ApplicationManager / NodeManager (NM)
  • Scheduler / NodeManager (NM)

S

• scheduler architectures
  • monolithic schedulers / Omega
  • two-level schedulers / Omega
• single-node installation
  • about / Single-node installation
  • prerequisites / Prerequisites
  • starting / Starting with the installation
  • standalone mode (local mode) / The standalone mode (local mode)
  • pseudo-distributed mode / The pseudo-distributed mode
• slave files / Slave files
• standalone mode (local mode) / The standalone mode (local mode)
• Storm-Starter examples
  • building / Building and running Storm-Starter examples
  • running / Building and running Storm-Starter examples
• Storm-YARN
  • about / Storm-YARN
  • prerequisites / Prerequisites
  • setting up / Setting up Storm-YARN
  • storm.yaml configuration, obtaining / Getting the storm.yaml configuration of the launched Storm cluster
  • Storm-Starter examples, building / Building and running Storm-Starter examples
  • Storm-Starter examples, running / Building and running Storm-Starter examples
• storm.yaml configuration
  • obtaining / Getting the storm.yaml configuration of the launched Storm cluster

T

• two-level schedulers / Omega

W

• web GUI
  • YARN applications, monitoring with / Monitoring YARN applications with web GUI

Y

• YARN
  • used, as modern operating system of Hadoop / YARN as the modern operating system of Hadoop
  • design goals / What are the design goals for YARN
  • used, in Hadoop / Understanding where YARN fits into Hadoop
  • multitenancy application support / YARN multitenancy application support
  • sample examples, running on / Running sample examples on YARN
  • sample Pi example, running / Running a sample Pi example
  • compatibility, with MapReduce applications / YARN's compatibility with MapReduce applications
  • Apache Spark, running on / Why run on YARN?
  • and , Mesos difference between / Mesos
  • importance, to Big Data industry / What YARN means to the big data industry
  • present / Journey – present and future
  • future / Journey – present and future
  • present on-going features / Present on-going features
  • future features / Future features
• YARN, features
  • Long Running Applications on Secure Clusters (YARN-896) / Present on-going features
  • Application Timeline Server (YARN-321, YARN-1530) / Present on-going features
  • Disk scheduling (YARN-2139) / Present on-going features
  • Reservation-based scheduling (YARN-1051) / Present on-going features
  • Container Resizing (YARN-1197) / Future features
  • Admin labels (YARN-796) / Future features
  • Container Delegation (YARN-1488) / Future features
• YARN-321
  • URL / Present on-going features
• YARN-796
  • URL / Future features
• YARN-896
  • URL / Present on-going features
• YARN-1197
  • URL / Future features
• YARN-1530
  • URL / Present on-going features
• YARN-2139
  • URL / Present on-going features
• YARN-supported frameworks
  • about / YARN-supported frameworks
• YARN administrations
  • about / Administration of YARN
  • configuration files / Administration of YARN
  • administrative tools / Administrative tools
  • nodes, adding from YARN cluster / Adding and removing nodes from a YARN cluster
  • nodes, removing from YARN cluster / Adding and removing nodes from a YARN cluster
  • YARN jobs, administrating / Administrating YARN jobs
  • MapReduce job, configurations / MapReduce job configurations
  • YARN log management / YARN log management
  • YARN web user interface / YARN web user interface
• YARN applications
  • monitoring, with web GUI / Monitoring YARN applications with web GUI
  • developing / Developing YARN applications
  • ApplicationClientProtocol / Developing YARN applications
  • ApplicationMasterProtocol / Developing YARN applications
  • ContainerManagerProtocol / Developing YARN applications
• YARN application workflow
  • about / The YARN application workflow
  • YARN client, writing / Writing the YARN client
  • ApplicationMaster, writing / Writing the YARN ApplicationMaster
• YARN architecture
  • components / Core components of YARN architecture
  • development / Recent developments in YARN architecture
• YARN architecture, components
  • ResourceManager / ResourceManager
  • ApplicationMaster (AM) / ApplicationMaster (AM)
  • NodeManager (NM) / NodeManager (NM)
• YARN client
  • writing / Writing the YARN client
• YARN cluster
  • nodes, adding from / Adding and removing nodes from a YARN cluster
  • nodes, removing from / Adding and removing nodes from a YARN cluster
• YARN jobs
  • administrating / Administrating YARN jobs
• YARN log management / YARN log management
• YARN MapReduce settings
  • example / Example YARN MapReduce settings
  • properties / Example YARN MapReduce settings
• YARN scheduler policies
  • about / YARN scheduler policies
  • FIFO scheduler / The FIFO (First In First Out) scheduler
  • Fair scheduler / The fair scheduler
  • capacity scheduler / The capacity scheduler
• YARN scheduling polices
  • about / YARN scheduling policies
  • FIFO scheduler / The FIFO (First In First Out) scheduler
  • capacity scheduler / The capacity scheduler
  • Fair scheduler / The fair scheduler
• YARN web user interface / YARN web user interface

Z

• Zookeeper
  • URL / Apache ZooKeeper should be installed