Although Hadoop has been very successful for most of the Big Data problems, it is not an optimal choice in many situations. In this recipe, we will introduce a few Hadoop alternatives.
Hadoop has the following drawbacks as a Big Data platform:
As an open source software, Hadoop is difficult to configure and manage, mainly due to the instability of the software and lack of properly maintained documentation and technical support
Hadoop is not an optimal choice for real-time, responsive Big Data applications
Hadoop is not a good fit for large graph datasets
Because of the preceding drawbacks as well as other reasons, such as special data processing requirements, we need to make an alternative choice.
We can choose Hadoop alternatives using the following guidelines:
Choose Enterprise Hadoop if there is no qualified Hadoop administrator and there is sufficient budget for deploying a Big Data platform.
Choose Spark or Storm if an application requires real-time data processing.
Choose GraphLab if an application requires handling of large graph datasets.
Enterprise Hadoop refers to Hadoop distributions by some Hadoop-oriented companies. Compared with the community Hadoop releases, Enterprise Hadoop distributions are enterprise ready, easy to configure, and sometimes new features are added. In addition, the training and support services provided by these companies make it much easier for organizations to adopt the Hadoop Big Data platform. Famous Hadoop-oriented companies include: Cloudera, Horntonworks, MapR, Hadapt, and so on.
Cloudera is one of the most famous companies that delivers Enterprise Hadoop Big Data solutions. It provides Hadoop consulting, training, and certification services. It is also one of the biggest contributors of the Hadoop codebase. Their Big Data solution uses Cloudera Desktop as the cluster management interface. You can learn more from www.cloudera.com.
Hortonworks and MapR both provide featured Hadoop distributions and Hadoop-based Big Data solutions. You can get more details from www.hortonworks.com and www.mapr.com.
Hadapt differentiates itself from the other Hadoop-oriented companies by the goal of integrating structured, semi-structured, and unstructured data into a uniform data operation platform. Hadapt unifies SQL and Hadoop and makes it easy to handle different varieties of data. You can learn more at http://hadapt.com/.
Spark is a real-time in-memory Big Data processing platform. It can be up to 40 times faster than Hadoop. So it is ideal for iterative and responsive Big Data applications. Besides, Spark can be integrated with Hadoop, and the Hadoop-compatible storage APIs enable it to access any Hadoop-supported systems. More information about Spark can be learned from http://spark-project.org/.
Storm is another famous real-time Big Data processing platform. It is developed and open sourced by Twitter. For more information, please check http://storm-project.net/.
GraphLab is an open source distributed system developed at Carnegie Mellon University. It was targeted for handling sparse iterative graph algorithms. For more information, please visit: http://graphlab.org/.
Tip
The MapReduce framework parallels computation by splitting data into a number of distributed nodes. Some large natural graph data, such as social network data, has the problem of being hard to partition and thus, hard to split for Hadoop parallel processing. The performance can be severely panelized if Hadoop is used.
Other Hadoop-like implementations include Phoenix (http://mapreduce.stanford.edu/), which is a shared memory implementation of the MapReduce data processing framework, and Haloop (http://code.google.com/p/haloop/), which is a modified version of Hadoop for iterative data processing.
As the Big Data problem floods the whole world, many systems have been designed to deal with the problem. Two such famous systems that do not follow the MapReduce route are Message Passing Interface (MPI) and High Performance Cluster Computing (HPCC).
MPI is a library specification for message passing. Different from Hadoop, MPI was designed for high performance on both massively parallel machines and on workstation clusters. In addition, MPI lacks fault tolerance and performance will be bounded when data becomes large. More documentation about MPI can be found at http://www.mpi-forum.org/.
HPCC is an open source Big Data platform developed by HPCC systems, which was acquired by LexisNexis Risk Solutions. It achieves high performance by clustering commodity hardware. The system includes configurations for both parallel batch processing and high performance online query applications using indexed data files. The HPCC platform contains two cluster processing subsystems: Data Refinery subsystem and Data Delivery subsystem. The Data Refinery subsystem is responsible for the general processing of massive raw data, and the Data Delivery subsystem is responsible for the delivery of clean data for online queries and analytics. More information about HPCC can be found at http://hpccsystems.com/.