By now, you should have a pretty good idea where the sink fits into the Flume architecture. In this chapter, we will first learn about the most-used sink with Hadoop, the HDFS sink. We will then cover two of the newer sinks that support common Near Real Time (NRT) log processing: the ElasticSearchSink and the MorphlineSolrSink. As you'd expect, the first writes data into Elasticsearch and the latter to Solr. The general architecture of Flume supports many other sinks we won't have space to cover in this book. Some come bundled with Flume and can write to HBase, IRC, and, as we saw in Chapter 2, A Quick Start Guide to Flume, a log4j and file sink. Other sinks are available on the Internet and can be used to write data to MongoDB, Cassandra, RabbitMQ, Redis, and just about any other data store you can think of. If you can't find a sink that suits your needs, you can write one easily by extending the org.apache.flume.sink.AbstractSink
class.
Apache Flume: Distributed Log Collection for Hadoop
By :
Apache Flume: Distributed Log Collection for Hadoop
By:
Overview of this book
Table of Contents (16 chapters)
Apache Flume: Distributed Log Collection for Hadoop Second Edition
Credits
About the Author
About the Reviewers
www.PacktPub.com
Preface
Free Chapter
Overview and Architecture
A Quick Start Guide to Flume
Sinks and Sink Processors
Sources and Channel Selectors
Interceptors, ETL, and Routing
Putting It All Together
Monitoring Flume
There Is No Spoon – the Realities of Real-time Distributed Data Collection
Index
Customer Reviews