Book Image

Apache Flume: Distributed Log Collection for Hadoop

By : Steven Hoffman
Book Image

Apache Flume: Distributed Log Collection for Hadoop

By: Steven Hoffman

Overview of this book

Related Content you might be interested in

Current Title:

Small book image
Apache Flume: Distributed Log Collection for Hadoop
Steven Hoffman
Feb, 2015 | 178 pages
No titles found
Table of Contents (16 chapters)
Apache Flume: Distributed Log Collection for Hadoop Second Edition
Apache Flume: Distributed Log Collection for Hadoop Second Edition
Credits
Credits
About the Author
About the Author
About the Reviewers
About the Reviewers
www.PacktPub.com
www.PacktPub.com
Preface
Preface
Free Chapter
1
Overview and Architecture
Overview and Architecture
Flume 0.9
Flume 1.X (Flume-NG)
The problem with HDFS and streaming data/logs
Sources, channels, and sinks
Flume events
The Kite SDK
Summary
2
A Quick Start Guide to Flume
A Quick Start Guide to Flume
Downloading Flume
An overview of the Flume configuration file
Starting up with "Hello, World!"
Summary
3
Channels
Channels
The memory channel
The file channel
Spillable Memory Channel
Summary
4
Sinks and Sink Processors
Sinks and Sink Processors
HDFS sink
Compression codecs
Event Serializers
Sink groups
MorphlineSolrSink
ElasticSearchSink
Summary
5
Sources and Channel Selectors
Sources and Channel Selectors
The problem with using tail
The Exec source
Spooling Directory Source
Syslog sources
JMS source
Channel selectors
Summary
6
Interceptors, ETL, and Routing
Interceptors, ETL, and Routing
Interceptors
Tiering flows
The embedded agent
Routing
Summary
7
Putting It All Together
Putting It All Together
Web logs to searchable UI
Archiving to HDFS
Summary
8
Monitoring Flume
Monitoring Flume
Monitoring the agent process
Monitoring performance metrics
Summary
9
There Is No Spoon – the Realities of Real-time Distributed Data Collection
There Is No Spoon – the Realities of Real-time Distributed Data Collection
Transport time versus log time
Time zones are evil
Capacity planning
Considerations for multiple data centers
Compliance and data expiry
Summary
Index
Index

Sources, channels, and sinks

The Flume agent's architecture can be viewed in this simple diagram. Inputs are called sources and outputs are called sinks. Channels provide the glue between sources and sinks. All of these run inside a daemon called an agent.

Note

Keep in mind:

  • A source writes events to one or more channels.

  • A channel is the holding area as events are passed from a source to a sink.

  • A sink receives events from one channel only.

  • An agent can have many channels.

Previous Section
End of Section 5
Next Section