Book Image

Apache Flume: Distributed Log Collection for Hadoop

By : Steven Hoffman
Book Image

Apache Flume: Distributed Log Collection for Hadoop

By: Steven Hoffman

Overview of this book

Related Content you might be interested in

Current Title:

Small book image
Apache Flume: Distributed Log Collection for Hadoop
Steven Hoffman
Feb, 2015 | 178 pages
No titles found
Table of Contents (16 chapters)
Apache Flume: Distributed Log Collection for Hadoop Second Edition
Apache Flume: Distributed Log Collection for Hadoop Second Edition
Credits
Credits
About the Author
About the Author
About the Reviewers
About the Reviewers
www.PacktPub.com
www.PacktPub.com
Preface
Preface
Free Chapter
1
Overview and Architecture
Overview and Architecture
Flume 0.9
Flume 1.X (Flume-NG)
The problem with HDFS and streaming data/logs
Sources, channels, and sinks
Flume events
The Kite SDK
Summary
2
A Quick Start Guide to Flume
A Quick Start Guide to Flume
Downloading Flume
An overview of the Flume configuration file
Starting up with "Hello, World!"
Summary
3
Channels
Channels
The memory channel
The file channel
Spillable Memory Channel
Summary
4
Sinks and Sink Processors
Sinks and Sink Processors
HDFS sink
Compression codecs
Event Serializers
Sink groups
MorphlineSolrSink
ElasticSearchSink
Summary
5
Sources and Channel Selectors
Sources and Channel Selectors
The problem with using tail
The Exec source
Spooling Directory Source
Syslog sources
JMS source
Channel selectors
Summary
6
Interceptors, ETL, and Routing
Interceptors, ETL, and Routing
Interceptors
Tiering flows
The embedded agent
Routing
Summary
7
Putting It All Together
Putting It All Together
Web logs to searchable UI
Archiving to HDFS
Summary
8
Monitoring Flume
Monitoring Flume
Monitoring the agent process
Monitoring performance metrics
Summary
9
There Is No Spoon – the Realities of Real-time Distributed Data Collection
There Is No Spoon – the Realities of Real-time Distributed Data Collection
Transport time versus log time
Time zones are evil
Capacity planning
Considerations for multiple data centers
Compliance and data expiry
Summary
Index
Index

About the Author

Steve Hoffman has 32 years of experience in software development, ranging from embedded software development to the design and implementation of large-scale, service-oriented, object-oriented systems. For the last 5 years, he has focused on infrastructure as code, including automated Hadoop and HBase implementations and data ingestion using Apache Flume. Steve holds a BS in computer engineering from the University of Illinois at Urbana-Champaign and an MS in computer science from DePaul University. He is currently a senior principal engineer at Orbitz Worldwide (http://orbitz.com/).

More information on Steve can be found at http://bit.ly/bacoboy and on Twitter at @bacoboy.

This is the first update to Steve's first book, Apache Flume: Distributed Log Collection for Hadoop, Packt Publishing.

I'd again like to dedicate this updated book to my loving and supportive wife, Tracy. She puts up with a lot, and that is very much appreciated. I couldn't ask for a better friend daily by my side.

My terrific children, Rachel and Noah, are a constant reminder that hard work does pay off and that great things can come from chaos.

I also want to give a big thanks to my parents, Alan and Karen, for molding me into the somewhat satisfactory human I've become. Their dedication to family and education above all else guides me daily as I attempt to help my own children find their happiness in the world.

Previous Section
Next Chapter