Browse Library
Sign In Start Free Trial
Close icon Search icon
Account
Browse Library Sign In Start Free Trial

Add to playlist

Create a Playlist

Modal Close icon
You need to login to use this feature.
  • Book Overview & Buying Apache Flume: Distributed Log Collection for Hadoop - Second Edition
  • Table Of Contents Toc
Apache Flume: Distributed Log Collection for Hadoop - Second Edition

Apache Flume: Distributed Log Collection for Hadoop - Second Edition - Second Edition

By : Steven Hoffman
5 (1)
Buy this Book
close
close
Apache Flume: Distributed Log Collection for Hadoop - Second Edition

Apache Flume: Distributed Log Collection for Hadoop - Second Edition

5 (1)
By: Steven Hoffman
Buy this Book

Overview of this book

If you are a Hadoop programmer who wants to learn about Flume to be able to move datasets into Hadoop in a timely and replicable manner, then this book is ideal for you. No prior knowledge about Apache Flume is necessary, but a basic knowledge of Hadoop and the Hadoop File System (HDFS) is assumed.
Table of Contents (11 chapters)
close
close
close
Preface
Icon
Preface
Icon
What this book covers
Icon
What you need for this book
Icon
Who this book is for
Icon
Conventions
Icon
Reader feedback
Icon
Customer support
Lock Free Chapter
1
1. Overview and Architecture
Icon
1. Overview and Architecture
Icon
Flume 0.9
Icon
Flume 1.X (Flume-NG)
Icon
The problem with HDFS and streaming data/logs
Icon
Sources, channels, and sinks
Icon
Flume events
Icon
The Kite SDK
Icon
Summary
2
2. A Quick Start Guide to Flume
Icon
2. A Quick Start Guide to Flume
Icon
Downloading Flume
Icon
An overview of the Flume configuration file
Icon
Starting up with "Hello, World!"
Icon
Summary
3
3. Channels
Icon
3. Channels
Icon
The memory channel
Icon
The file channel
Icon
Spillable Memory Channel
Icon
Summary
4
4. Sinks and Sink Processors
Icon
4. Sinks and Sink Processors
Icon
HDFS sink
Icon
Compression codecs
Icon
Event Serializers
Icon
Sink groups
Icon
MorphlineSolrSink
Icon
ElasticSearchSink
Icon
Summary
5
5. Sources and Channel Selectors
Icon
5. Sources and Channel Selectors
Icon
The problem with using tail
Icon
The Exec source
Icon
Spooling Directory Source
Icon
Syslog sources
Icon
JMS source
Icon
Channel selectors
Icon
Summary
6
6. Interceptors, ETL, and Routing
Icon
6. Interceptors, ETL, and Routing
Icon
Interceptors
Icon
Tiering flows
Icon
The embedded agent
Icon
Routing
Icon
Summary
7
7. Putting It All Together
Icon
7. Putting It All Together
Icon
Web logs to searchable UI
Icon
Archiving to HDFS
Icon
Summary
8
8. Monitoring Flume
chevron up
Icon
8. Monitoring Flume
Icon
Monitoring the agent process
Icon
Monitoring performance metrics
Icon
Summary
9
9. There Is No Spoon – the Realities of Real-time Distributed Data Collection
Icon
9. There Is No Spoon – the Realities of Real-time Distributed Data Collection
Icon
Transport time versus log time
Icon
Time zones are evil
Icon
Capacity planning
Icon
Considerations for multiple data centers
Icon
Compliance and data expiry
Icon
Summary
10
Index
Icon
Index

Monitoring the agent process

The most obvious type of monitoring you'll want to perform is Flume agent process monitoring, that is, making sure the agent is still running. There are many products that do this kind of process monitoring, so there is no way we can cover them all. If you work at a company of any reasonable size, chances are there is already a system in place for this. If this is the case, do not go off and build your own. The last thing operations wants is yet another screen to watch 24/7.

Monit

If you do not already have something in place, one freemium option is Monit (http://mmonit.com/monit/). The developers of Monit have a paid version that provides more bells and whistles you may want to consider. Even in the free form, it can provide you with a way to check whether the Flume agent is running, restart it if it isn't, and send you an e-mail when this happens so that you can look into why it died.

Monit does much more, but this functionality is what we will cover here. If...

CONTINUE READING
83
Tech Concepts
36
Programming languages
73
Tech Tools
Icon Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.
Icon Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.
Icon 50+ new titles added per month and exclusive early access to books as they are being written.
Apache Flume: Distributed Log Collection for Hadoop - Second Edition
End of Section 8
Next Section
notes
bookmark Notes and Bookmarks search Search in title playlist Add to playlist font-size Font size

Change the font size

margin-width Margin width

Change margin width

day-mode Day/Sepia/Night Modes

Change background colour

Close icon Search
Country selected
Close icon Your notes and bookmarks

Confirmation

Modal Close icon
claim successful
Order Page Read Now

Buy this book with your credits?

Modal Close icon
Are you sure you want to buy this book with one of your credits?
Close
YES, BUY

Submit Your Feedback

Modal Close icon
Modal Close icon
Modal Close icon