Browse Library
Sign In Start Free Trial
Close icon Search icon
Account
Browse Library Advanced Search Sign In Start Free Trial

Add to playlist

Create a Playlist

Modal Close icon
You need to login to use this feature.
  • Book Overview & Buying Apache Flume: Distributed Log Collection for Hadoop - Second Edition
  • Table Of Contents Toc
  • Feedback & Rating feedback
Apache Flume: Distributed Log Collection for Hadoop - Second Edition

Apache Flume: Distributed Log Collection for Hadoop - Second Edition - Second Edition

By : Steven Hoffman
5 (1)
Buy this Book
close
close
Apache Flume: Distributed Log Collection for Hadoop - Second Edition

Apache Flume: Distributed Log Collection for Hadoop - Second Edition

5 (1)
By: Steven Hoffman
Buy this Book

Overview of this book

If you are a Hadoop programmer who wants to learn about Flume to be able to move datasets into Hadoop in a timely and replicable manner, then this book is ideal for you. No prior knowledge about Apache Flume is necessary, but a basic knowledge of Hadoop and the Hadoop File System (HDFS) is assumed.
Table of Contents (11 chapters)
close
close
close
Preface
Icon
Preface
Icon
What this book covers
Icon
What you need for this book
Icon
Who this book is for
Icon
Conventions
Icon
Reader feedback
Icon
Customer support
Lock Free Chapter
1
1. Overview and Architecture
Icon
1. Overview and Architecture
Icon
Flume 0.9
Icon
Flume 1.X (Flume-NG)
Icon
The problem with HDFS and streaming data/logs
Icon
Sources, channels, and sinks
Icon
Flume events
Icon
The Kite SDK
Icon
Summary
2
2. A Quick Start Guide to Flume
Icon
2. A Quick Start Guide to Flume
Icon
Downloading Flume
Icon
An overview of the Flume configuration file
Icon
Starting up with "Hello, World!"
Icon
Summary
3
3. Channels
Icon
3. Channels
Icon
The memory channel
Icon
The file channel
Icon
Spillable Memory Channel
Icon
Summary
4
4. Sinks and Sink Processors
Icon
4. Sinks and Sink Processors
Icon
HDFS sink
Icon
Compression codecs
Icon
Event Serializers
Icon
Sink groups
Icon
MorphlineSolrSink
Icon
ElasticSearchSink
Icon
Summary
5
5. Sources and Channel Selectors
Icon
5. Sources and Channel Selectors
Icon
The problem with using tail
Icon
The Exec source
Icon
Spooling Directory Source
Icon
Syslog sources
Icon
JMS source
Icon
Channel selectors
Icon
Summary
6
6. Interceptors, ETL, and Routing
Icon
6. Interceptors, ETL, and Routing
Icon
Interceptors
Icon
Tiering flows
Icon
The embedded agent
Icon
Routing
Icon
Summary
7
7. Putting It All Together
Icon
7. Putting It All Together
Icon
Web logs to searchable UI
Icon
Archiving to HDFS
Icon
Summary
8
8. Monitoring Flume
chevron up
Icon
8. Monitoring Flume
Icon
Monitoring the agent process
Icon
Monitoring performance metrics
Icon
Summary
9
9. There Is No Spoon – the Realities of Real-time Distributed Data Collection
Icon
9. There Is No Spoon – the Realities of Real-time Distributed Data Collection
Icon
Transport time versus log time
Icon
Time zones are evil
Icon
Capacity planning
Icon
Considerations for multiple data centers
Icon
Compliance and data expiry
Icon
Summary
10
Index
Icon
Index

Monitoring performance metrics

Now that we have covered some options for process monitoring, how do you know whether your application is actually doing the work you think it is? On many occasions, I've seen a stuck syslog-ng process that appears to be running, but it just wasn't sending any data. I'm not picking on syslog-ng specifically; all software does this when conditions that are not designed for occur.

When talking about Flume data flows, you need to monitor the following:

  • Data entering sources is within expected rates

  • Data isn't overflowing your channels

  • Data is exiting sinks at expected rates

Flume has a pluggable monitoring framework, but as mentioned at the beginning of the chapter, it is still very much a work in progress. This does not mean you shouldn't use it, as that would be foolish. It means you'll want to prepare extra testing and integration time anytime you upgrade.

Note

While not covered in the Flume documentation, it is common to enable JMX in your Flume JVM (http://bit.ly...

Visually different images
CONTINUE READING
83
Tech Concepts
36
Programming languages
73
Tech Tools
Icon Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.
Icon Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.
Icon 50+ new titles added per month and exclusive early access to books as they are being written.
Apache Flume: Distributed Log Collection for Hadoop - Second Edition
End of Section 8
Next Section
notes
bookmark Notes and Bookmarks search Search in title playlist Add to playlist font-size Font size

Change the font size

margin-width Margin width

Change margin width

day-mode Day/Sepia/Night Modes

Change background colour

Close icon Search
Country selected
Close icon Your notes and bookmarks

Confirmation

Modal Close icon
claim successful
Order Page Read Now

Buy this book with your credits?

Modal Close icon
Are you sure you want to buy this book with one of your credits?
Close
YES, BUY

Submit Your Feedback

Modal Close icon
Modal Close icon
Modal Close icon