In any enterprise, how ever big or small it could be, logging is very important for security and improvement of services. With more and more systems going online, not just logging data but also reviewing the logs is very important.
Systems today are generating millions of lines of data, which could be simple web events, some database query logs, network logs, or any audit logs. Dealing with such large logs manually is neither practical nor viable in terms of time; there must be centralized logging and analysis tools to quickly extract the useful information.
In this chapter, we begin our journey by exploring the concepts of logging at both the system level and Hadoop level.
We'll cover the following topics in this chapter:
Different log levels
Logs written by various Hadoop daemons
Understanding audit logs and how they could be useful