Like any development project, the ones in Hadoop start with proof of concept (POC). Especially because the technology is new and continuously evolving, the focus always begins with figuring out what it can offer and how to leverage it to solve different business problems, be it consumer analysis, breaking news processing, and so on. Being an open source framework, it has its own nuances and requires a learning curve. As these POCs mature and move to pilot and then to production phase, a new infrastructure has to be set up. Then questions arise around maintaining the newly setup infrastructure, including questions on data security and the overall ecosystem's security. Few of the questions that the infrastructure administrators and security paranoids would ask are:

How secure is a Hadoop ecosystem? How secure is the data residing in Hadoop? How would different teams including business analysts, data scientists, developers, and others in the enterprise access the Hadoop ecosystem in a secure manner? How to enforce existing Enterprise Security Models in this new infrastructure? Are there any best practices for securing such an infrastructure?

This chapter will begin the journey to answer these questions and provide an overview of the typical challenges faced in securing Hadoop-based Big Data ecosystem. We will look at the key security considerations and then present the security reference architecture that can be used for securing Hadoop.

The following topics will be covered in this chapter: