We will conclude with a brief rundown of some of the popular security tools we may encounter while developing with Apache Spark - and some advice about when to use them.
As the Hadoop ecosystem grows ever larger, products such as Hive, HBase, HDFS, Sqoop, and Spark all have different security implementations. This means that duplicate policies are often required across the product stack in order to provide the user with a seamless experience, as well as enforce the overarching security manifest. This can quickly become complicated and time consuming to manage, which often leads to mistakes and even security breaches (whether intentional or otherwise). Apache Sentry pulls many of the mainstream Hadoop products together, particularly with Hive/HS2, to provide fine-grained (up to column level) controls.
Using ACLs is simple, but high maintenance. The setting of permissions for a large number of new files and amending umasks is very cumbersome and time consuming...