New Security Architecture for Big Data Hadoop
Hadoop is a system that provides distributed job processing on large clusters using commodity servers and distributed file system that allows storing and managing petabytes of data which is collected from disparate data sources. Although Hadoop is used on private clusters behind firewalls and physical security, Hadoop is often runs in public cloud for providing shared multi tenant service and used to store sensitive data; as a result strong authorization and authentication is required. Hadoop has two vulnerabilities that can dramatically impact its compromise resilience. That is, overloaded authentication key and the lack of fine grained access control at the data access level. Providing security to Hadoop is more challenging due to all the interactions do not follow the classic client server pattern: the file system is partitioned and distributed over the network, requiring authorization checks at multiple points; the system scales to thousands of servers and tens of thousands of concurrent tasks; submitted batch job is executed at a later time on nodes different from the node on which the client authenticated and submitted the job and jobs from different users are executed on the same node. To address these challenges, the base Kerberos authentication protocol is used which is supplemented by delegation, access token, and job tokens. This paper examines current security mechanisms of Hadoop clusters using Kerberos supplemented by other tokens, and then we analyzed the security problems and risks of Hadoop. Further, we proposed a new authentication protocol inside Hadoop and usage of modified tokens for improving security.