A Development of Streaming Big Data Analysis System Using In memory Cluster Computing Framework: Spark
In this paper, to deal with stream big data processing issue, we design and implement a big data analysis system using Spark which is an In memory cluster computing framework. Spark is provided by ASF (Apache Software Foundation) open source community, and is regarded as a next generation high performance cluster computing technology. From the performance evaluation of the proposed system, we can see that Spark is 20+ times faster than conventional Mapreduce based Hive SQL in terms of the response time. According to these results, we can confirm that the proposed system can be applied to solve the soft real time big data analysis jobs for sensor data generated in a smart factory.