PRIORITY CLAIM

This U.S. patent application claims priority under 35 U.S.C. § 119 to India Application No. 201721030094, filed on Aug. 24, 2017. The entire contents of the abovementioned application are incorporated herein by reference.

TECHNICAL FIELD

The embodiments herein generally relates to a system and method for predicting performance of an application on a big data platform for large data size and, more particularly, predicting performance of the application on Spark™ platform for large data and cluster size with different configurations.

BACKGROUND

The digitization wave has led to challenge of processing high volume and high velocity data in real time. The wide availability of the open source big data processing frameworks, such as Spark™, has increased migration of existing applications and deployment of new applications to these cost-effective platforms. The Spark™ is one of the commodity cluster platform available in open source to address this need due to its in-memory processing capability. However, one of the challenges is assuring performance of an application before deploying in the production system with increase in data size.

Most of the existing state of the art is on performance analysis of the applications executing on the Spark™ system. Some of them doing performance prediction or auto tuning of analytic workload is using machine learning algorithms. They have used a black box approach while being transparent to underlying the big data system. However, these approaches may fail since each of the big data system has various parameters to configure which can impact performance of the application executing on the system. Performance prediction model accuracy may be improved by including knowledge of underlying system.

The present disclosure proposes a system and method which can meaningfully predict an application performance on the Spark™ platform for higher data size using measurements on low volume data in a small size Spark cluster.

SUMMARY

The following presents a simplified summary of some embodiments of the disclosure in order to provide a basic understanding of the embodiments. This summary is not an extensive overview of the embodiments. It is not intended to identify key/critical elements of the embodiments or to delineate the scope of the embodiments. Its sole purpose is to present some embodiments in a simplified form as a prelude to the more detailed description that is presented below.

In view of the foregoing, an embodiment herein provides a system and method for predicting performance of an application on Spark™ platform in a production environment. In general, the embodiments herein are predicting performance of the application on Spark™ platform for large data and cluster size with different configurations. It facilitates agile development by doing performance testing of application for large data size in development environment before deployment. It can be used to do capacity sizing for a desired performance and hence can assure performance over time with increase in data size.

In one aspect, a method for predicting performance of an application on Spark™ platform in a production environment. The method comprising steps of executing the application with a small size Spark cluster on a small data size of the application, wherein the application is executed as a serial execution of one or more Spark jobs and creates an application log. Further, the process includes parsing of the created application log to collect granular level performance data of the application, estimating an average scheduler delay of each task in the one or more Spark jobs as function of one or more parameters of the Spark™ platform and the small data size of the application using a scheduler delay predictor module, estimating an average execution time of shuffle operation of each task in the one or more Spark jobs as function of parameters of the Spark™ platform and the small data size of the application using a shuffle time predictor module, estimating an average execution time of java virtual machine (JVM) of each task in the one or more Spark jobs as function of parameters of the Spark™ platform and the small data size of the application using an execution time predictor module, simulating stage execution for the one or more Spark jobs for estimating an execution time of the application on a Spark™ platform in the production environment. Finally, the process predicts performance of the application using the estimated average scheduler delay of each task, the estimated average execution time of shuffle operation of each task, the estimated average execution time of JVM overheads of each task, and the estimated execution time of each stage of the one or more Spark jobs.

In another aspect, a system for predicting performance of an application on a Spark™ platform in a production environment. The system comprising a memory with set of instructions and a processor. Further, the system comprises an execution module configured to execute the application with a Spark cluster on small data size of the application, wherein the application is executed as a serial execution of one or more Spark jobs and creates an application log. Furthermore, the system comprises a Spark log parser module configured to parse the created application log to collect granular level performance data of the application, a scheduler delay estimation module configured to estimate an average scheduler delay of each task in the one or more Spark jobs as function of one or more parameters and the data size, a shuffle time estimation module configured to estimate an average execution time of shuffle operation of each task in the one or more Spark jobs as function of one or more parameters and the data size, a JVM execution time estimation module to estimate an average execution time of java virtual machine (JVM) overheads of each task in the one or more Spark jobs as function of one or more parameters and the application, a simulation module configured to simulate for estimating execution time of each stage of the one or more Spark jobs of the application and an estimation module configured to estimate the performance of the application using the estimated average scheduler delay of each task, the estimated average execution time of shuffle operation of each task, the estimated average execution time of JVM overheads of each task, and the estimated execution time of each stage of each Spark job.

It should be appreciated by those skilled in the art that any block diagram herein represents conceptual views of illustrative systems embodying the principles of the present subject matter. Similarly, it will be appreciated that any flow charts, flow diagrams, state transition diagrams, pseudo code, and the like represent various processes which may be substantially represented in computer readable medium and so executed by a computing device or processor, whether or not such computing device or processor is explicitly shown.

