BACKGROUND

Computer software packages are traditionally executed by a single computer having one or more processors and memory shared between the processors. For large datasets, a software package can be programmed to distribute tasks across multiple computers operating on a portion of the dataset. However, software packages specifically programmed for a certain implementation architecture may not be scalable. Furthermore, the code used to distribute tasks in one software package may not be easily adapted for use in other software packages.

Distributed computing frameworks have been developed to overcome these difficulties. Generally, distributed computer frameworks use of a cluster of computer systems to implement a software program by dividing the program into a plurality of tasks, some of which can be performed in parallel by a subset of the computer systems within the cluster. A distributed software framework can establish rules for distributing data and tasks to different computer systems in order to provide scalability and redundancy. In a distributed software framework, data and computing resources can be dynamically distributed among the networked computer systems in order to quickly and efficiently process a software program that operates on a large data set. As such, distributed software frameworks provide scalability since different software programs can be implemented on them and since the framework can operate on computer system clusters of different sizes as needed.

MapReduce is a programming model for implementing software programs on a distributed computing framework. At a high level, the MapReduce algorithm includes two parts, mapping and reducing. A mapper process operating on a computer system can receive a set of input data and convert each element of the input data into a key/value pair. A plurality of mappers can operate on each element of input data independently, enabling the mapping process to be distributed among a plurality of computers that each operate on a subset of the input data. A reducer process operating on a computer system can take the key/value pairs output from the mapper as input and combine the key/value pairs into a smaller set of key-value pairs. As such, MapReduce is advantageous because it enables a large number of computing systems to operate on a very large set of data in parallel.

Currently, many software programs in operation and development are not designed to be implemented on a distributed computing framework, and thus, cannot be easily scaled to operate on a very large dataset. While some software programs could be rewritten to use MapReduce, thereby enabling them to be implemented on a distributed computing framework, doing so is nontrivial and requires software developers with distributing computer technical expertise since selecting improper mapper keys could result in incorrect operation.

Embodiments of the present invention address these and other problems individually and collectively.

BRIEF SUMMARY

Some embodiments of the invention relate to systems and methods for compiling a distributed software package using source code written in a data-driven programming language for implementation on a non-distributed computer system. The systems and methods can use a transcompiler to identify loops within the non-distributed software program using the non-distributed source code. The transcompiler can identify the loops of the non-distributed source code by determining the functions performed within the non-distributed source code, since the non-distributed is data-driven.

The systems and methods can use the transcompiler to determine grouping indicators associated with each of the loops. The grouping indicators may be included in one or more configuration files associated with the non-distributed source code or one or more configuration files associated with the particular function of the loop itself. The configuration file may associate the function of the loop with a particular grouping indicator, thereby indicating the particular piece of data or type of data that the operation performed by the first loop is grouped by.

The systems and methods can use the transcompiler to generate MapReduce jobs for each of the loops of the non-distributed source code. The MapReduce jobs may implement the same functionality as the loops of the using the non-distributed source code using mapper keys determined based on the grouping indicators. The transcompiler can generate a distributed software package including the MapReduce jobs. The systems and methods may also optimize the distributed software package using machine learning searching algorithms or through analysis of execution usage statistics.

As such, the systems and methods can use the transcompiler to automatically generate a distributed software package for implementation on a distributed software framework that provides the same, or similar, functionality as non-distributed software based on the non-distributed source code. Therefore, non-distributed software programs written in data-driven programming languages can quickly, easily, accurately, and automatically be ported to distributed software frameworks, thereby providing reducing time and costs compared to software porting performed by human software developers. By porting these software programs to be implemented on a distributed computing system, their speed and efficiency in processing very large data sets is improved compared to the same software functionality implemented on a non-distributed system. In addition, any inefficiencies cause by the different programming methods used by the automatic transcompiler, compared to software developers, may be offset through optimization based on usage statistics and the use of machine learning searching algorithms.

A better understanding of the nature and advantages of embodiments of the present invention may be gained with reference to the following detailed description and the accompanying drawings.

