BACKGROUND

This description relates to processing data from multiple sources. Data can be stored in a variety of sources, including, for example, an HDFS (Hadoop Distributed File System) cluster. A data processing system can perform operations on data received from an HDFS cluster and also data received from other types of sources.

SUMMARY

In a first aspect, a method includes, at a node of a Hadoop cluster, the node storing a first portion of data in HDFS data storage, executing a first instance of a data processing engine capable of receiving data from a data source external to the Hadoop cluster, receiving a computer-executable program by the data processing engine, executing at least part of the program by the first instance of the data processing engine, receiving, by the data processing engine, a second portion of data from the external data source, storing the second portion of data other than in HDFS storage, and performing, by the data processing engine, a data processing operation identified by the program using at least the first portion of data and the second portion of data.

In a second aspect, a method includes, at a node storing a first portion of data and operating in conjunction with a cluster of nodes, the cluster storing an aggregation of data that can be operated on in parallel, executing a first instance of a data processing engine capable of receiving data from a data source external to the cluster, receiving a computer-executable program by the data processing engine, executing at least part of the program by the first instance of the data processing engine, receiving, by the data processing engine, a second portion of data from the external data source, storing the second portion of data in volatile memory of the node, and performing, by the data processing engine, a data processing operation identified by the program using at least the first portion of data and the second portion of data.

In a third aspect according to the first or second aspect, the Hadoop cluster includes nodes each executing an instance of the data processing engine, the instances of the data processing engine running concurrently to perform the data processing operation together in parallel on a) a first body of data that includes the first portion of data, the first body of data also including other portions of data being processed by the other nodes of the Hadoop cluster, and b) a second body of data that includes the second portion of data, the second body of data being stored in a format native to a relational database system, and the second body of data being divided into portions that each can be stored in volatile memory of the nodes of the Hadoop cluster.

In a fourth aspect according to any of the first through third aspects, the computer program is a dataflow graph executed by a graph execution engine of the data processing engine, wherein the dataflow graph includes a) at least one component representing the Hadoop cluster, b) at least one component representing the source of the second portion of data, and c) at least one link that represents at least one dataflow associated with the operation to be performed on the data received from at least one source of data.

In a fifth aspect according to the fourth aspect, at least one component of the dataflow graph is connected to a link representing a flow of data from the Hadoop cluster, and wherein the at least one component is connected to a link representing a flow of data from the source of the second portion of data.

In a sixth aspect according to any of the first through fifth aspects, the data processing engine does not implement the MapReduce programming model.

In a seventh aspect according to any of the first through sixth aspects, the second portion of data is stored in volatile memory.

In an eighth aspect according to any of the first through seventh aspects, the method includes receiving a database query, the database query including at least one operation to be performed on data received from at least one source of data that includes the Hadoop cluster; and the computer program includes components representing operations corresponding to the database query, wherein the computer program includes at least one component representing the at least one source of data and at least one link that represents at least one dataflow associated with the operation to be performed on data received from at least one source of data.

In a ninth aspect according to any of the first through eighth aspects, the second portion of data was chosen based on characteristics of the first portion of data.

In a tenth aspect according to any of the first through ninth aspects, the second portion of data includes a subset of rows of a relational database, and the second portion of data includes a subset of columns of the relational database

In an eleventh aspect according to any of the first through tenth aspects, the second portion of data is distinct from a third portion of data received at a second node of the Hadoop cluster from the external data source.

In a twelfth aspect according to any of the first through eleventh aspects, the method includes executing at least part of the program by a second instance of the data processing engine outside of the Hadoop cluster.

In a thirteenth aspect according to any of the first through twelfth aspects, the method includes communicating with at least part of the program being executed by a second instance of the data processing engine outside of the Hadoop cluster.

In a fourteenth aspect, a method includes, at a data processing engine of a node of a Hadoop cluster, performing a data processing operation identified by a computer-executable program being executed by the data processing engine, the data processing operation being performed using at least a first portion of data stored in HDFS data storage at the node and at least a second portion of data received from a data source external to the Hadoop cluster and stored other than in HDFS the storage.

In a fifteenth aspect, a method includes receiving a SQL query specifying sources of data including a Hadoop cluster and a relational database, generating a computer-executable program that corresponds to the SQL query, executing the computer-executable program at a data processing engine of a node of the Hadoop cluster, and performing, by the data processing engine, a data processing operation identified by the computer-executable program using at least data of the Hadoop cluster and data of the relational database.

One or more of the aspects, alone or in combination, may be represented as a system, or an apparatus, or as a computer readable storage device storing a computer program product including machine readable instructions that, when executed by a computer system, carry out operations of the aspect. As one example, a computer readable storage device can store a computer program product including machine readable instructions that, when executed by a computer system, carry out operations according to any one of the first through fifteenth aspects. As another example, a computer system including one or more processors can include a computer-readable storage device storing a computer program product that includes machine readable instructions that, when executed by the one or more processors, carry out operations according to any one of the first through fifteenth aspects.

One or more of the above aspects may provide the following advantages. First, a Hadoop node can operate on data stored in volatile memory, and need not write data to disk before performing operations on the data. Second, a Hadoop node can be configured to receive data from multiple types of data sources. Third, a Hadoop node can be configured to operate in association with a general-purpose data processing operating system, e.g., a data processing operating system not specific to Hadoop nodes. Fourth, a Hadoop node can be configured to operate with dataflow graphs that carry out data processing operations.

Other features and advantages of the invention will become apparent from the following description, and from the claims.

