BACKGROUND

Data systems are heterogeneous; in many cases, a company or team have both a stream processing system and a batch processing system. The stream processing system provides insights in low latency but with high engineering cost and high resource cost (CPU, memory, storage, etc.). The batch processing system provides more comprehensive insights with low engineering cost and low running cost, but with high latency (hours or day).

In industry, a typical stream processing system requires distributed messaging queues which provide a publish-subscribe model to make stream processing easy and flexible. The messaging queue has a dedicated service and storage that provides real-time data processing but introduces high resource costs.

The subject matter claimed herein is not limited to embodiments that solve any disadvantages or that operate only in environments such as those described above. Rather, this background is only provided to illustrate one exemplary technology area where some embodiments described herein may be practiced.

BRIEF SUMMARY

This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

A stream processing system, which is built on top of a messaging queue, is able to provide real-time (sub-second or few second) data processing of data in a distributed messaging queue. While this may be advantageous in circumstances where real-time data processing is needed, providing this real-time data processing comes at a high resource cost. For example, the stream processing system, which is built on top of a messaging queue, requires a large amount of dedicated storage and processing resources to ensure that the new data is quickly processed. This is especially true in high data volume circumstances. In addition, it is typically required to know the number of partition files ahead of time so that the proper resources can be allocated. Accordingly, the stream processing system, which is built on top of a messaging queue, requires a company or the team that is implementing the stream processing system to have dedicated servers and storage for the system.

A batch processing system typically requires less resource cost than a stream processing system. For example, since the data in partition files is only processed after a given time window ends, fewer processing resources are needed because the data is processed in a large batch or chunk of data instead of event by event. Thus, the number of IOs can be reduced, leading to higher processing efficiency. However, since the processing is done after the end of the time window, there is high latency for receiving the processed results. For example, if the time window is a day, then the data will not be processed until after the day is done. Further, if any data from one time period is delayed in being provided to the system until after the end of the time period, that data will typically not be processed until the end of the next time window adding additional latency. An example of this is in the banking industry where a deposit received on a given day after 3 PM is added into the deposits for the next day. Thus, a batch processing system provides large data volume processing, but at a cost of delayed processing results.

Since both of the stream processing system and the batch processing system provide different advantages, it is common for a company or team to implement both systems. Thus, these companies or teams are required to have separate, dedicated resources for both systems. In addition, it quite common for the data that is to be processed to come from the same data producers and thus have to be duplicated so that both the stream processing system and the batch processing system are able to process the data in their respective ways. The need for two separate processing systems with their own dedicated resources can be costly.

The embodiments described herein aim to solve at least some of the above-mentioned problems by providing for a near-real-time stream processing system that is implemented using the same distributed file system as the batch processing system. In this way, data received by the distributed file system need not be duplicated. In some instances, the implementation of the near-real-time stream processing system may result in 70% or more reduction in engineering and resource costs.

In one embodiment a data container and one or more partition files within the data container are generated according to a defined partition window. The defined partition window specifies a first time range that controls when data is to be included in the one or more partition files of the data container. The data container is scanned to determine if the one or more partition files are within a defined partition lifetime window. The defined partition lifetime window specifies a second time range that controls how long the one or more partition files are considered active such that processing is to be performed on the one or more partitions. For each partition file within the defined lifetime window, one or more processing tasks are created based on an amount of data included in the one or more partition files. The data in the one or more partition files is accessed and the one or more processing tasks are performed. Information about the one or more partition files is recorded in a configuration data store.

In an embodiment, the one or more processing tasks comprise a sequential process where there is only one active processing task at any given time for a given partition file of the one or more partition files, a parallel process where two or more processing tasks are completed at substantially the same time by different processing instances for a given partition file of the one or more partition files, or a parallel process within a same processing instance where two or more processing tasks are assigned to be performed by the same processing instance for a given partition file of the one or more partition files.

In an embodiment, the one or more partition files are discovered dynamically at runtime. In another embodiment, the defined partition lifetime window is larger than the defined partition window.

In an embodiment, a second data container and one or more second partition files within the second data container are generated according to a second defined partition window. The second defined partition window specifies a third time range that has a beginning after an ending of the first defined partition window but before an ending of the defined partition lifetime window. Prior to the ending of the defined partition lifetime window, the one or more partition files in the first data container and the one or more second partition files in the second data container are scanned. The one or more partition files and the one or more second partition files are both considered active during the defined partition lifetime window One or more processing tasks are created for the one or more partition files and the one or more second partition files. The assigned processing tasks are performed. the information about the one or more partition files and information about the one or more second partition files are recorded in the configuration data store.

In an embodiment, the one or more partition files are closed when it is determined that the partition lifetime window has reached an end and it is determined that processing has been performed on all the data in the one or more partition files. In another embodiment, the information about the one or more partition files recorded in the configuration data store comprises information about the progress of the one or more processing tasks. In a further embodiment, the one or more processing tasks are defined by a partition file identifier, a start offset address location of the data, and an end offset address location of the data.

In an embodiment, a task manager component is configured to scan a data container to discover if the data container includes one or more partition files and upon discovery, to determine if the one or more partition files are within a defined partition lifetime window, the partition lifetime specifying a time range that controls how long the one or more partition files are considered active such that processing can be performed on the received data in the one or more partitions. The task manager component is further configured to create, for those partition files that are within the defined partition lifetime, one or more processing tasks. One or more task worker components are configured to receive the one or more processing tasks from the task manager component and perform processing on the data in the one or more partition files according to the one or more processing tasks. The task manager component is further configured to cause information about the one or more partition files and about progress of the one or more processing tasks to be stored in a configuration data store.

Additional features and advantages will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by the practice of the teachings herein. Features and advantages of the invention may be realized and obtained by means of the instruments and combinations particularly pointed out in the appended claims. Features of the present invention will become more fully apparent from the following description and appended claims or may be learned by the practice of the invention as set forth hereinafter.

