CROSS-REFERENCE TO RELATED APPLICATION

This application is a 371 of international application of PCT application serial no. PCT/CN2019/124086, filed on Dec. 9, 2019, which claims the priority benefit of China application no. 201811535124.4, filed on Dec. 14, 2018. The entirety of each of the above mentioned patent applications is hereby incorporated by reference herein and made a part of this specification.

TECHNICAL FIELD

This application belongs to the field of cloud computing technology, and particularly relates to a task scheduling simulation system.

DESCRIPTION OF RELATED ART

The task scheduling simulation system on the cloud computing platform can use several machine nodes far less than the number of machines in the production environment according to the log records of task operation in the production environment, to truly reflect the number of tasks to be processed by the resource management scheduling system at a certain time, as well as the changes such as the downtime and addition of cluster nodes, in order to provide researchers with an experimental environment that is highly consistent with the real production environment for the research of the scheduling algorithm, thereby providing support for the effectiveness of the scheduling algorithm. On the other hand, compared with the case that the production environment lacks the records of the worst task scheduling considerations, the task scheduling simulation system can simulate the experimental scenario where the peak value of the number of submitted tasks or the resource utilization of the entire cluster reaches a critical value through some special settings. Thus, it provides an operating environment for testing the operating efficiency of the newly designed scheduling algorithm in the worst case.

The Yarn simulation system (Scheduler Load Simulator, SLS) in the big data processing system Hadoop simulates and runs batch tasks based on Map-Reduce. The input of SLS running is the running log of batch tasks, including the running time of each task and the CPU and memory resources it requires. However, in the data center 24-hour log records published by some cloud computing platforms, it is the mixed scheduling and running of batch tasks and online tasks on some cloud computing data centers. The SLS can only be aimed at the simulation system of Map-Reduce single batch tasks in Hadoop.

SUMMARY

1. Technical Problems to be Solved

The Yarn simulation system (Scheduler Load Simulator, SLS) in the big data processing system Hadoop simulates and runs batch tasks based on Map-Reduce. The input of SLS running is the running log of batch tasks, including the running time of each task and the CPU and memory resources it requires. However, in the data center 24-hour log records published by some cloud computing platforms, it is the mixed scheduling and running of batch tasks and online tasks on some cloud computing data centers. The SLS can only be aimed at the issues of the simulation system of Map-Reduce single batch tasks in Hadoop. This application provides a task scheduling simulation system.

2. Technical Solution

In order to achieve the above objective, this application provides a task scheduling simulation system, which includes a data preprocessing subsystem and a task scheduling subsystem.

The data preprocessing subsystem is used to filter the input cloud computing log information for abnormal data and extract the running time of each task.

The task scheduling subsystem is used to enqueue or dequeue tasks from the batch task and real-time task running queues of each node, and keep the number and status of tasks currently running in the cluster consistent with the actual production environment, and update the number of CPU cores and the used and available memory capacity of each node according to resource requirement of each task, to obtain the latest topology of the overall cluster resource utilization.

Optionally, the data preprocessing subsystem includes a data exception and missing processing module, a task information counting module, a resource demand counting module, and a running time counting module.

The data exception and missing processing module is used to read the native cloud computing cluster operation log to exclude abnormal data.

The task information counting module is used to count the task information and the number of task instances of each submitted job.

The resource demand counting module is used to count the total required number of CPU cores and the total required capacity of memory for each job.

The running time counting module is used to calculate the running time of each task instance, and to counting the estimated running time of each job.

Optionally, the task information counting module, the resource demand counting module and the running time counting module may simultaneously start3threads for parallel processing.

Optionally, the task scheduling subsystem includes a task operation information processing unit, a control unit, and a machine node event processing unit.

The task operation information processing unit includes a task operation information loading module, a task event driving module and a task scheduling algorithm module.

The control unit includes a simulator operation control module and a machine node resource information counting and collecting module.

The machine node event processing unit includes a machine node event information module and a machine node event driving module.

Optionally, the task event driving module includes a batch task event driving submodule and an online task event driving submodule.

Optionally, the machine node event information module includes a node adding submodule and a node deleting submodule.

Optionally, the machine node event driving module includes a hash table.

3. Beneficial Effects

Compared with the prior art, the beneficial effects of the task scheduling simulation system provided by this application are as follows.

The task scheduling simulation system provided by this application realizes the mixed scheduling simulation of batch tasks and online tasks by setting the data preprocessing subsystem and task scheduling subsystem, and can also simulates the resource of the CPU core number and memory capacity of cluster nodes heterogeneity.

