FIELD OF THE INVENTION

The invention generally relates to the field of data processing and in particular migration of databases.

BACKGROUND

Migrating (i.e., copying) data between computer systems is a common task for information technology (IT) organizations. Data may be migrated on a recurring basis between a production system and a development system, or as part of a backup or data warehousing operation. Migration may also be required when a system is moved to a new data center or to a cloud-based facility. Bottlenecks during the migration process frequently lead to the need for manual administrator intervention.

SUMMARY

Embodiments of the present invention provide a system and methods, executed by a system of distributed processors, for migrating data from a source database of a source host to a target database of a target host, including: executing a first number of extraction components on the source host, the first number being limited by an extraction threshold parameter, wherein each extraction component transfers a portion of the data as a data package from the source database to a source host transient memory; executing a second number of send components on the source host, the second number being limited by a send threshold parameter, wherein each send component transmits a data package from the source host transient memory, as stored by an extraction component, to a receive component on the target host, and wherein each receive component transfers a data package received from a send component to a target host transient memory; executing a third number of store components on the target host, the third number being limited by a store threshold parameter, and wherein each store component transfers to the target database a data package from target host transient memory, as stored by a receive component; and executing a monitor component to monitor performance of the source database, the source host transient memory, the target host transient memory, and the target database, and responsively, according to a pre-defined state table, changing at least one of the number of extraction, send, and store components.

In some embodiments, states specified in the predefined state table may be defined as being less than or greater than a given value for a maximum target performance. Alternatively or additionally, states specified in the predefined state table may be defined as being less than a target minimum performance, within an optimal performance range, or greater than a maximum performance Changing the number of extraction components, send, or store components may include changing at least one of the respective extraction, send or store threshold parameters.

The first number of extraction components may include two or more components that operate in parallel to transfer chunks of data from a chunk queue to the source local map. Alternatively or additionally, the first number of extraction components may include two or more components operating in parallel to select data from a single table of the source database, allocating the data according to a modulo (“MOD”) parameter. The data may include one or more of table rows, table columns, collections, stored procedures, indexes, and views.

In further embodiments, the monitor component may be further configured to receive and store status indicators of package progress from the source database to the source host transient memory, to the target host transient memory, and to the target database, in order to perform recovery in the event of a component or network failure by resending a lost package.

Embodiments of the present invention further provide a system configured for migrating data from a source database of a source host to a target database of a target host. The system may include the source host, having a processor and memory including instructions that when executed on the processor perform steps of: executing a first number of extraction components, wherein the first number is limited by an extraction threshold parameter, and wherein each extraction component transfers a portion of the data as a data package from the source database to a source host transient memory; executing a second number of send components, wherein the second number is limited by a send threshold parameter, wherein each send component transmits a data package from the source host transient memory, as stored by an extraction component of the source host, to a receive component on the target host, and wherein each receive component transfers a data package received from a send component to a target host transient memory. The system may also include the target host, having a processor and memory including instructions that when executed on the processor perform steps of: executing a third number of store components, wherein the third number is limited by a store threshold parameter, and wherein each store component transfers to the target database a data package from target host transient memory, as stored by the receive component of the target host. The system may also include a manager host, comprising a processor and memory including instructions that when executed on the processor perform steps of: executing a monitor component to monitor performance of the source database, the source host transient memory, the target host transient memory, and the target database, and responsively, according to a pre-defined state table, changing at least one of the number of extraction, send, and store components.

