<commit_before>---
title: Migration Enhancements
category: feature
authors: mpolednik, mskrivan, ofrenkel, sandrobonazzola, tjelinek
wiki_category: Feature|Migration Enhancements
wiki_title: Features/Migration Enhancements
wiki_revision_count: 112
wiki_last_updated: 2015-09-14
---

# Migration Enhancements

## Summary

There are two main problems with the current migrations:

1.  All the policies are in VDSM, are simple and not tunable
2.  The migration bandwidth is set during initialization of migration and is not adjusted during migration.

The proposed solution to enhance issue #1 is to remove all the policies handling migrations from VDSM and move them to oVirt engine.
Engine will then expose couple of well defined and well described policies, from which the user will be able to pick the specific one per cluster, with an option to override it per VM.
Engine will then monitor the migration and change the parameters of the migration according to the defined policy.
The policies will differ in aggressiveness and in safety (e.g. switch to post-copy which guarantees that the VM will be moved and start working on the destination with the risk that it may fail to complete later). For more details about post-copy migration see <https://en.wikipedia.org/wiki/Live_migration#Post-copy_memory_migration>
The proposed solution to enhance issue #2 is to take into account the current situation (max bandwidth, num of migrating VMs)
while calculating the max bandwidth of the specific VM, and to keep adjusting it.
The logic of calculating the adjustments will be done in engine.

## Owner

*   Name: [<NAME>](User:TJelinek)
*   Email: <<EMAIL>>

## VDSM Changes

### Policies

Currently the policies handling migrations are in VDSM - the monitor thread which aborts a migration after a certain time of stalling and the downtime thread which is enlarging the downtime. The proposal is to make the VDSM to fire events on migration stalling and to expose the settings of the migration parameters during migration. This way the engine will be able to implement number of different policies and also to expose the creation of the policies to user.

*   Enrich the migrate verb so it will contain the following parameters
    -   Current parameters:
        -   **dst**: remote host or hibernation image filename
        -   **dstparams**: hibernation image filename for VDSM parameters
        -   **mode** remote/file
        -   **method**: online
        -   **downtime**: allowed down time during online migration
        -   **consoleAddress**: remote host graphics address
        -   **dstqemu**: remote host address dedicated for migration
        -   **compressed**: compress repeated pages during live migration (XBZRLE)
        -   **autoConverge**: force convergence during live migration
    -   Newly proposed:
        -   **migrationProgressTimeout**: a hard limit of migration progress (the timeout after which VDSM aborts migration even no other commands from engine arrives. This acts as a hard limit which will abort the migration in case the connection between engine and VDSM is lost for a long time so the engine policies will not apply). Optional argument, default: migration_progress_timeout from conf
        -   **downtimesList**: list of downtimes calculated on engine. Optional argument, default: DowntimeThread. Its meaning is that when this value is set explicitly, the downtime thread is disabled and the engine wishes to take care of the downtime adjustments. The engine does not send only one downtime but a list of them (some kind of last will in case of the engine disappears). If the stalling event occurs, VDSM will pick the next downtime from the list and applies it. This way the VDSM will be able to converge even the engine disappears while all the logic of calculating the downtimes is still on engine. A reaction to the stalling event by engine may be to change the **downtimesList**. In case engine will be happy with the current downtime list, it will not adjust it.
        -   **endAction**: what to do on stalling event if no more downtimes specified in **downtimesList**. Possible values:
            -   **abort**: abort migration
            -   **postCopy**: change to post copy
        -   **stallingLimit**: initial value (if the migration will be stalling for this amount of time, VDSM will send an event to which the engine will listen to). Optional argument, default: 0 (e.g. disabled)
        -   **maxBandwidth**: the maximal bandwidth which can be used by migrations. Optional argument, default migration_max_bandwidth from conf. It is an absolute value and applies only to the current migration

<!-- -->

*   Add a new verb called **migrateChangeParams** with the following parameters:
    -   **vmID**: vm UUID
    -   **downtimesList**: new list of max downtimes
    -   **endAction**: what to do on stalling event if no more downtimes specified in **downtimesList**. Possible values:
        -   **abort**: abort migration
        -   **postCopy**: change to post copy
    -   **stallingLimit**: if the migration will be stalling for this amount of time, VDSM will send an event to which the engine will listen to
    -   **maxBandwidth**: the maximal bandwidth which can be used by migrations. Optional argument, default migration_max_bandwidth from conf

When this verb will be called, the VDSM will store the new "last will" of the engine (e.g. the downtimeList, endAction and stallingLimit) and will apply the new value from "downtimeList" immediately. It is an absolute value and applies only to the current migration

#### Example Flow

Engine will decide to start a new migration so it will send the VDSM a migrate verb. According to the max downtime and the selected policy it will send a migrate verb which will contain:

*   **downtimesList**: [20, 40, 100, 120, 300, 500] - in milliseconds
*   **stallingLimit**: 5 seconds
*   **endAction**: abort

Now:

*   Migration will start, downtime will be set to 20ms (first value) and will be proceeding.
*   At some point it starts stalling.
*   When it will be stalling for 5 seconds (**stallingLimit**) VDSM will send an event to the engine and set the downtime to 40ms (second from the list)
*   Engine will grab the event and evaluate the situation
*   Lets say it will be happy with the current setting and will not change it (e.g. do nothing)
*   Migration will be stalling again for 5 seconds, VDSM will move to 100ms and send a message to engine
*   For example a new outgoing migration started in meanwhile, it means the bandwidth for the current migration is lower and the current downtime list is not correct since it is not possible to converge on the current bandwidth and downtime, so it calculates a new list where the minimum will be the first value the VM actually can converge, say it is 300. It will send a new list (for example like this): [300, 350, 400, 500]. Note that there are 4 new values since there were 4 values left in the original list.
*   engine will call the **migrateChangeParams** with this new list
*   VDSM will immediately apply the new downtime (first from list - 300) and will replace the current list of downtimes by the new one
*   lets say from now nothing changes and VDSM will eat up all the values from downtimesList and will still be stalling<commit_msg>Updated Migration Enhancements<commit_after>---
title: Migration Enhancements
category: feature
authors: mpolednik, mskrivan, ofrenkel, sandrobonazzola, tjelinek
wiki_category: Feature|Migration Enhancements
wiki_title: Features/Migration Enhancements
wiki_revision_count: 112
wiki_last_updated: 2015-09-14
---

# Migration Enhancements

## Summary

There are two main problems with the current migrations:

1.  All the policies are in VDSM, are simple and not tunable
2.  The migration bandwidth is set during initialization of migration and is not adjusted during migration.

The proposed solution to enhance issue #1 is to remove all the policies handling migrations from VDSM and move them to oVirt engine.
Engine will then expose couple of well defined and well described policies, from which the user will be able to pick the specific one per cluster, with an option to override it per VM.
Engine will then monitor the migration and change the parameters of the migration according to the defined policy.
The policies will differ in aggressiveness and in safety (e.g. switch to post-copy which guarantees that the VM will be moved and start working on the destination with the risk that it may fail to complete later). For more details about post-copy migration see <https://en.wikipedia.org/wiki/Live_migration#Post-copy_memory_migration>
The proposed solution to enhance issue #2 is to take into account the current situation (max bandwidth, num of migrating VMs)
while calculating the max bandwidth of the specific VM, and to keep adjusting it.
The logic of calculating the adjustments will be done in engine.

## Owner

*   Name: [<NAME>](User:TJelinek)
*   Email: <<EMAIL>>

## VDSM Changes

### Policies

Currently the policies handling migrations are in VDSM - the monitor thread which aborts a migration after a certain time of stalling and the downtime thread which is enlarging the downtime. The proposal is to make the VDSM to fire events on migration stalling and to expose the settings of the migration parameters during migration. This way the engine will be able to implement number of different policies and also to expose the creation of the policies to user.

*   Enrich the migrate verb so it will contain the following parameters
    -   Current parameters:
        -   **dst**: remote host or hibernation image filename
        -   **dstparams**: hibernation image filename for VDSM parameters
        -   **mode** remote/file
        -   **method**: online
        -   **downtime**: allowed down time during online migration
        -   **consoleAddress**: remote host graphics address
        -   **dstqemu**: remote host address dedicated for migration
        -   **compressed**: compress repeated pages during live migration (XBZRLE)
        -   **autoConverge**: force convergence during live migration
    -   Newly proposed:
        -   **migrationProgressTimeout**: a hard limit of migration progress (the timeout after which VDSM aborts migration even no other commands from engine arrives. This acts as a hard limit which will abort the migration in case the connection between engine and VDSM is lost for a long time so the engine policies will not apply). Optional argument, default: migration_progress_timeout from conf
        -   **maxBandwidth**: the maximal bandwidth which can be used by migrations. Optional argument, default migration_max_bandwidth from conf. It is an absolute value and applies only to the current migration
        -   **convergenceSchedule**: list of pairs: (stallingLimit, action) where
            -   **stallingLimit**: if the migration is stalling for this amount of time, execute the action and move to next pair
            -   **action**: one of:
                -   **setDowntime(N)**: sets the downtime to N
                -   **abort**: abort migration
                -   **postCopy**: change to post copy

An example how the **convergenceSchedule** would look like: [ (10, setDowntime(10)), (20, setDowntime(30)), (20, setDowntime(60)), (50, abort) ]

The behavior of the VDSM in this case will be as follows:

*   Starts the migration with the **downtime** from the current parameters
*   In the monitor thread monitors the migration
*   If the migration progresses, does nothing, just keeps monitoring
*   If the migration starts stalling for more than 10 seconds, executes the action setDowntime 10 (e.g. sets the downtime to 10ms)
*   If the migration stalls for another 20 seconds, sets the timeout to 30
*   If the migration stalls for another 20 seconds, sets the timeout to 60
*   If the migration stalls for another 50 seconds aborts the migration

<!-- -->

*   Add a new verb called **migrateChangeParams** with the following parameters:
    -   **vmID**: vm UUID
    -   **convergenceSchedule**: list of pairs: (stallingLimit, action) where
    -   **maxBandwidth**: the maximal bandwidth which can be used by migrations. Optional argument, default migration_max_bandwidth from conf

When this verb will be called, the VDSM will store the new "last will" of the engine (e.g. the **convergenceSchedule**) and apply the new **maxBandwidth** immediately.

#### Example Flow

Engine will decide to start a new migration so it will send the VDSM a migrate verb. According to the max downtime and the selected policy it will send a migrate verb which will contain:

*   **downtimesList**: [20, 40, 100, 120, 300, 500] - in milliseconds
*   **stallingLimit**: 5 seconds
*   **endAction**: abort

Now:

*   Migration will start, downtime will be set to 20ms (first value) and will be proceeding.
*   At some point it starts stalling.
*   When it will be stalling for 5 seconds (**stallingLimit**) VDSM will send an event to the engine and set the downtime to 40ms (second from the list)
*   Engine will grab the event and evaluate the situation
*   Lets say it will be happy with the current setting and will not change it (e.g. do nothing)
*   Migration will be stalling again for 5 seconds, VDSM will move to 100ms and send a message to engine
*   For example a new outgoing migration started in meanwhile, it means the bandwidth for the current migration is lower and the current downtime list is not correct since it is not possible to converge on the current bandwidth and downtime, so it calculates a new list where the minimum will be the first value the VM actually can converge, say it is 300. It will send a new list (for example like this): [300, 350, 400, 500]. Note that there are 4 new values since there were 4 values left in the original list.
*   engine will call the **migrateChangeParams** with this new list
*   VDSM will immediately apply the new downtime (first from list - 300) and will replace the current list of downtimes by the new one
*   lets say from now nothing changes and VDSM will eat up all the values from downtimesList and will still be stalling
{'fraction_non_alphanumeric': 0.055794293765410355, 'fraction_numerical': 0.010778443113772455, 'mean_word_length': 3.843398157625384, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6193100', 'n_tokens_mistral': 3660, 'n_tokens_neox': 3427, 'n_words': 2054}