<commit_before>---
title: Decommission Master Domain and SPM
category: feature
authors: aglitke, fsimonce
wiki_category: Feature|Decommission_Master_Domain_and_SPM
wiki_title: Features/Decommission Master Domain and SPM
wiki_revision_count: 19
wiki_last_updated: 2015-04-16
wiki_conversion_fallback: true
wiki_warnings: conversion-fallback
---

# Decommission Master Domain and SPM

### Summary

By introducing advanced flows (live snapshots, live storage migration, live merge) engine orchestration and VDSM commands implementation grew in complexity because of the SPM-HSM roles differences. In addition the concept of Storage Pool (initially introduced to simplify cross-domain operations) became an obstacle in managing and scaling storage domains.

The goal of this feature is to remove the SPM role and the Storage Pool concept from the oVirt storage subsystem. As a consequence the Master Domain and its special content will disappear as well (pool metadata, asynchronous tasks persistency and ovf backups) simplifing the implementation and getting rid of several known problems. A set of [New Storage API](Features/Decommission_Master_Domain_and_SPM#New_Storage_API) will be introduced to address the new architecture (no task persistency and short-lived sanlock metadata lock).

More information:

*   Remove Master Domain and File-System: stop designating a master domain and maintaining a file-system for asynchronous tasks peristency and ovf backups
    -   [Decommission Storage Pool Metadata](Features/StoragePool_Metadata_Removal)
    -   [Store OVF on Regular Volumes](Feature/OvfOnWantedDomains)
    -   Drop task persistency with the [New Storage API](Features/Decommission_Master_Domain_and_SPM#New_Storage_API)
*   Remove SPM: the SPM role is replaced with a short-lived domain metadata lock on the relevant storage domain(s)
    -   [New Storage API](Features/Decommission_Master_Domain_and_SPM#New_Storage_API)

### Owner

*   Name: [<NAME>](user:Fsimonce)
*   Email: <<EMAIL>>
*   Name: [<NAME>](user:Laravot)
*   Email: <<EMAIL>>
*   Name: [<NAME>](user:Aglitke)
*   Email: <<EMAIL>>
*   PM Requirements : <NAME>
*   Email: <<EMAIL>>

### Current status

*   Target Release: 3.6
*   Status: under design and discussion.

### Background

### Connect Storage Pool

     connectStoragePool(spUUID, hostID, msdUUID, masterVersion, domainsMap)

For backward compatibility with running VMs we must maintain the volume links in /rhev/data-center (and also to support live migration without any additional change). The only purpose to keep this API is to communicate the list of domains to monitor and the spUUID used to maintain the symlinks (/rhev/data-center/spUUID).

VDSM will allow a connectStoragePool request using a blank msdUUID:

     msdUUID = 00000000-0000-0000-0000-000000000000
     masterVersion = 0

In this case:

*   connectStoragePool with blank msdUUID will fail if the host is the SPM (incompatible with this request)
*   in this state the host will refuse to become an SPM
*   the mailbox won't be active (sending/receiving)
*   all the major flows involving the master domain will be prevented (masterMigrate, reconstructMaster, etc.)
*   the refresh flow will still be valid and the links will be maintained (except for the masterd symlink)

This API may be removed in the future but at the moment it's not strictly related to (nor required by) the Master Domain and SPM removal. In order to allow incremental achievable changes connectStoragePool will be addressed in the future with a solution that also covers running VMs.

### New Task Infrastructure and the taskUUID Argument

The old SPM verbs are tightly coupled with the Asynchronous Tasks persisted on the master domain. In fact all the volume operations (creation, deletion, etc.) are updating their status in the master domain in order to rollback when interrupted. The new tasks infrastructure won't provide any persistency for the ongoing Asynchronous Tasks, therefore the new storage API should be designed in such a way that all operations when interrupted should leave garbage that is easy to identify and clean up.

Relevant RFE:

*   <https://bugzilla.redhat.com/show_bug.cgi?id=1080379>
*   <https://bugzilla.redhat.com/show_bug.cgi?id=1080384>

In the future when JSON-RPC will be introduced all the incoming requests from the engine will automatically have an UUID that can be used to identify asynchronous operations.

### New Storage API

Goals of the new storage API are:

*   **Splitting Metadata and Data Operations** (allowing to decouple short metadata changes and long data operations)
*   **Replace SPM with a Short-Lived Domain Metadata Lock** (non-atomic metadata changes will require a domain metadata lock)
*   **Consolidate API** (API should be minimal and allow composition for complex flows)
*   **No Task Peristency** (storage operations must drop the old tasks persistency)
*   **Garbage Collection** (leftovers of partial operations should be easy to identify and collect)

The old API commands that should be re-implemented are:

VDSM API Changes

New API

cloneImageStructure

multiple createVolumeContainer

copyImage

multiple createVolumeContainer and copyVolumeData (similar to cloneImageStructure)

createVolume

createVolumeContainer + allocateVolume (to preallocate a volume when needed)

deleteImage

multiple removeVolume + wipeVolume (to post-zero the volume)

deleteVolume

removeVolume

downloadImage

createVolumeContainer + copyVolumeData

downloadImageFromStream

createVolumeContainer + copyVolumeData

extendVolumeSize

extendVolumeSize

mergeSnapshots

createVolumeContainer + mergeSnapshotsV2

moveImage

N/A (to be removed, engine will use copyImage + deleteImage)

moveMultipleImages

N/A (to be removed, engine will use copyImage + deleteImage)

syncImageData

multiple copyVolumeData

uploadImage

copyVolumeData

#### Create Volume

     createVolumeContainer(sdUUID, imgUUID, size, volFormat, diskType, volUUID, desc, srcImgUUID, srcVolUUID)

**Parameters:**

*   **sdUUID**, **imgUUID**, **volUUID**: domain, image and volume uuids
*   **size**: The intended capacity of the new volume (in sectors)
*   **volFormat**: Request either cow (4) or raw (5)
*   **diskType**: (Legacy) Set the type of disk to be created: unknown (0), system (1), data (2), shared (3), swap (4), temporary (5)
*   **desc**: Set a volume description
*   **srcImgUUID**, **srcVolUUID**: parent image and volume uuids

This (synchronous) API will allow to create a volume in a storage domain. The operation mostly involves metadata changes and it should leave the volume fully prepared and ready to be used or in an inconsistent way easily identifiable from the garbage collector.

Overview of the flow on block domains:

*   create a new volume (marked as not-ready)
*   allocate the new volume metadata and lease area
*   initialize the qcow2 format (when needed)
*   mark the parent as internal volume
*   mark the volume as ready

Overview of the flow on file domains:

*   prepare a new image directory (when needed)
*   hard-link the template volume, metadata and lease files (when needed)
*   create a new volume (marked as not-ready)
*   create the new volume metadata and lease files
*   initialize the qcow2 format (when needed)
*   mark the parent as internal volume
*   mark the volume as ready

**Completion check**: on success getVolumeInfo will return the new volume info (on failure VolumeDoesNotExist will be raised)

Garbage collection (for unfinished volumes):

*   for files use two simple garbage collectors
*   volumes that are marked as not-ready will be removed
*   unmark the parent as internal volume (when needed)

#### Delete Volume

     removeVolume(sdUUID, imgUUID, volUUID)

**Parameters:**

*   **sdUUID**, **imgUUID**, **volUUID**: domain, image and volume uuids

This (synchronous) API will allow to delete a volume in a storage domain. It will allow to remove leaf volumes and volumes that have no relevant data (e.g. live merged).

Overview of the flow on file and block domains:
<commit_msg>Delete Volume<commit_after>---
title: Decommission Master Domain and SPM
category: feature
authors: aglitke, fsimonce
wiki_category: Feature|Decommission_Master_Domain_and_SPM
wiki_title: Features/Decommission Master Domain and SPM
wiki_revision_count: 19
wiki_last_updated: 2015-04-16
wiki_conversion_fallback: true
wiki_warnings: conversion-fallback
---

# Decommission Master Domain and SPM

### Summary

By introducing advanced flows (live snapshots, live storage migration, live merge) engine orchestration and VDSM commands implementation grew in complexity because of the SPM-HSM roles differences. In addition the concept of Storage Pool (initially introduced to simplify cross-domain operations) became an obstacle in managing and scaling storage domains.

The goal of this feature is to remove the SPM role and the Storage Pool concept from the oVirt storage subsystem. As a consequence the Master Domain and its special content will disappear as well (pool metadata, asynchronous tasks persistency and ovf backups) simplifing the implementation and getting rid of several known problems. A set of [New Storage API](Features/Decommission_Master_Domain_and_SPM#New_Storage_API) will be introduced to address the new architecture (no task persistency and short-lived sanlock metadata lock).

More information:

*   Remove Master Domain and File-System: stop designating a master domain and maintaining a file-system for asynchronous tasks peristency and ovf backups
    -   [Decommission Storage Pool Metadata](Features/StoragePool_Metadata_Removal)
    -   [Store OVF on Regular Volumes](Feature/OvfOnWantedDomains)
    -   Drop task persistency with the [New Storage API](Features/Decommission_Master_Domain_and_SPM#New_Storage_API)
*   Remove SPM: the SPM role is replaced with a short-lived domain metadata lock on the relevant storage domain(s)
    -   [New Storage API](Features/Decommission_Master_Domain_and_SPM#New_Storage_API)

### Owner

*   Name: [<NAME>](user:Fsimonce)
*   Email: <<EMAIL>>
*   Name: [<NAME>](user:Laravot)
*   Email: <<EMAIL>>
*   Name: [<NAME>](user:Aglitke)
*   Email: <<EMAIL>>
*   PM Requirements : <NAME>
*   Email: <<EMAIL>>

### Current status

*   Target Release: 3.6
*   Status: under design and discussion.

### Background

### Connect Storage Pool

     connectStoragePool(spUUID, hostID, msdUUID, masterVersion, domainsMap)

For backward compatibility with running VMs we must maintain the volume links in /rhev/data-center (and also to support live migration without any additional change). The only purpose to keep this API is to communicate the list of domains to monitor and the spUUID used to maintain the symlinks (/rhev/data-center/spUUID).

VDSM will allow a connectStoragePool request using a blank msdUUID:

     msdUUID = 00000000-0000-0000-0000-000000000000
     masterVersion = 0

In this case:

*   connectStoragePool with blank msdUUID will fail if the host is the SPM (incompatible with this request)
*   in this state the host will refuse to become an SPM
*   the mailbox won't be active (sending/receiving)
*   all the major flows involving the master domain will be prevented (masterMigrate, reconstructMaster, etc.)
*   the refresh flow will still be valid and the links will be maintained (except for the masterd symlink)

This API may be removed in the future but at the moment it's not strictly related to (nor required by) the Master Domain and SPM removal. In order to allow incremental achievable changes connectStoragePool will be addressed in the future with a solution that also covers running VMs.

### New Task Infrastructure and the taskUUID Argument

The old SPM verbs are tightly coupled with the Asynchronous Tasks persisted on the master domain. In fact all the volume operations (creation, deletion, etc.) are updating their status in the master domain in order to rollback when interrupted. The new tasks infrastructure won't provide any persistency for the ongoing Asynchronous Tasks, therefore the new storage API should be designed in such a way that all operations when interrupted should leave garbage that is easy to identify and clean up.

Relevant RFE:

*   <https://bugzilla.redhat.com/show_bug.cgi?id=1080379>
*   <https://bugzilla.redhat.com/show_bug.cgi?id=1080384>

In the future when JSON-RPC will be introduced all the incoming requests from the engine will automatically have an UUID that can be used to identify asynchronous operations.

### New Storage API

Goals of the new storage API are:

*   **Splitting Metadata and Data Operations** (allowing to decouple short metadata changes and long data operations)
*   **Replace SPM with a Short-Lived Domain Metadata Lock** (non-atomic metadata changes will require a domain metadata lock)
*   **Consolidate API** (API should be minimal and allow composition for complex flows)
*   **No Task Peristency** (storage operations must drop the old tasks persistency)
*   **Garbage Collection** (leftovers of partial operations should be easy to identify and collect)

The old API commands that should be re-implemented are:

VDSM API Changes

New API

cloneImageStructure

multiple createVolumeContainer

copyImage

multiple createVolumeContainer and copyVolumeData (similar to cloneImageStructure)

createVolume

createVolumeContainer + allocateVolume (to preallocate a volume when needed)

deleteImage

multiple removeVolume + wipeVolume (to post-zero the volume)

deleteVolume

removeVolume

downloadImage

createVolumeContainer + copyVolumeData

downloadImageFromStream

createVolumeContainer + copyVolumeData

extendVolumeSize

extendVolumeSize

mergeSnapshots

createVolumeContainer + mergeSnapshotsV2

moveImage

N/A (to be removed, engine will use copyImage + deleteImage)

moveMultipleImages

N/A (to be removed, engine will use copyImage + deleteImage)

syncImageData

multiple copyVolumeData

uploadImage

copyVolumeData

#### Create Volume

     createVolumeContainer(sdUUID, imgUUID, size, volFormat, diskType, volUUID, desc, srcImgUUID, srcVolUUID)

**Parameters:**

*   **sdUUID**, **imgUUID**, **volUUID**: domain, image and volume uuids
*   **size**: The intended capacity of the new volume (in sectors)
*   **volFormat**: Request either cow (4) or raw (5)
*   **diskType**: (Legacy) Set the type of disk to be created: unknown (0), system (1), data (2), shared (3), swap (4), temporary (5)
*   **desc**: Set a volume description
*   **srcImgUUID**, **srcVolUUID**: parent image and volume uuids

This (synchronous) API will allow to create a volume in a storage domain. The operation mostly involves metadata changes and it should leave the volume fully prepared and ready to be used or in an inconsistent way easily identifiable from the garbage collector.

Overview of the flow on block domains:

*   create a new volume (marked as not-ready)
*   allocate the new volume metadata and lease area
*   initialize the qcow2 format (when needed)
*   mark the parent as internal volume
*   mark the volume as ready

Overview of the flow on file domains:

*   prepare a new image directory (when needed)
*   hard-link the template volume, metadata and lease files (when needed)
*   create a new volume (marked as not-ready)
*   create the new volume metadata and lease files
*   initialize the qcow2 format (when needed)
*   mark the parent as internal volume
*   mark the volume as ready

**Completion check**: on success getVolumeInfo will return the new volume info (on failure VolumeDoesNotExist will be raised)

Garbage collection (for unfinished volumes):

*   for files use two simple garbage collectors
*   volumes that are marked as not-ready will be removed
*   unmark the parent as internal volume (when needed)

#### Delete Volumes

     removeVolumes(sdUUID, imgUUID, volumeList)

**Parameters:**

*   **sdUUID**, **imgUUID**, **volumeList**: domain UUID, image UUID, and volume UUID list

This (synchronous) API will allow to delete one or more volumes from an image. The volume list must be in ascendant order (beginning with the leaf and including additional consecutive volumes as desired).

Overview of the flow on file and block domains:

{'fraction_non_alphanumeric': 0.06211372064276885, 'fraction_numerical': 0.00865265760197775, 'mean_word_length': 4.812140804597701, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 31, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 4, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1084562', 'n_tokens_mistral': 4413, 'n_tokens_neox': 4222, 'n_words': 2047}