<commit_before>========
Upgrades
========

With the exception of Object Storage, upgrading from one version of
OpenStack to another can take a great deal of effort. This chapter
provides some guidance on the operational aspects that you should
consider for performing an upgrade for an OpenStack environment.

Pre-upgrade considerations
~~~~~~~~~~~~~~~~~~~~~~~~~~

Upgrade planning
----------------

-  Thoroughly review the `release
   notes <http://releases.openstack.org/>`_ to learn
   about new, updated, and deprecated features. Find incompatibilities
   between versions.

-  Consider the impact of an upgrade to users. The upgrade process
   interrupts management of your environment including the dashboard. If
   you properly prepare for the upgrade, existing instances, networking,
   and storage should continue to operate. However, instances might
   experience intermittent network interruptions.

-  Consider the approach to upgrading your environment. You can perform
   an upgrade with operational instances, but this is a dangerous
   approach. You might consider using live migration to temporarily
   relocate instances to other compute nodes while performing upgrades.
   However, you must ensure database consistency throughout the process;
   otherwise your environment might become unstable. Also, don't forget
   to provide sufficient notice to your users, including giving them
   plenty of time to perform their own backups.

-  Consider adopting structure and options from the service
   configuration files and merging them with existing configuration
   files. The `OpenStack Configuration
   Reference <http://docs.openstack.org/newton/config-reference/>`_
   contains new, updated, and deprecated options for most services.

-  Like all major system upgrades, your upgrade could fail for one or
   more reasons. You can prepare for this situation by having the
   ability to roll back your environment to the previous release,
   including databases, configuration files, and packages. We provide an
   example process for rolling back your environment in
   :ref:`rolling_back_a_failed_upgrade`.

-  Develop an upgrade procedure and assess it thoroughly by using a test
   environment similar to your production environment.

Pre-upgrade testing environment
-------------------------------

The most important step is the pre-upgrade testing. If you are upgrading
immediately after release of a new version, undiscovered bugs might
hinder your progress. Some deployers prefer to wait until the first
point release is announced. However, if you have a significant
deployment, you might follow the development and testing of the release
to ensure that bugs for your use cases are fixed.

Each OpenStack cloud is different even if you have a near-identical
architecture as described in this guide. As a result, you must still
test upgrades between versions in your environment using an approximate
clone of your environment.

However, that is not to say that it needs to be the same size or use
identical hardware as the production environment. It is important to
consider the hardware and scale of the cloud that you are upgrading. The
following tips can help you minimise the cost:

Use your own cloud
    The simplest place to start testing the next version of OpenStack is
    by setting up a new environment inside your own cloud. This might
    seem odd, especially the double virtualization used in running
    compute nodes. But it is a sure way to very quickly test your
    configuration.

Use a public cloud
    Consider using a public cloud to test the scalability limits of your
    cloud controller configuration. Most public clouds bill by the hour,
    which means it can be inexpensive to perform even a test with many
    nodes.

Make another storage endpoint on the same system
    If you use an external storage plug-in or shared file system with
    your cloud, you can test whether it works by creating a second share
    or endpoint. This allows you to test the system before entrusting
    the new version on to your storage.

Watch the network
    Even at smaller-scale testing, look for excess network packets to
    determine whether something is going horribly wrong in
    inter-component communication.

To set up the test environment, you can use one of several methods:

-  Do a full manual install by using the `Installation Tutorials and Guides
   <http://docs.openstack.org/project-install-guide/newton/>`_ for
   your platform. Review the final configuration files and installed
   packages.

-  Create a clone of your automated configuration infrastructure with
   changed package repository URLs.

   Alter the configuration until it works.

Either approach is valid. Use the approach that matches your experience.

An upgrade pre-testing system is excellent for getting the configuration
to work. However, it is important to note that the historical use of the
system and differences in user interaction can affect the success of
upgrades.

If possible, we highly recommend that you dump your production database
tables and test the upgrade in your development environment using this
data. Several MySQL bugs have been uncovered during database migrations
because of slight table differences between a fresh installation and
tables that migrated from one version to another. This will have impact
on large real datasets, which you do not want to encounter during a
production outage.

Artificial scale testing can go only so far. After your cloud is
upgraded, you must pay careful attention to the performance aspects of
your cloud.

Upgrade Levels
--------------

Upgrade levels are a feature added to OpenStack Compute since the
Grizzly release to provide version locking on the RPC (Message Queue)
communications between the various Compute services.

This functionality is an important piece of the puzzle when it comes to
live upgrades and is conceptually similar to the existing API versioning
that allows OpenStack services of different versions to communicate
without issue.

Without upgrade levels, an X+1 version Compute service can receive and
understand X version RPC messages, but it can only send out X+1 version
RPC messages. For example, if a nova-conductor process has been upgraded
to X+1 version, then the conductor service will be able to understand
messages from X version nova-compute processes, but those compute
services will not be able to understand messages sent by the conductor
service.

During an upgrade, operators can add configuration options to
``nova.conf`` which lock the version of RPC messages and allow live
upgrading of the services without interruption caused by version
mismatch. The configuration options allow the specification of RPC
version numbers if desired, but release name alias are also supported.
For example:

.. code-block:: ini

   [upgrade_levels]
   compute=X+1
   conductor=X+1
   scheduler=X+1

will keep the RPC version locked across the specified services to the
RPC version used in X+1. As all instances of a particular service are
upgraded to the newer version, the corresponding line can be removed
from ``nova.conf``.

Using this functionality, ideally one would lock the RPC version to the
OpenStack version being upgraded from on nova-compute nodes, to ensure
that, for example X+1 version nova-compute processes will continue to
work with X version nova-conductor processes while the upgrade
completes. Once the upgrade of nova-compute processes is complete, the
operator can move onto upgrading nova-conductor and remove the version
locking for nova-compute in ``nova.conf``.

Upgrade process
~~~~~~~~~~~~~~~

This section describes the process to upgrade a basic OpenStack
deployment based on the basic two-node architecture in the `Installation
Tutorials and Guides
<http://docs.openstack.org/project-install-guide/newton/>`_. All
nodes must run a supported distribution of Linux with a recent kernel
and the current release packages.

Service specific upgrade instructions
-------------------------------------

Refer to the following upgrade notes for information on upgrading specific
OpenStack services:

* `Upgrading the Networking service
  <http://docs.openstack.org/developer/neutron/devref/upgrade.html>`_
* `Upgrading the Compute service
  <http://docs.openstack.org/developer/nova/upgrade.html>`_
* `Upgrading the Identity service
  <http://docs.openstack.org/developer/keystone/upgrading.html>`_

Prerequisites
-------------
<commit_msg>Add Cinder upgrade documentation link to ops-guide<commit_after>========
Upgrades
========

With the exception of Object Storage, upgrading from one version of
OpenStack to another can take a great deal of effort. This chapter
provides some guidance on the operational aspects that you should
consider for performing an upgrade for an OpenStack environment.

Pre-upgrade considerations
~~~~~~~~~~~~~~~~~~~~~~~~~~

Upgrade planning
----------------

-  Thoroughly review the `release
   notes <http://releases.openstack.org/>`_ to learn
   about new, updated, and deprecated features. Find incompatibilities
   between versions.

-  Consider the impact of an upgrade to users. The upgrade process
   interrupts management of your environment including the dashboard. If
   you properly prepare for the upgrade, existing instances, networking,
   and storage should continue to operate. However, instances might
   experience intermittent network interruptions.

-  Consider the approach to upgrading your environment. You can perform
   an upgrade with operational instances, but this is a dangerous
   approach. You might consider using live migration to temporarily
   relocate instances to other compute nodes while performing upgrades.
   However, you must ensure database consistency throughout the process;
   otherwise your environment might become unstable. Also, don't forget
   to provide sufficient notice to your users, including giving them
   plenty of time to perform their own backups.

-  Consider adopting structure and options from the service
   configuration files and merging them with existing configuration
   files. The `OpenStack Configuration
   Reference <http://docs.openstack.org/newton/config-reference/>`_
   contains new, updated, and deprecated options for most services.

-  Like all major system upgrades, your upgrade could fail for one or
   more reasons. You can prepare for this situation by having the
   ability to roll back your environment to the previous release,
   including databases, configuration files, and packages. We provide an
   example process for rolling back your environment in
   :ref:`rolling_back_a_failed_upgrade`.

-  Develop an upgrade procedure and assess it thoroughly by using a test
   environment similar to your production environment.

Pre-upgrade testing environment
-------------------------------

The most important step is the pre-upgrade testing. If you are upgrading
immediately after release of a new version, undiscovered bugs might
hinder your progress. Some deployers prefer to wait until the first
point release is announced. However, if you have a significant
deployment, you might follow the development and testing of the release
to ensure that bugs for your use cases are fixed.

Each OpenStack cloud is different even if you have a near-identical
architecture as described in this guide. As a result, you must still
test upgrades between versions in your environment using an approximate
clone of your environment.

However, that is not to say that it needs to be the same size or use
identical hardware as the production environment. It is important to
consider the hardware and scale of the cloud that you are upgrading. The
following tips can help you minimise the cost:

Use your own cloud
    The simplest place to start testing the next version of OpenStack is
    by setting up a new environment inside your own cloud. This might
    seem odd, especially the double virtualization used in running
    compute nodes. But it is a sure way to very quickly test your
    configuration.

Use a public cloud
    Consider using a public cloud to test the scalability limits of your
    cloud controller configuration. Most public clouds bill by the hour,
    which means it can be inexpensive to perform even a test with many
    nodes.

Make another storage endpoint on the same system
    If you use an external storage plug-in or shared file system with
    your cloud, you can test whether it works by creating a second share
    or endpoint. This allows you to test the system before entrusting
    the new version on to your storage.

Watch the network
    Even at smaller-scale testing, look for excess network packets to
    determine whether something is going horribly wrong in
    inter-component communication.

To set up the test environment, you can use one of several methods:

-  Do a full manual install by using the `Installation Tutorials and Guides
   <http://docs.openstack.org/project-install-guide/newton/>`_ for
   your platform. Review the final configuration files and installed
   packages.

-  Create a clone of your automated configuration infrastructure with
   changed package repository URLs.

   Alter the configuration until it works.

Either approach is valid. Use the approach that matches your experience.

An upgrade pre-testing system is excellent for getting the configuration
to work. However, it is important to note that the historical use of the
system and differences in user interaction can affect the success of
upgrades.

If possible, we highly recommend that you dump your production database
tables and test the upgrade in your development environment using this
data. Several MySQL bugs have been uncovered during database migrations
because of slight table differences between a fresh installation and
tables that migrated from one version to another. This will have impact
on large real datasets, which you do not want to encounter during a
production outage.

Artificial scale testing can go only so far. After your cloud is
upgraded, you must pay careful attention to the performance aspects of
your cloud.

Upgrade Levels
--------------

Upgrade levels are a feature added to OpenStack Compute since the
Grizzly release to provide version locking on the RPC (Message Queue)
communications between the various Compute services.

This functionality is an important piece of the puzzle when it comes to
live upgrades and is conceptually similar to the existing API versioning
that allows OpenStack services of different versions to communicate
without issue.

Without upgrade levels, an X+1 version Compute service can receive and
understand X version RPC messages, but it can only send out X+1 version
RPC messages. For example, if a nova-conductor process has been upgraded
to X+1 version, then the conductor service will be able to understand
messages from X version nova-compute processes, but those compute
services will not be able to understand messages sent by the conductor
service.

During an upgrade, operators can add configuration options to
``nova.conf`` which lock the version of RPC messages and allow live
upgrading of the services without interruption caused by version
mismatch. The configuration options allow the specification of RPC
version numbers if desired, but release name alias are also supported.
For example:

.. code-block:: ini

   [upgrade_levels]
   compute=X+1
   conductor=X+1
   scheduler=X+1

will keep the RPC version locked across the specified services to the
RPC version used in X+1. As all instances of a particular service are
upgraded to the newer version, the corresponding line can be removed
from ``nova.conf``.

Using this functionality, ideally one would lock the RPC version to the
OpenStack version being upgraded from on nova-compute nodes, to ensure
that, for example X+1 version nova-compute processes will continue to
work with X version nova-conductor processes while the upgrade
completes. Once the upgrade of nova-compute processes is complete, the
operator can move onto upgrading nova-conductor and remove the version
locking for nova-compute in ``nova.conf``.

Upgrade process
~~~~~~~~~~~~~~~

This section describes the process to upgrade a basic OpenStack
deployment based on the basic two-node architecture in the `Installation
Tutorials and Guides
<http://docs.openstack.org/project-install-guide/newton/>`_. All
nodes must run a supported distribution of Linux with a recent kernel
and the current release packages.

Service specific upgrade instructions
-------------------------------------

Refer to the following upgrade notes for information on upgrading specific
OpenStack services:

* `Upgrading the Networking service
  <http://docs.openstack.org/developer/neutron/devref/upgrade.html>`_
* `Upgrading the Compute service
  <http://docs.openstack.org/developer/nova/upgrade.html>`_
* `Upgrading the Identity service
  <http://docs.openstack.org/developer/keystone/upgrading.html>`_
* `Upgrading the Block Storage service
  <http://docs.openstack.org/developer/cinder/upgrade.html>`_

Prerequisites
-------------

{'fraction_non_alphanumeric': 0.052050473186119876, 'fraction_numerical': 0.0009346886318495151, 'mean_word_length': 4.111675126903553, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 18, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2540458', 'n_tokens_mistral': 3996, 'n_tokens_neox': 3817, 'n_words': 2346}