<commit_before>// Module included in the following assemblies:
//
// * disaster_recovery/scenario-2-restoring-cluster-state.adoc

[id="dr-scenario-2-restoring-cluster-state_{context}"]
= Restoring to a previous cluster state

You can use a saved etcd backup to restore back to a previous cluster state. You use the etcd backup to restore a single master host. Then the etcd cluster Operator handles scaling to the remaining master hosts.

.Prerequisites

* Access to the cluster as a user with the `cluster-admin` role.
* SSH access to master hosts.
* A backup directory containing both the etcd snapshot and the resources for the static Pods, which were from the same backup. The file names in the directory must be in the following formats: `snapshot_<datetimestamp>.db` and `static_kuberesources_<datetimestamp>.tar.gz`.

.Procedure

. Select a master host to use as the recovery host. This is the host that you will run the restore operation on.

. Establish SSH connectivity to each of the master nodes, including the recovery host.
+
The Kubernetes API server will become inaccessible once the restore process has started, so you cannot access the master nodes. For this reason, it is recommended to establish SSH connectivity to each master host in a separate terminal.
+
[IMPORTANT]
====
If you do not complete this step, you will not be able to access the recovery host to complete the restore procedure, and you will be unable to recover your cluster from this state.
====

. Copy the etcd backup directory to the recovery master host.
+
This procedure assumes that you copied the `backup` directory containing the etcd snapshot and the resources for the static Pods to the `/home/core/` directory of your recovery master host.

. Stop the static Pods on all other master nodes.
+
[NOTE]
====
It is not required to manually stop the Pods on the recovery host. The recovery script will stop the Pods on the recovery host.
====

.. Access a master host that is not the recovery host.

.. Move the existing etcd Pod file out of the Kubelet manifest directory:
+
----
[core@ip-10-0-154-194 ~]$ sudo mv /etc/kubernetes/manifests/etcd-pod.yaml /tmp
----

.. Verify that the etcd Pods are stopped.
+
----
[core@ip-10-0-154-194 ~]$ sudo crictl ps | grep etcd
----
+
The output of this command should be empty.

.. Move the existing Kubernetes API server Pod file out of the Kubelet manifest directory:
+
----
[core@ip-10-0-154-194 ~]$ sudo mv /etc/kubernetes/manifests/kube-apiserver-pod.yaml /tmp
----

.. Move the etcd data directory to a different location:
+
----
[core@ip-10-0-154-194 ~]$ sudo mv /var/lib/etcd/ /tmp
----

.. Repeat this step on each of the other master hosts that is not the recovery host.

. Access the recovery master host.

. Run the restore script on the recovery master host and pass in the path to the etcd backup directory:
+
----
[core@ip-10-0-143-125 ~]$ sudo /usr/local/bin/cluster-restore.sh /home/core/backup
...stopping kube-scheduler-pod.yaml
...stopping kube-controller-manager-pod.yaml
...stopping etcd-pod.yaml
...stopping kube-apiserver-pod.yaml
Waiting for container etcd to stop
.complete
Waiting for container etcdctl to stop
.............................complete
Waiting for container etcd-metrics to stop
complete
Waiting for container kube-controller-manager to stop
complete
Waiting for container kube-apiserver to stop
..........................................................................................complete
Waiting for container kube-scheduler to stop
complete
Moving etcd data-dir /var/lib/etcd/member to /var/lib/etcd-backup
starting restore-etcd static pod
starting kube-apiserver-pod.yaml
static-pod-resources/kube-apiserver-pod-7/kube-apiserver-pod.yaml
starting kube-controller-manager-pod.yaml
static-pod-resources/kube-controller-manager-pod-7/kube-controller-manager-pod.yaml
starting kube-scheduler-pod.yaml
static-pod-resources/kube-scheduler-pod-8/kube-scheduler-pod.yaml
----

. Verify that the single member control plane has started successfully.

.. From the recovery host, verify that the etcd container is running.
+
----
[core@ip-10-0-143-125 ~]$ sudo crictl ps | grep etcd
3ad41b7908e32       36f86e2eeaaffe662df0d21041eb22b8198e0e58abeeae8c743c3e6e977e8009                                                         About a minute ago   Running             etcd                                          0                   7c05f8af362f0
----

.. From the recovery host, verify that the etcd Pod is running.
+
----
[core@ip-10-0-143-125 ~]$ oc get pods -n openshift-etcd | grep etcd

NAME                                             READY   STATUS      RESTARTS   AGE
etcd-ip-10-0-143-125.ec2.internal                1/1     Running     1          2m47s
----
+
If the status is `Pending`, wait a few minutes and try again.

. Force etcd redeployment.
+
In a terminal that has access to the cluster as a `cluster-admin` user, run the following command:
+
----
$ oc patch etcd cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date --rfc-3339=ns )"'"}}' --type=merge <1>
----
<1> The `forceRedeploymentReason` value must be unique, which is why a timestamp is appended.
+
When the etcd cluster Operator performs a redeployment, the existing nodes are started with new Pods similar to the initial bootstrap scale up.

. Verify all nodes are updated to the latest revision.
+
In a terminal that has access to the cluster as a `cluster-admin` user, run the following command:
+
----
$ oc get etcd -o=jsonpath='{range .items[0].status.conditions[?(@.type=="NodeInstallerProgressing")]}{.reason}{"\n"}{.message}{"\n"}'
----
+
Review the `NodeInstallerProgressing` status condition for etcd to verify that all nodes are at the latest revision. The output shows `AllNodesAtLatestRevision` upon successful update:
+
----
AllNodesAtLatestRevision
3 nodes are at revision 3
----
+
If the output shows a message such as `2 nodes are at revision 3; 1 nodes are at revision 4`, this means that the update is still in progress. Wait a few minutes and try again.

. After etcd is redeployed, force new rollouts for the control plane. The Kubernetes API server will reinstall itself on the other nodes because the kubelet is connected to API servers using an internal load balancer.
+
In a terminal that has access to the cluster as a `cluster-admin` user, run the following commands.

.. Update the `kubeapiserver`:
+
----
$ oc patch kubeapiserver cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date --rfc-3339=ns )"'"}}' --type=merge
----
+
Verify all nodes are updated to the latest revision.
+
----
$ oc get kubeapiserver -o=jsonpath='{range .items[0].status.conditions[?(@.type=="NodeInstallerProgressing")]}{.reason}{"\n"}{.message}{"\n"}'
----
+
Review the `NodeInstallerProgressing` status condition to verify that all nodes are at the latest revision. The output shows `AllNodesAtLatestRevision` upon successful update:
+
----
AllNodesAtLatestRevision
3 nodes are at revision 3
----

.. Update the `kubecontrollermanager`:
+
----
$ oc patch kubecontrollermanager cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date --rfc-3339=ns )"'"}}' --type=merge
----
+
Verify all nodes are updated to the latest revision.
+
----
$ oc get kubecontrollermanager -o=jsonpath='{range .items[0].status.conditions[?(@.type=="NodeInstallerProgressing")]}{.reason}{"\n"}{.message}{"\n"}'
----
+
Review the `NodeInstallerProgressing` status condition to verify that all nodes are at the latest revision. The output shows `AllNodesAtLatestRevision` upon successful update:
+
----
AllNodesAtLatestRevision
3 nodes are at revision 3
----

.. Update the the `kubescheduler`:
+
----
$ oc patch kubescheduler cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date --rfc-3339=ns )"'"}}' --type=merge
----
+
Verify all nodes are updated to the latest revision.
+
----
$ oc get kubescheduler -o=jsonpath='{range .items[0].status.conditions[?(@.type=="NodeInstallerProgressing")]}{.reason}{"\n"}{.message}{"\n"}'
----
+
Review the `NodeInstallerProgressing` status condition to verify that all nodes are at the latest revision. The output shows `AllNodesAtLatestRevision` upon successful update:
+
----
AllNodesAtLatestRevision
3 nodes are at revision 3
----

. Verify that all master hosts have started and joined the cluster.
+
In a terminal that has access to the cluster as a `cluster-admin` user, run the following command:
+
----
$ oc get pods -n openshift-etcd | grep etcd
etcd-ip-10-0-143-125.ec2.internal                2/2     Running     0          9h
etcd-ip-10-0-154-194.ec2.internal                2/2     Running     0          9h
etcd-ip-10-0-173-171.ec2.internal                2/2     Running     0          9h
----

Note that it might take several minutes after completing this procedure for all services to be restored. For example, authentication by using `oc login` might not immediately work until the OAuth server Pods are restarted.
<commit_msg>Fixing kubelet capitalization<commit_after>// Module included in the following assemblies:
//
// * disaster_recovery/scenario-2-restoring-cluster-state.adoc

[id="dr-scenario-2-restoring-cluster-state_{context}"]
= Restoring to a previous cluster state

You can use a saved etcd backup to restore back to a previous cluster state. You use the etcd backup to restore a single master host. Then the etcd cluster Operator handles scaling to the remaining master hosts.

.Prerequisites

* Access to the cluster as a user with the `cluster-admin` role.
* SSH access to master hosts.
* A backup directory containing both the etcd snapshot and the resources for the static Pods, which were from the same backup. The file names in the directory must be in the following formats: `snapshot_<datetimestamp>.db` and `static_kuberesources_<datetimestamp>.tar.gz`.

.Procedure

. Select a master host to use as the recovery host. This is the host that you will run the restore operation on.

. Establish SSH connectivity to each of the master nodes, including the recovery host.
+
The Kubernetes API server will become inaccessible once the restore process has started, so you cannot access the master nodes. For this reason, it is recommended to establish SSH connectivity to each master host in a separate terminal.
+
[IMPORTANT]
====
If you do not complete this step, you will not be able to access the recovery host to complete the restore procedure, and you will be unable to recover your cluster from this state.
====

. Copy the etcd backup directory to the recovery master host.
+
This procedure assumes that you copied the `backup` directory containing the etcd snapshot and the resources for the static Pods to the `/home/core/` directory of your recovery master host.

. Stop the static Pods on all other master nodes.
+
[NOTE]
====
It is not required to manually stop the Pods on the recovery host. The recovery script will stop the Pods on the recovery host.
====

.. Access a master host that is not the recovery host.

.. Move the existing etcd Pod file out of the kubelet manifest directory:
+
----
[core@ip-10-0-154-194 ~]$ sudo mv /etc/kubernetes/manifests/etcd-pod.yaml /tmp
----

.. Verify that the etcd Pods are stopped.
+
----
[core@ip-10-0-154-194 ~]$ sudo crictl ps | grep etcd
----
+
The output of this command should be empty.

.. Move the existing Kubernetes API server Pod file out of the kubelet manifest directory:
+
----
[core@ip-10-0-154-194 ~]$ sudo mv /etc/kubernetes/manifests/kube-apiserver-pod.yaml /tmp
----

.. Move the etcd data directory to a different location:
+
----
[core@ip-10-0-154-194 ~]$ sudo mv /var/lib/etcd/ /tmp
----

.. Repeat this step on each of the other master hosts that is not the recovery host.

. Access the recovery master host.

. Run the restore script on the recovery master host and pass in the path to the etcd backup directory:
+
----
[core@ip-10-0-143-125 ~]$ sudo /usr/local/bin/cluster-restore.sh /home/core/backup
...stopping kube-scheduler-pod.yaml
...stopping kube-controller-manager-pod.yaml
...stopping etcd-pod.yaml
...stopping kube-apiserver-pod.yaml
Waiting for container etcd to stop
.complete
Waiting for container etcdctl to stop
.............................complete
Waiting for container etcd-metrics to stop
complete
Waiting for container kube-controller-manager to stop
complete
Waiting for container kube-apiserver to stop
..........................................................................................complete
Waiting for container kube-scheduler to stop
complete
Moving etcd data-dir /var/lib/etcd/member to /var/lib/etcd-backup
starting restore-etcd static pod
starting kube-apiserver-pod.yaml
static-pod-resources/kube-apiserver-pod-7/kube-apiserver-pod.yaml
starting kube-controller-manager-pod.yaml
static-pod-resources/kube-controller-manager-pod-7/kube-controller-manager-pod.yaml
starting kube-scheduler-pod.yaml
static-pod-resources/kube-scheduler-pod-8/kube-scheduler-pod.yaml
----

. Verify that the single member control plane has started successfully.

.. From the recovery host, verify that the etcd container is running.
+
----
[core@ip-10-0-143-125 ~]$ sudo crictl ps | grep etcd
3ad41b7908e32       36f86e2eeaaffe662df0d21041eb22b8198e0e58abeeae8c743c3e6e977e8009                                                         About a minute ago   Running             etcd                                          0                   7c05f8af362f0
----

.. From the recovery host, verify that the etcd Pod is running.
+
----
[core@ip-10-0-143-125 ~]$ oc get pods -n openshift-etcd | grep etcd

NAME                                             READY   STATUS      RESTARTS   AGE
etcd-ip-10-0-143-125.ec2.internal                1/1     Running     1          2m47s
----
+
If the status is `Pending`, wait a few minutes and try again.

. Force etcd redeployment.
+
In a terminal that has access to the cluster as a `cluster-admin` user, run the following command:
+
----
$ oc patch etcd cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date --rfc-3339=ns )"'"}}' --type=merge <1>
----
<1> The `forceRedeploymentReason` value must be unique, which is why a timestamp is appended.
+
When the etcd cluster Operator performs a redeployment, the existing nodes are started with new Pods similar to the initial bootstrap scale up.

. Verify all nodes are updated to the latest revision.
+
In a terminal that has access to the cluster as a `cluster-admin` user, run the following command:
+
----
$ oc get etcd -o=jsonpath='{range .items[0].status.conditions[?(@.type=="NodeInstallerProgressing")]}{.reason}{"\n"}{.message}{"\n"}'
----
+
Review the `NodeInstallerProgressing` status condition for etcd to verify that all nodes are at the latest revision. The output shows `AllNodesAtLatestRevision` upon successful update:
+
----
AllNodesAtLatestRevision
3 nodes are at revision 3
----
+
If the output shows a message such as `2 nodes are at revision 3; 1 nodes are at revision 4`, this means that the update is still in progress. Wait a few minutes and try again.

. After etcd is redeployed, force new rollouts for the control plane. The Kubernetes API server will reinstall itself on the other nodes because the kubelet is connected to API servers using an internal load balancer.
+
In a terminal that has access to the cluster as a `cluster-admin` user, run the following commands.

.. Update the `kubeapiserver`:
+
----
$ oc patch kubeapiserver cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date --rfc-3339=ns )"'"}}' --type=merge
----
+
Verify all nodes are updated to the latest revision.
+
----
$ oc get kubeapiserver -o=jsonpath='{range .items[0].status.conditions[?(@.type=="NodeInstallerProgressing")]}{.reason}{"\n"}{.message}{"\n"}'
----
+
Review the `NodeInstallerProgressing` status condition to verify that all nodes are at the latest revision. The output shows `AllNodesAtLatestRevision` upon successful update:
+
----
AllNodesAtLatestRevision
3 nodes are at revision 3
----

.. Update the `kubecontrollermanager`:
+
----
$ oc patch kubecontrollermanager cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date --rfc-3339=ns )"'"}}' --type=merge
----
+
Verify all nodes are updated to the latest revision.
+
----
$ oc get kubecontrollermanager -o=jsonpath='{range .items[0].status.conditions[?(@.type=="NodeInstallerProgressing")]}{.reason}{"\n"}{.message}{"\n"}'
----
+
Review the `NodeInstallerProgressing` status condition to verify that all nodes are at the latest revision. The output shows `AllNodesAtLatestRevision` upon successful update:
+
----
AllNodesAtLatestRevision
3 nodes are at revision 3
----

.. Update the the `kubescheduler`:
+
----
$ oc patch kubescheduler cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date --rfc-3339=ns )"'"}}' --type=merge
----
+
Verify all nodes are updated to the latest revision.
+
----
$ oc get kubescheduler -o=jsonpath='{range .items[0].status.conditions[?(@.type=="NodeInstallerProgressing")]}{.reason}{"\n"}{.message}{"\n"}'
----
+
Review the `NodeInstallerProgressing` status condition to verify that all nodes are at the latest revision. The output shows `AllNodesAtLatestRevision` upon successful update:
+
----
AllNodesAtLatestRevision
3 nodes are at revision 3
----

. Verify that all master hosts have started and joined the cluster.
+
In a terminal that has access to the cluster as a `cluster-admin` user, run the following command:
+
----
$ oc get pods -n openshift-etcd | grep etcd
etcd-ip-10-0-143-125.ec2.internal                2/2     Running     0          9h
etcd-ip-10-0-154-194.ec2.internal                2/2     Running     0          9h
etcd-ip-10-0-173-171.ec2.internal                2/2     Running     0          9h
----

Note that it might take several minutes after completing this procedure for all services to be restored. For example, authentication by using `oc login` might not immediately work until the OAuth server Pods are restarted.

{'fraction_non_alphanumeric': 0.11760775141997995, 'fraction_numerical': 0.023833389018821694, 'mean_word_length': 3.1930889563390146, 'pattern_counts': {'":': 16, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3505573', 'n_tokens_mistral': 5401, 'n_tokens_neox': 5088, 'n_words': 2068}