<commit_before>// Module included in the following assemblies:
//
// * backup_and_restore/replacing-unhealthy-etcd-member.adoc

[id="restore-replace-stopped-etcd-member_{context}"]
= Replacing an unhealthy etcd member whose machine is not running or whose node is not ready

This procedure details the steps to replace an etcd member that is unhealthy either because the machine is not running or because the node is not ready.

.Prerequisites

* You have identified the unhealthy etcd member.
* You have verified that either the machine is not running or the node is not ready.
* You have access to the cluster as a user with the `cluster-admin` role.
* You have taken an etcd backup.
+
[IMPORTANT]
====
It is important to take an etcd backup before performing this procedure so that your cluster can be restored if you encounter any issues.
====

.Procedure

. Remove the unhealthy member.

.. Choose a Pod that is _not_ on the affected node:
+
In a terminal that has access to the cluster as a `cluster-admin` user, run the following command:
+
----
$ oc get pods -n openshift-etcd | grep etcd
etcd-ip-10-0-131-183.ec2.internal                3/3     Running     0          123m
etcd-ip-10-0-164-97.ec2.internal                 3/3     Running     0          123m
etcd-ip-10-0-154-204.ec2.internal                3/3     Running     0          124m
----

.. Connect to the running etcd container, passing in the name of a Pod that is not on the affected node:
+
In a terminal that has access to the cluster as a `cluster-admin` user, run the following command:
+
----
$ oc rsh -n openshift-etcd etcd-ip-10-0-154-204.ec2.internal
----

.. View the member list:
+
----
sh-4.2# etcdctl member list -w table

+------------------+---------+------------------------------+---------------------------+---------------------------+
|        ID        | STATUS  |             NAME             |        PEER ADDRS         |       CLIENT ADDRS        |
+------------------+---------+------------------------------+---------------------------+---------------------------+
| 6fc1e7c9db35841d | started | ip-10-0-131-183.ec2.internal | https://10.0.131.183:2380 | https://10.0.131.183:2379 |
| 757b6793e2408b6c | started |  ip-10-0-164-97.ec2.internal |  https://10.0.164.97:2380 |  https://10.0.164.97:2379 |
| ca8c2990a0aa29d1 | started | ip-10-0-154-204.ec2.internal | https://10.0.154.204:2380 | https://10.0.154.204:2379 |
+------------------+---------+------------------------------+---------------------------+---------------------------+
----

.. Remove the unhealthy etcd member by providing the ID to the `etcdctl member remove` command:
+
----
sh-4.2# etcdctl member remove 6fc1e7c9db35841d
Member 6fc1e7c9db35841d removed from cluster baa565c8919b060e
----

.. View the member list again and verify that the member was removed:
+
----
sh-4.2# etcdctl member list -w table

+------------------+---------+------------------------------+---------------------------+---------------------------+
|        ID        | STATUS  |             NAME             |        PEER ADDRS         |       CLIENT ADDRS        |
+------------------+---------+------------------------------+---------------------------+---------------------------+
| 757b6793e2408b6c | started |  ip-10-0-164-97.ec2.internal |  https://10.0.164.97:2380 |  https://10.0.164.97:2379 |
| ca8c2990a0aa29d1 | started | ip-10-0-154-204.ec2.internal | https://10.0.154.204:2380 | https://10.0.154.204:2379 |
+------------------+---------+------------------------------+---------------------------+---------------------------+
----
+
You can now exit the node shell.

. Delete and recreate the master machine. After this machine is recreated, a new revision is forced and etcd scales up automatically.
+
If you are running an installer-provision installation, or you used the Machine API to create your machines, follow these steps. Otherwise, you must create the new master using the same method that was used to originally create it.

.. Obtain the machine for the unhealthy member.
+
In a terminal that has access to the cluster as a `cluster-admin` user, run the following command:
+
----
$ oc get machines -n openshift-machine-api -o wide

NAME                                        PHASE     TYPE        REGION      ZONE         AGE     NODE                           PROVIDERID                              STATE
clustername-8qw5l-master-0                  Running   m4.xlarge   us-east-1   us-east-1a   3h37m   ip-10-0-131-183.ec2.internal   aws:///us-east-1a/i-0ec2782f8287dfb7e   stopped <1>
clustername-8qw5l-master-1                  Running   m4.xlarge   us-east-1   us-east-1b   3h37m   ip-10-0-154-204.ec2.internal   aws:///us-east-1b/i-096c349b700a19631   running
clustername-8qw5l-master-2                  Running   m4.xlarge   us-east-1   us-east-1c   3h37m   ip-10-0-164-97.ec2.internal    aws:///us-east-1c/i-02626f1dba9ed5bba   running
clustername-8qw5l-worker-us-east-1a-wbtgd   Running   m4.large    us-east-1   us-east-1a   3h28m   ip-10-0-129-226.ec2.internal   aws:///us-east-1a/i-010ef6279b4662ced   running
clustername-8qw5l-worker-us-east-1b-lrdxb   Running   m4.large    us-east-1   us-east-1b   3h28m   ip-10-0-144-248.ec2.internal   aws:///us-east-1b/i-0cb45ac45a166173b   running
clustername-8qw5l-worker-us-east-1c-pkg26   Running   m4.large    us-east-1   us-east-1c   3h28m   ip-10-0-170-181.ec2.internal   aws:///us-east-1c/i-06861c00007751b0a   running
----
<1> This is the master machine for the unhealthy node, `ip-10-0-131-183.ec2.internal`.

.. Save the machine configuration to a file on your file system:
+
----
$ oc get machine clustername-8qw5l-master-0 \ <1>
    -n openshift-machine-api \<commit_msg>Fixing typo<commit_after>// Module included in the following assemblies:
//
// * backup_and_restore/replacing-unhealthy-etcd-member.adoc

[id="restore-replace-stopped-etcd-member_{context}"]
= Replacing an unhealthy etcd member whose machine is not running or whose node is not ready

This procedure details the steps to replace an etcd member that is unhealthy either because the machine is not running or because the node is not ready.

.Prerequisites

* You have identified the unhealthy etcd member.
* You have verified that either the machine is not running or the node is not ready.
* You have access to the cluster as a user with the `cluster-admin` role.
* You have taken an etcd backup.
+
[IMPORTANT]
====
It is important to take an etcd backup before performing this procedure so that your cluster can be restored if you encounter any issues.
====

.Procedure

. Remove the unhealthy member.

.. Choose a Pod that is _not_ on the affected node:
+
In a terminal that has access to the cluster as a `cluster-admin` user, run the following command:
+
----
$ oc get pods -n openshift-etcd | grep etcd
etcd-ip-10-0-131-183.ec2.internal                3/3     Running     0          123m
etcd-ip-10-0-164-97.ec2.internal                 3/3     Running     0          123m
etcd-ip-10-0-154-204.ec2.internal                3/3     Running     0          124m
----

.. Connect to the running etcd container, passing in the name of a Pod that is not on the affected node:
+
In a terminal that has access to the cluster as a `cluster-admin` user, run the following command:
+
----
$ oc rsh -n openshift-etcd etcd-ip-10-0-154-204.ec2.internal
----

.. View the member list:
+
----
sh-4.2# etcdctl member list -w table

+------------------+---------+------------------------------+---------------------------+---------------------------+
|        ID        | STATUS  |             NAME             |        PEER ADDRS         |       CLIENT ADDRS        |
+------------------+---------+------------------------------+---------------------------+---------------------------+
| 6fc1e7c9db35841d | started | ip-10-0-131-183.ec2.internal | https://10.0.131.183:2380 | https://10.0.131.183:2379 |
| 757b6793e2408b6c | started |  ip-10-0-164-97.ec2.internal |  https://10.0.164.97:2380 |  https://10.0.164.97:2379 |
| ca8c2990a0aa29d1 | started | ip-10-0-154-204.ec2.internal | https://10.0.154.204:2380 | https://10.0.154.204:2379 |
+------------------+---------+------------------------------+---------------------------+---------------------------+
----

.. Remove the unhealthy etcd member by providing the ID to the `etcdctl member remove` command:
+
----
sh-4.2# etcdctl member remove 6fc1e7c9db35841d
Member 6fc1e7c9db35841d removed from cluster baa565c8919b060e
----

.. View the member list again and verify that the member was removed:
+
----
sh-4.2# etcdctl member list -w table

+------------------+---------+------------------------------+---------------------------+---------------------------+
|        ID        | STATUS  |             NAME             |        PEER ADDRS         |       CLIENT ADDRS        |
+------------------+---------+------------------------------+---------------------------+---------------------------+
| 757b6793e2408b6c | started |  ip-10-0-164-97.ec2.internal |  https://10.0.164.97:2380 |  https://10.0.164.97:2379 |
| ca8c2990a0aa29d1 | started | ip-10-0-154-204.ec2.internal | https://10.0.154.204:2380 | https://10.0.154.204:2379 |
+------------------+---------+------------------------------+---------------------------+---------------------------+
----
+
You can now exit the node shell.

. Delete and recreate the master machine. After this machine is recreated, a new revision is forced and etcd scales up automatically.
+
If you are running an installer-provisioned installation, or you used the Machine API to create your machines, follow these steps. Otherwise, you must create the new master using the same method that was used to originally create it.

.. Obtain the machine for the unhealthy member.
+
In a terminal that has access to the cluster as a `cluster-admin` user, run the following command:
+
----
$ oc get machines -n openshift-machine-api -o wide

NAME                                        PHASE     TYPE        REGION      ZONE         AGE     NODE                           PROVIDERID                              STATE
clustername-8qw5l-master-0                  Running   m4.xlarge   us-east-1   us-east-1a   3h37m   ip-10-0-131-183.ec2.internal   aws:///us-east-1a/i-0ec2782f8287dfb7e   stopped <1>
clustername-8qw5l-master-1                  Running   m4.xlarge   us-east-1   us-east-1b   3h37m   ip-10-0-154-204.ec2.internal   aws:///us-east-1b/i-096c349b700a19631   running
clustername-8qw5l-master-2                  Running   m4.xlarge   us-east-1   us-east-1c   3h37m   ip-10-0-164-97.ec2.internal    aws:///us-east-1c/i-02626f1dba9ed5bba   running
clustername-8qw5l-worker-us-east-1a-wbtgd   Running   m4.large    us-east-1   us-east-1a   3h28m   ip-10-0-129-226.ec2.internal   aws:///us-east-1a/i-010ef6279b4662ced   running
clustername-8qw5l-worker-us-east-1b-lrdxb   Running   m4.large    us-east-1   us-east-1b   3h28m   ip-10-0-144-248.ec2.internal   aws:///us-east-1b/i-0cb45ac45a166173b   running
clustername-8qw5l-worker-us-east-1c-pkg26   Running   m4.large    us-east-1   us-east-1c   3h28m   ip-10-0-170-181.ec2.internal   aws:///us-east-1c/i-06861c00007751b0a   running
----
<1> This is the master machine for the unhealthy node, `ip-10-0-131-183.ec2.internal`.

.. Save the machine configuration to a file on your file system:
+
----
$ oc get machine clustername-8qw5l-master-0 \ <1>
    -n openshift-machine-api \
{'fraction_non_alphanumeric': 0.21633941093969145, 'fraction_numerical': 0.09133941093969145, 'mean_word_length': 1.6288018433179723, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 20, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 20, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2005647', 'n_tokens_mistral': 4063, 'n_tokens_neox': 3523, 'n_words': 999}