<commit_before>// Module included in the following assemblies:
//
// * virt/node_network/virt-troubleshooting-node-network.adoc
// * networking/k8s_nmstate/k8s-nmstate-troubleshooting-node-network.adoc

[id="virt-troubleshooting-incorrect-policy-config_{context}"]
= Troubleshooting an incorrect node network configuration policy configuration

You can apply changes to the node network configuration across your entire cluster by applying a node network configuration policy.
If you apply an incorrect configuration, you can use the following example to troubleshoot and correct the failed node network policy.

In this example, a Linux bridge policy is applied to an example cluster that has three master nodes and three worker nodes.
The policy fails to be applied because it references an incorrect interface.
To find the error, investigate the available NMState resources. You can then update the policy with the correct configuration.

.Procedure

. Create a policy and apply it to your cluster. The following example creates a simple bridge on the `ens01` interface:
+
[source,yaml]
----
apiVersion: nmstate.io/v1beta1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: ens01-bridge-testfail
spec:
  desiredState:
    interfaces:
      - name: br1
        description: Linux bridge with the wrong port
        type: linux-bridge
        state: up
        ipv4:
          dhcp: true
          enabled: true
        bridge:
          options:
            stp:
              enabled: false
          port:
            - name: ens01
----
+
[source,terminal]
----
$ oc apply -f ens01-bridge-testfail.yaml
----
+
.Example output
[source,terminal]
----
nodenetworkconfigurationpolicy.nmstate.io/ens01-bridge-testfail created
----

. Verify the status of the policy by running the following command:
+
[source,terminal]
----
$ oc get nncp
----
+
The output shows that the policy failed:
+
.Example output
[source,terminal]
----
NAME                    STATUS
ens01-bridge-testfail   FailedToConfigure
----
+
However, the policy status alone does not indicate if it failed on all nodes or a subset of nodes.

. List the node network configuration enactments to see if the policy was successful on any of the nodes. If the policy failed for only a subset of nodes, it suggests that the problem is with a specific node configuration. If the policy failed on all nodes, it suggests that the problem is with the policy.
+
[source,terminal]
----
$ oc get nnce
----
+
The output shows that the policy failed on all nodes:
+
.Example output
[source,terminal]
----
NAME                                   STATUS
master-1.ens01-bridge-testfail         FailedToConfigure
master-2.ens01-bridge-testfail         FailedToConfigure
master-3.ens01-bridge-testfail         FailedToConfigure
worker-1.ens01-bridge-testfail         FailedToConfigure
worker-2.ens01-bridge-testfail         FailedToConfigure
worker-3.ens01-bridge-testfail         FailedToConfigure
----

. View one of the failed enactments and look at the traceback. The following command uses the output tool `jsonpath` to filter the output:
+
[source,terminal]
----
$ oc get nnce worker-1.ens01-bridge-testfail -o jsonpath='{.status.conditions[?(@.type=="Failing")].message}'
----
+
This command returns a large traceback that has been edited for brevity:
+
.Example output
[source,terminal]
----
error reconciling NodeNetworkConfigurationPolicy at desired state apply: , failed to execute nmstatectl set --no-commit --timeout 480: 'exit status 1' ''
...
libnmstate.error.NmstateVerificationError:
desired
=======
---
name: br1
type: linux-bridge
state: up
bridge:
  options:
    group-forward-mask: 0
    mac-ageing-time: 300
    multicast-snooping: true
    stp:
      enabled: false
      forward-delay: 15
      hello-time: 2
      max-age: 20
      priority: 32768
  port:
  - name: ens01
description: Linux bridge with the wrong port
ipv4:
  address: []
  auto-dns: true
  auto-gateway: true
  auto-routes: true
  dhcp: true
  enabled: true
ipv6:
  enabled: false
mac-address: 01-23-45-67-89-AB
mtu: 1500

current
=======
---
name: br1
type: linux-bridge
state: up
bridge:
  options:
    group-forward-mask: 0
    mac-ageing-time: 300
    multicast-snooping: true
    stp:
      enabled: false
      forward-delay: 15
      hello-time: 2
      max-age: 20
      priority: 32768
  port: []
description: Linux bridge with the wrong port
ipv4:
  address: []
  auto-dns: true
  auto-gateway: true
  auto-routes: true
  dhcp: true
  enabled: true
ipv6:
  enabled: false
mac-address: 01-23-45-67-89-AB
mtu: 1500

difference
==========
--- desired
+++ current
@@ -13,8 +13,7 @@
       hello-time: 2
       max-age: 20
       priority: 32768
-  port:
-  - name: ens01
+  port: []
 description: Linux bridge with the wrong port
 ipv4:
   address: []
  line 651, in _assert_interfaces_equal\n    current_state.interfaces[ifname],\nlibnmstate.error.NmstateVerificationError:
----
+
The `NmstateVerificationError` lists the `desired` policy configuration, the `current` configuration of the policy on the node, and the `difference` highlighting the parameters that do not match. In this example, the `port` is included in the `difference`, which suggests that the problem is the port configuration in the policy.

. To ensure that the policy is configured properly, view the network configuration for one or all of the nodes by requesting the `NodeNetworkState` object. The following command returns the network configuration for the `master-1` node:
+
----
$ oc get nns master-1 -o yaml
----
+
The output shows that the interface name on the nodes is `ens1` but the failed policy incorrectly uses `ens01`:
+
.Example output
[source,yaml]
----
   - ipv4:
 ...
      name: ens1
      state: up
      type: ethernet
----

. Correct the error by editing the existing policy:
+
[source,terminal]
----
$ oc edit nncp ens01-bridge-testfail
----
+
[source,yaml]
----
...
          port:
            - name: ens1<commit_msg>Replacing the term 'master' with 'control plane' for node names as per the new guidelines for inclusive language. Also replaced 'worker' with 'compute' but this is a larger can of worms and is apparently currently blocked by decision at the platform level.<commit_after>// Module included in the following assemblies:
//
// * virt/node_network/virt-troubleshooting-node-network.adoc
// * networking/k8s_nmstate/k8s-nmstate-troubleshooting-node-network.adoc

[id="virt-troubleshooting-incorrect-policy-config_{context}"]
= Troubleshooting an incorrect node network configuration policy configuration

You can apply changes to the node network configuration across your entire cluster by applying a node network configuration policy.
If you apply an incorrect configuration, you can use the following example to troubleshoot and correct the failed node network policy.

In this example, a Linux bridge policy is applied to an example cluster that has three control plane nodes (master) and three compute (worker) nodes.
The policy fails to be applied because it references an incorrect interface.
To find the error, investigate the available NMState resources. You can then update the policy with the correct configuration.

.Procedure

. Create a policy and apply it to your cluster. The following example creates a simple bridge on the `ens01` interface:
+
[source,yaml]
----
apiVersion: nmstate.io/v1beta1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: ens01-bridge-testfail
spec:
  desiredState:
    interfaces:
      - name: br1
        description: Linux bridge with the wrong port
        type: linux-bridge
        state: up
        ipv4:
          dhcp: true
          enabled: true
        bridge:
          options:
            stp:
              enabled: false
          port:
            - name: ens01
----
+
[source,terminal]
----
$ oc apply -f ens01-bridge-testfail.yaml
----
+
.Example output
[source,terminal]
----
nodenetworkconfigurationpolicy.nmstate.io/ens01-bridge-testfail created
----

. Verify the status of the policy by running the following command:
+
[source,terminal]
----
$ oc get nncp
----
+
The output shows that the policy failed:
+
.Example output
[source,terminal]
----
NAME                    STATUS
ens01-bridge-testfail   FailedToConfigure
----
+
However, the policy status alone does not indicate if it failed on all nodes or a subset of nodes.

. List the node network configuration enactments to see if the policy was successful on any of the nodes. If the policy failed for only a subset of nodes, it suggests that the problem is with a specific node configuration. If the policy failed on all nodes, it suggests that the problem is with the policy.
+
[source,terminal]
----
$ oc get nnce
----
+
The output shows that the policy failed on all nodes:
+
.Example output
[source,terminal]
----
NAME                                   STATUS
control-plane-1.ens01-bridge-testfail        FailedToConfigure
control-plane-2.ens01-bridge-testfail        FailedToConfigure
control-plane-3.ens01-bridge-testfail        FailedToConfigure
compute-1.ens01-bridge-testfail              FailedToConfigure
compute-2.ens01-bridge-testfail              FailedToConfigure
compute-3.ens01-bridge-testfail              FailedToConfigure
----

. View one of the failed enactments and look at the traceback. The following command uses the output tool `jsonpath` to filter the output:
+
[source,terminal]
----
$ oc get nnce compute-1.ens01-bridge-testfail -o jsonpath='{.status.conditions[?(@.type=="Failing")].message}'
----
+
This command returns a large traceback that has been edited for brevity:
+
.Example output
[source,terminal]
----
error reconciling NodeNetworkConfigurationPolicy at desired state apply: , failed to execute nmstatectl set --no-commit --timeout 480: 'exit status 1' ''
...
libnmstate.error.NmstateVerificationError:
desired
=======
---
name: br1
type: linux-bridge
state: up
bridge:
  options:
    group-forward-mask: 0
    mac-ageing-time: 300
    multicast-snooping: true
    stp:
      enabled: false
      forward-delay: 15
      hello-time: 2
      max-age: 20
      priority: 32768
  port:
  - name: ens01
description: Linux bridge with the wrong port
ipv4:
  address: []
  auto-dns: true
  auto-gateway: true
  auto-routes: true
  dhcp: true
  enabled: true
ipv6:
  enabled: false
mac-address: 01-23-45-67-89-AB
mtu: 1500

current
=======
---
name: br1
type: linux-bridge
state: up
bridge:
  options:
    group-forward-mask: 0
    mac-ageing-time: 300
    multicast-snooping: true
    stp:
      enabled: false
      forward-delay: 15
      hello-time: 2
      max-age: 20
      priority: 32768
  port: []
description: Linux bridge with the wrong port
ipv4:
  address: []
  auto-dns: true
  auto-gateway: true
  auto-routes: true
  dhcp: true
  enabled: true
ipv6:
  enabled: false
mac-address: 01-23-45-67-89-AB
mtu: 1500

difference
==========
--- desired
+++ current
@@ -13,8 +13,7 @@
       hello-time: 2
       max-age: 20
       priority: 32768
-  port:
-  - name: ens01
+  port: []
 description: Linux bridge with the wrong port
 ipv4:
   address: []
  line 651, in _assert_interfaces_equal\n    current_state.interfaces[ifname],\nlibnmstate.error.NmstateVerificationError:
----
+
The `NmstateVerificationError` lists the `desired` policy configuration, the `current` configuration of the policy on the node, and the `difference` highlighting the parameters that do not match. In this example, the `port` is included in the `difference`, which suggests that the problem is the port configuration in the policy.

. To ensure that the policy is configured properly, view the network configuration for one or all of the nodes by requesting the `NodeNetworkState` object. The following command returns the network configuration for the `control-plane-1` node:
+
----
$ oc get nns control-plane-1 -o yaml
----
+
The output shows that the interface name on the nodes is `ens1` but the failed policy incorrectly uses `ens01`:
+
.Example output
[source,yaml]
----
   - ipv4:
 ...
      name: ens1
      state: up
      type: ethernet
----

. Correct the error by editing the existing policy:
+
[source,terminal]
----
$ oc edit nncp ens01-bridge-testfail
----
+
[source,yaml]
----
...
          port:
            - name: ens1
{'fraction_non_alphanumeric': 0.09385721287371344, 'fraction_numerical': 0.022381963731416436, 'mean_word_length': 3.0566600397614314, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3629805', 'n_tokens_mistral': 3759, 'n_tokens_neox': 3498, 'n_words': 1343}