### Which jobs are failing?

Hi Folks,

I'm checking the e2e test related to webhook. Notice that that webhook first failed now below. Notice
W0424 17:48:11.860818       1 dispatcher.go:187] Failed calling webhook, failing closed deny-unwanted-configmap-data.k8s.io: failed calling webhook "deny-unwanted-configmap-data.k8s.io": failed to call webhook: Post "https://e2e-test-webhook.webhook-2542.svc:8443/configmaps?timeout=10s": context deadline exceeded


Now, if I create a pod in the Kube System, I can reach this: https://e2e-test-webhook.webhook-2542.svc:8443. 
I checked with Curl, so the webhook is active. 

For example, this for patching/updating a validating webhook should work [Conformance]

First, I see this.

W0424 17:56:43.733083       1 dispatcher.go:195] rejected by webhook "validating-is-webhook-configuration-ready.k8s.io": &errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:"", Continue:"", RemainingItemCount:(*int64)(nil)}, Status:"Failure", Message:"admission webhook \"validating-is-webhook-configuration-ready.k8s.io\" denied the request: this webhook denies all requests", Reason:"", Details:(*v1.StatusDetails)(nil), Code:400}}

Then follow up

W0424 17:56:43.740017       1 dispatcher.go:195] rejected by webhook "deny-unwanted-configmap-data.k8s.io": &errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:"", Continue:"", RemainingItemCount:(*int64)(nil)}, Status:"Failure", Message:"admission webhook \"deny-unwanted-configmap-data.k8s.io\" denied the request: the configmap contains unwanted key and value", Reason:"the configmap contains unwanted key and value", Details:(*v1.StatusDetails)(nil), Code:400}}


```
I0424 17:48:00.826632       1 alloc.go:327] "allocated clusterIPs" service="webhook-2542/e2e-test-webhook" clusterIPs=map[IPv4:100.64.163.203]
W0424 17:48:01.840232       1 dispatcher.go:195] rejected by webhook "validating-is-webhook-configuration-ready.k8s.io": &errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:"", Continue:"", RemainingItemCount:(*int64)(nil)}, Status:"Failure", Message:"admission webhook \"validating-is-webhook-configuration-ready.k8s.io\" denied the request: this webhook denies all requests", Reason:"", Details:(*v1.StatusDetails)(nil), Code:400}}
W0424 17:48:01.847172       1 dispatcher.go:195] rejected by webhook "deny-unwanted-configmap-data.k8s.io": &errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:"", Continue:"", RemainingItemCount:(*int64)(nil)}, Status:"Failure", Message:"admission webhook \"deny-unwanted-configmap-data.k8s.io\" denied the request: the configmap contains unwanted key and value", Reason:"the configmap contains unwanted key and value", Details:(*v1.StatusDetails)(nil), Code:400}}
W0424 17:48:11.860818       1 dispatcher.go:187] Failed calling webhook, failing closed deny-unwanted-configmap-data.k8s.io: failed calling webhook "deny-unwanted-configmap-data.k8s.io": failed to call webhook: Post "https://e2e-test-webhook.webhook-2542.svc:8443/configmaps?timeout=10s": context deadline exceeded
I0424 17:48:11.860951       1 trace.go:219] Trace[259587671]: "Create" accept:application/vnd.kubernetes.protobuf, */*,audit-id:4f0b5848-6ce2-4b34-b656-34c8adc18a49,client:192.168.41.20,protocol:HTTP/2.0,resource:configmaps,scope:resource,url:/api/v1/namespaces/webhook-2542/configmaps,user-agent:e2e.test/v1.26.8 (linux/amd64) kubernetes/395f0a2 -- [sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] patching/updating a validating webhook should work [Conformance],verb:POST (24-Apr-2024 17:48:01.860) (total time: 10000ms):
Trace[259587671]: ["Call validating webhook" configuration:webhook-2542,webhook:deny-unwanted-configmap-data.k8s.io,resource:/v1, Resource=configmaps,subresource:,operation:CREATE,UID:77f75f83-a189-47f4-ac50-435f52c2cbbe 10000ms (17:48:01.860)]
Trace[259587671]: [10.000736612s] [10.000736612s] END
```
### Which tests are failing?

sig-api-machinery webhook-related patch / crd reject and a few other all related to web hooks

### Since when has it been failing?

First, pass e2e successfully after re-running the test. Note that I checked all stale items, and there are none.

### Testgrid link

_No response_

### Reason for failure (if possible)

_No response_

### Anything else we need to know?

Logs from e2e during a test run.

STEP: Deploying the webhook service 04/24/24 18:03:06.685
    STEP: Verifying the service has paired with the endpoint 04/24/24 18:03:06.69
    Apr 24 18:03:07.690: INFO: Waiting for amount of service:e2e-test-webhook endpoints to be 1
    [It] patching/updating a validating webhook should work [Conformance]
      test/e2e/apimachinery/webhook.go:413
    STEP: Creating a validating webhook configuration 04/24/24 18:03:07.692
    STEP: Creating a configMap that does not comply to the validation webhook rules 04/24/24 18:03:07.704
    STEP: Updating a validating webhook configuration's rules to not include the create operation 04/24/24 18:03:07.711
    STEP: Creating a configMap that does not comply to the validation webhook rules 04/24/24 18:03:07.715
    STEP: Patching a validating webhook configuration's rules to include the create operation 04/24/24 18:03:07.719
    STEP: Creating a configMap that does not comply to the validation webhook rules 04/24/24 18:03:07.723
    Apr 24 18:03:17.726: INFO: Unexpected error: Waiting for configMap in namespace webhook-4917 to be denied creation by validating webhook:
        <*errors.StatusError | 0xc000352960>: {
            ErrStatus:
                code: 500
                details:
                  causes:
                  - message: 'failed calling webhook "deny-unwanted-configmap-data.k8s.io": failed
                      to call webhook: Post "https://e2e-test-webhook.webhook-4917.svc:8443/configmaps?timeout=10s":
                      context deadline exceeded'
                message: 'Internal error occurred: failed calling webhook "deny-unwanted-configmap-data.k8s.io":
                  failed to call webhook: Post "https://e2e-test-webhook.webhook-4917.svc:8443/configmaps?timeout=10s":
                  context deadline exceeded'
                metadata: {}
                reason: InternalError
                status: Failure,
        }
    Apr 24 18:03:17.726: FAIL: Waiting for configMap in namespace webhook-4917 to be denied creation by validating webhook: Internal error occurred: failed calling webhook "deny-unwanted-configmap-data.k8s.io": failed to call webhook: Post "https://e2e-test-webhook.webhook-4917.svc:8443/configmaps?timeout=10s": context deadline exceeded

### Relevant SIG(s)

/sig-api-machinery
[sig-api-machinery] webhook related test
This is a GitHub Issue
repo:kubernetes
owner:kubernetes
Title : [sig-api-machinery] webhook related test
Issue date:

--- start body ---
### Which jobs are failing?

Hi Folks,

I'm checking the e2e test related to webhook. Notice that that webhook first failed now below. Notice
W0424 17:48:11.860818       1 dispatcher.go:187] Failed calling webhook, failing closed deny-unwanted-configmap-data.k8s.io: failed calling webhook "deny-unwanted-configmap-data.k8s.io": failed to call webhook: Post "https://e2e-test-webhook.webhook-2542.svc:8443/configmaps?timeout=10s": context deadline exceeded


Now, if I create a pod in the Kube System, I can reach this: https://e2e-test-webhook.webhook-2542.svc:8443. 
I checked with Curl, so the webhook is active. 

For example, this for patching/updating a validating webhook should work [Conformance]

First, I see this.

W0424 17:56:43.733083       1 dispatcher.go:195] rejected by webhook "validating-is-webhook-configuration-ready.k8s.io": &errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:"", Continue:"", RemainingItemCount:(*int64)(nil)}, Status:"Failure", Message:"admission webhook \"validating-is-webhook-configuration-ready.k8s.io\" denied the request: this webhook denies all requests", Reason:"", Details:(*v1.StatusDetails)(nil), Code:400}}

Then follow up

W0424 17:56:43.740017       1 dispatcher.go:195] rejected by webhook "deny-unwanted-configmap-data.k8s.io": &errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:"", Continue:"", RemainingItemCount:(*int64)(nil)}, Status:"Failure", Message:"admission webhook \"deny-unwanted-configmap-data.k8s.io\" denied the request: the configmap contains unwanted key and value", Reason:"the configmap contains unwanted key and value", Details:(*v1.StatusDetails)(nil), Code:400}}


```
I0424 17:48:00.826632       1 alloc.go:327] "allocated clusterIPs" service="webhook-2542/e2e-test-webhook" clusterIPs=map[IPv4:100.64.163.203]
W0424 17:48:01.840232       1 dispatcher.go:195] rejected by webhook "validating-is-webhook-configuration-ready.k8s.io": &errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:"", Continue:"", RemainingItemCount:(*int64)(nil)}, Status:"Failure", Message:"admission webhook \"validating-is-webhook-configuration-ready.k8s.io\" denied the request: this webhook denies all requests", Reason:"", Details:(*v1.StatusDetails)(nil), Code:400}}
W0424 17:48:01.847172       1 dispatcher.go:195] rejected by webhook "deny-unwanted-configmap-data.k8s.io": &errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:"", Continue:"", RemainingItemCount:(*int64)(nil)}, Status:"Failure", Message:"admission webhook \"deny-unwanted-configmap-data.k8s.io\" denied the request: the configmap contains unwanted key and value", Reason:"the configmap contains unwanted key and value", Details:(*v1.StatusDetails)(nil), Code:400}}
W0424 17:48:11.860818       1 dispatcher.go:187] Failed calling webhook, failing closed deny-unwanted-configmap-data.k8s.io: failed calling webhook "deny-unwanted-configmap-data.k8s.io": failed to call webhook: Post "https://e2e-test-webhook.webhook-2542.svc:8443/configmaps?timeout=10s": context deadline exceeded
I0424 17:48:11.860951       1 trace.go:219] Trace[259587671]: "Create" accept:application/vnd.kubernetes.protobuf, */*,audit-id:4f0b5848-6ce2-4b34-b656-34c8adc18a49,client:192.168.41.20,protocol:HTTP/2.0,resource:configmaps,scope:resource,url:/api/v1/namespaces/webhook-2542/configmaps,user-agent:e2e.test/v1.26.8 (linux/amd64) kubernetes/395f0a2 -- [sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] patching/updating a validating webhook should work [Conformance],verb:POST (24-Apr-2024 17:48:01.860) (total time: 10000ms):
Trace[259587671]: ["Call validating webhook" configuration:webhook-2542,webhook:deny-unwanted-configmap-data.k8s.io,resource:/v1, Resource=configmaps,subresource:,operation:CREATE,UID:77f75f83-a189-47f4-ac50-435f52c2cbbe 10000ms (17:48:01.860)]
Trace[259587671]: [10.000736612s] [10.000736612s] END
```
### Which tests are failing?

sig-api-machinery webhook-related patch / crd reject and a few other all related to web hooks

### Since when has it been failing?

First, pass e2e successfully after re-running the test. Note that I checked all stale items, and there are none.

### Testgrid link

_No response_

### Reason for failure (if possible)

_No response_

### Anything else we need to know?

Logs from e2e during a test run.

STEP: Deploying the webhook service 04/24/24 18:03:06.685
    STEP: Verifying the service has paired with the endpoint 04/24/24 18:03:06.69
    Apr 24 18:03:07.690: INFO: Waiting for amount of service:e2e-test-webhook endpoints to be 1
    [It] patching/updating a validating webhook should work [Conformance]
      test/e2e/apimachinery/webhook.go:413
    STEP: Creating a validating webhook configuration 04/24/24 18:03:07.692
    STEP: Creating a configMap that does not comply to the validation webhook rules 04/24/24 18:03:07.704
    STEP: Updating a validating webhook configuration's rules to not include the create operation 04/24/24 18:03:07.711
    STEP: Creating a configMap that does not comply to the validation webhook rules 04/24/24 18:03:07.715
    STEP: Patching a validating webhook configuration's rules to include the create operation 04/24/24 18:03:07.719
    STEP: Creating a configMap that does not comply to the validation webhook rules 04/24/24 18:03:07.723
    Apr 24 18:03:17.726: INFO: Unexpected error: Waiting for configMap in namespace webhook-4917 to be denied creation by validating webhook:
        <*errors.StatusError | 0xc000352960>: {
            ErrStatus:
                code: 500
                details:
                  causes:
                  - message: 'failed calling webhook "deny-unwanted-configmap-data.k8s.io": failed
                      to call webhook: Post "https://e2e-test-webhook.webhook-4917.svc:8443/configmaps?timeout=10s":
                      context deadline exceeded'
                message: 'Internal error occurred: failed calling webhook "deny-unwanted-configmap-data.k8s.io":
                  failed to call webhook: Post "https://e2e-test-webhook.webhook-4917.svc:8443/configmaps?timeout=10s":
                  context deadline exceeded'
                metadata: {}
                reason: InternalError
                status: Failure,
        }
    Apr 24 18:03:17.726: FAIL: Waiting for configMap in namespace webhook-4917 to be denied creation by validating webhook: Internal error occurred: failed calling webhook "deny-unwanted-configmap-data.k8s.io": failed to call webhook: Post "https://e2e-test-webhook.webhook-4917.svc:8443/configmaps?timeout=10s": context deadline exceeded

### Relevant SIG(s)

/sig-api-machinery
--- end body ---

None