### What would you like to be added?

https://github.com/kubernetes/kubernetes/pull/124323 reminded us that test coverage of checkpointing and the interaction between kubelet and DRA drivers could be improved. We can do error injection and/or add delays by extending https://github.com/kubernetes/kubernetes/blob/4946c1fde24174ca71a9c122330f2fdf335593ea/test/e2e/dra/deploy.go#L420-L431 such that a test can add callbacks. This needs to be connected to the `NewDriver` call in E2E tests, perhaps by extending the `configureResources` call such that it both configures resources and the driver.

Here are some test scenarios that may be relevant. In all cases the test must ensure that NodePreparedResources succeeds before running a pod and NodeUnprepareResources succeeds for all claims which were previously prepared (not called out explicitly below). All pods must stop running.

## Retry NodePrepareResources

- single claim, single driver
- NodePrepareResources fails until allowed to succeed
- wait that NodePrepareResources failed at least once
- ensure that pod does not get started for a certain duration
- allow NodePrepareResources to succeed
- wait for pod to run
- kill pod

## Retry NodeUnprepareResources

- single claim, single driver
- run pod
- kill pod
- NodeUnprepareResources fails until allowed to succeed
- wait that NodeUnprepareResources failed at least once
- ensure that pod is not marked as completed for a certain duration
- allow NodeUnprepareResources to succeed

## Retry NodePrepareResources after restart

As above, but now also restart kubelet before allowing NodePrepareResources to succeed.

## Retry NodeUnprepareResources

Same for NodeUnprepareResources.

## Partial success for NodePrepareResources

- single claim, two drivers for that claim
- NodePrepareResources for driver A succeeds, the one for driver B fails.
- Rest of the test as for "NodePrepareResources".

## Partial success for NodeUnprepareResources

Same for NodeUnprepareResources.

## Partial success for NodePrepareResources with restart

Restart kubelet while the NodePrepareResources call for driver B is running.

## Partial success for NodeUnprepareResources

Restart kubelet while the NodeUnprepareResources call for driver B is running.

## Pod deletion during NodePrepareResources

- single claim, single driver
- stop kubelet while NodePrepareResources is running
- (force-)delete pod
- ensure that pod is truly gone
- restart kubelet
- ensure that NodePrepareResources does not get called again for a certain period of time

## Pod deletion during NodeUnprepareResources

- single claim, single driver
- stop kubelet while NodeUnprepareResources is running
- (force-)delete pod
- ensure that pod is truly gone
- restart kubelet
- ensure that NodeUnprepareResources gets called

/cc @klueska 
/assign @bart0sh 
/sig node








### Why is this needed?

Better test coverage.
DRA: add e2e_node tests
This is a GitHub Issue
repo:kubernetes
owner:kubernetes
Title : DRA: add e2e_node tests
Issue date:

--- start body ---
### What would you like to be added?

https://github.com/kubernetes/kubernetes/pull/124323 reminded us that test coverage of checkpointing and the interaction between kubelet and DRA drivers could be improved. We can do error injection and/or add delays by extending https://github.com/kubernetes/kubernetes/blob/4946c1fde24174ca71a9c122330f2fdf335593ea/test/e2e/dra/deploy.go#L420-L431 such that a test can add callbacks. This needs to be connected to the `NewDriver` call in E2E tests, perhaps by extending the `configureResources` call such that it both configures resources and the driver.

Here are some test scenarios that may be relevant. In all cases the test must ensure that NodePreparedResources succeeds before running a pod and NodeUnprepareResources succeeds for all claims which were previously prepared (not called out explicitly below). All pods must stop running.

## Retry NodePrepareResources

- single claim, single driver
- NodePrepareResources fails until allowed to succeed
- wait that NodePrepareResources failed at least once
- ensure that pod does not get started for a certain duration
- allow NodePrepareResources to succeed
- wait for pod to run
- kill pod

## Retry NodeUnprepareResources

- single claim, single driver
- run pod
- kill pod
- NodeUnprepareResources fails until allowed to succeed
- wait that NodeUnprepareResources failed at least once
- ensure that pod is not marked as completed for a certain duration
- allow NodeUnprepareResources to succeed

## Retry NodePrepareResources after restart

As above, but now also restart kubelet before allowing NodePrepareResources to succeed.

## Retry NodeUnprepareResources

Same for NodeUnprepareResources.

## Partial success for NodePrepareResources

- single claim, two drivers for that claim
- NodePrepareResources for driver A succeeds, the one for driver B fails.
- Rest of the test as for "NodePrepareResources".

## Partial success for NodeUnprepareResources

Same for NodeUnprepareResources.

## Partial success for NodePrepareResources with restart

Restart kubelet while the NodePrepareResources call for driver B is running.

## Partial success for NodeUnprepareResources

Restart kubelet while the NodeUnprepareResources call for driver B is running.

## Pod deletion during NodePrepareResources

- single claim, single driver
- stop kubelet while NodePrepareResources is running
- (force-)delete pod
- ensure that pod is truly gone
- restart kubelet
- ensure that NodePrepareResources does not get called again for a certain period of time

## Pod deletion during NodeUnprepareResources

- single claim, single driver
- stop kubelet while NodeUnprepareResources is running
- (force-)delete pod
- ensure that pod is truly gone
- restart kubelet
- ensure that NodeUnprepareResources gets called

/cc @klueska 
/assign @bart0sh 
/sig node








### Why is this needed?

Better test coverage.
--- end body ---

None