What type of PR is this?

/kind bug

What this PR does / why we need it:

This is an attempt to merge changes proposed as part of #114640. The PR was reverted due to serial test failures and this PR updates device plugin and device manager e2e tests to avoid serial test failures.

Core changes were squshed into a single commit: dc1a592632 node: device-mgr: Handle recovery by checking if healthy devices exist which was already approved in #114640 (0aa6a57, a799ffb and 5b2a3db).

Other commits are related to device plugin and device manager e2e tests. The PR adds a bunch of test cases to ensure comprehensive testing but the main focus is to recreate a scenario (where on kubelet restart application pod requesting devices comes up before device plugin pod re-registers itself) resulting in pod admission error.

Rationale of the PR as captured in the previous PR:

In case of node reboot/kubelet restart, the flow of events involves obtaining the state from the checkpoint file followed by setting the healthDevices/unhealthyDevices to its zero value. This is done to allow the device plugin to re-register itself so that capacity can be updated appropriately.

During the allocation phase, we need to check if the resources requested by the pod have been registered AND healthy devices are present on the node to be allocated.

Also we need to move this check above needed==0 where needed is required - devices allocated to the container (which is obtained from the checkpoint file) because even in cases where no additional devices have to be allocated (as they were pre-allocated), we still need to make sure that the devices that were previously allocated are healthy.

For more details refer to the comment here: #109595 (comment).

Which issue(s) this PR fixes:

Fixes #109595 and (possibly) #107928 (TODO: need to check the latter)

Special notes for your reviewer:

NOTE:

1. This PR depends on node: e2e device plugin test improvements #117057 for device plugin e2e test refactoring work and builds on top of that.

2. This PR depends on sample device plugin changes proposed in: node: device-mgr: sample device plugin: Add support to control registration process #115107.

Those changes are included here (first two commits) as they are needed for e2e test implementation.

NOTE: On node reboot, in case the application pod requesting devices starts after the device plugin pod providing those devices as has been described in #109595 and #107928 we exit early as there are pre-allocated devices without checking if the devices have registered and are healthy. This PR adds both those checks.

Because of this change, in cases where application pods are started before device plugin pod (say after node reboot), because devices are not healthy, the pod would fail with UnexpectedAdmissionError error. If the pod is part of a deployment, another pod would be created but that would stay in Pending state waiting for the pod to be scheduled to the node. This pod would go Running after the node capacity is updated followed by start up of device plugin pod and its registration.

The pod which fails at admission time continues to exist on the node and needs to be removed manually:

kubectl delete pods --field-selector status.phase=Failed

Does this PR introduce a user-facing change?

During device plugin allocation, resources requested by the pod can only be allocated if the device plugin has registered itself to kubelet AND healthy devices are present on the node to be allocated. If these conditions are not sattsfied, the pod would fail with `UnexpectedAdmissionError` error.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.: