Describe changes

This is a bundle of bug fixes and improvements that were collected during the load tests I have been performing on the ZenML server.

1. adding a new authorized API endpoint, GET /ap1/v1/load-info, that can be called to fetch some rudimentary information about the current state of the server: how many threads are currently running and how many database connections are established, used or in overflow, in the following format:

{
  "threads": 6,
  "db_connections_total": 13,
  "db_connections_active": 7,
  "db_connections_overflow": 5
}

2. some Kubernetes orchestrator improvements:

• using a role binding instead of a cluster role binding when creating the service account used to run pipelines, because a role binding is less intrusive (only grants permissions in one namespace), whereas the cluster role grants permissions across the entire cluster
• setting some default CPU and memory resource requests for the orchestrator and step pods. It is recommended to have memory resource requests defined for pods, otherwise the Kubernetes scheduler can place them all on the same node where they can blow up and affect other pods or even take down the entire node itself (which is what I observed during my tests). 400 MiB is the minimum memory use for these pods.

3. catching urllib3.HTTPError when verifying the Kubernetes service connector credentials helps report better errors to the client in case of connection errors (e.g. missing or invalid certificates)

4. set _warn_on_response_updates to False on the ServiceConnectorResponse removes some pesky and very verbose warnings that show up in the server logs because the type field is intentionally overwritten in transit.

5. makes some Helm chart fixes and improvements

• setting the zenml.database.poolSize and zenml.database.maxOverflow values is currently breaking the deployment, because these values are integer and not processed properly in the Helm chart secret definition
• with this opportunity, I created a special template section for all ZENML_STORE_* environment variables instead of setting them individually all over the place, just like we do with server settings and secret store settings.

6. updated all our middlewares to be better behaved and catch all exceptions that the next middleware called with call_next might throw and return a response, instead of letting the exception slip. This is a recommendation that came up during load testing, as some of these errors were polluting the logs without revealing the real source of the problem.

7. this PR contains some fixes and improvements for the database backup/restore functionality

• when extracting table information from the database during the backup, ordering rows by anything other than their index is extremely slow with larger tables. E.g. fetching 50 pipeline metadata rows sorted by created date takes 50 seconds when that table has around 80k rows. For this reason, we always sort by the index columns (with the exception below)
• similarly, for self-referencing tables (step runs), during backup we had to sort the rows using their created timestamp, to avoid breaking the foreign key constraint later on when the database dump is restored. This is now replaced with a different strategy: we detect which tables have self-references and then we simply disable the constraint checks during the database restore operation only for these tables.
• increased batch size from 50 rows to 100 rows to speed up the backup

8. boost server availability during Kubernetes rolling updates: done simply by making the server pod wait a bit before receiving the SIGTERM signal. This completely elliminates HTTP 503 errors received by clients during rolling updates !!!

9. make the ZenML client more robust by retrying on various status codes and read errors.

Pre-requisites

Please ensure you have done the following:

• I have read the CONTRIBUTING.md document.
• If my change requires a change to docs, I have updated the documentation accordingly.
• I have added tests to cover my changes.
• I have based my new branch on develop and the open PR is targeting develop. If your branch wasn't based on develop read Contribution guide on rebasing branch to develop.
• If my changes require changes to the dashboard, these changes are communicated/requested.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Other (add details above)