WebSocketBadStatusException: Handshake status 500 Internal Server Error: container not found (re-raised as ApiException) #39
Open
Description
Migrated from internal repo.
Complete stack trace and logs (sensitive) https://github.com/AI-Safety-Institute/aisi-inspect-tools/issues/154
Original date: 27 Oct 2024
WebSocketBadStatusException: Handshake status 500 Internal Server Error -+-+- {'content-length': '31', 'content-type': 'text/plain; charset=utf-8', 'date': 'Sun, 27 Oct 2024 00:06:14 GMT'} -+-+- b'container not found ("default")'
...
K8sError: Error during: Execute command in pod. {"pod": "agent-env-a7mgqcsr-default-0", ...
With timestamps
2024-10-27 00:06:14,802 - SANDBOX - K8S: Starting: Execute command in pod. {"pod": "agent-env-a7mgqcsr-default-0", ...
2024-10-27 00:06:15,115 - ERROR - K8S: Error during: Execute command in pod. {"cause": "(0)\nReason: Handshake status 500 Internal Server Error -+-+- {'content-length': '31', 'content-type': 'text/plain; charset=utf-8', 'date': 'Sun, 27 Oct 2024 00:06:14 GMT'} -+-+- b'container not found (\"default\")'\n", ...
Cluster events (note "SandboxChanged")
2024-10-26T23:45:04Z Normal agent-env-a7mgqcsr-default-0 Scheduled Successfully assigned agent/agent-env-a7mgqcsr-default-0 to ip-192-168-151-2.eu-west-2.compute.internal
2024-10-26T23:45:04Z Normal agent-env-a7mgqcsr-default SuccessfulCreate create Pod agent-env-a7mgqcsr-default-0 in StatefulSet agent-env-a7mgqcsr-default successful
2024-10-26T23:45:05Z Normal agent-env-a7mgqcsr-default-0 Created Created container resolve-coredns-ip
2024-10-26T23:45:05Z Normal agent-env-a7mgqcsr-default-0 Pulled Container image "toolbelt/dig:2024-09-23" already present on machine
2024-10-26T23:45:05Z Normal agent-env-a7mgqcsr-default-0 Started Started container resolve-coredns-ip
2024-10-26T23:45:06Z Normal agent-env-a7mgqcsr-default-0 Pulled Container image "python:3.12-bookworm" already present on machine
2024-10-26T23:45:07Z Normal agent-env-a7mgqcsr-default-0 Started Started container default
2024-10-26T23:45:07Z Normal agent-env-a7mgqcsr-default-0 Created Created container default
2024-10-27T00:06:08Z Normal agent-env-a7mgqcsr-default-0 SandboxChanged Pod sandbox changed, it will be killed and re-created.
2024-10-27T00:06:13Z Normal agent-env-a7mgqcsr-default-0 Created Created container resolve-coredns-ip
2024-10-27T00:06:13Z Normal agent-env-a7mgqcsr-default-0 Pulled Container image "toolbelt/dig:2024-09-23" already present on machine
2024-10-27T00:06:14Z Normal agent-env-a7mgqcsr-default-0 Started Started container resolve-coredns-ip
2024-10-27T00:06:15Z Normal agent-env-a7mgqcsr-default-0 Pulled Container image "python:3.12-bookworm" already present on machine
2024-10-27T00:06:15Z Normal agent-env-a7mgqcsr-default-0 Created Created container default
2024-10-27T00:06:15Z Normal agent-env-a7mgqcsr-default-0 Started Started container default
2024-10-27T00:06:17Z Normal agent-env-a7mgqcsr-default-0 Killing Stopping container default
06 Nov 2024
│ redacted/.venv/lib/python3.12/site-packages/websocket/_handshake.py:150 in _get_resp_headers │
│ │
│ 147 │ │ │ ) # read the body of the HTTP error message response and include it in the │
│ 148 │ │ else: │
│ 149 │ │ │ response_body = None │
│ > 150 │ │ raise WebSocketBadStatusException( │
│ 151 │ │ │ f"Handshake status {status} {status_message} -+-+- {resp_headers} -+-+- {res │
│ 152 │ │ │ status, │
│ 153 │ │ │ status_message, │
└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
WebSocketBadStatusException: Handshake status 500 Internal Server Error -+-+- {'content-length': '31', 'content-type': 'text/plain; charset=utf-8', 'date': 'Tue, 05 Nov 2024 23:15:49 GMT'} -+-+- b'container not found ("default")'
...
│ redacted/.venv/lib/python3.12/site-packages/kubernetes/stream/ws_client.py:538 in │
│ websocket_call │
│ │
│ 535 │ │ else: │
│ 536 │ │ │ return WSResponse('%s' % ''.join(all)) │
│ 537 │ except (Exception, KeyboardInterrupt, SystemExit) as e: │
│ > 538 │ │ raise ApiException(status=0, reason=str(e)) │
│ 539 │
│ 540 │
│ 541 def portforward_call(configuration, _method, url, **kwargs): │
└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
ApiException: (0)
Reason: Handshake status 500 Internal Server Error -+-+- {'content-length': '31', 'content-type': 'text/plain; charset=utf-8', 'date': 'Tue, 05 Nov 2024 23:15:49 GMT'} -+-+- b'container not found ("default")'
...
K8sError: Error during: Execute command in pod. {"pod": "agent-env-akqvhzlv-default-0", ...
2024-11-05 22:28:53,161 - SANDBOX - K8S: Starting: Write file to pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:28:53,320 - SANDBOX - K8S: Completed: Write file to pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:29:49,475 - SANDBOX - K8S: Starting: Execute command in pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:29:49,693 - SANDBOX - K8S: Completed: Execute command in pod. {"result": "ExecResult(success=True, returncode ...
2024-11-05 22:29:49,693 - SANDBOX - K8S: Starting: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:29:49,802 - SANDBOX - K8S: Completed: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:30:29,761 - SANDBOX - K8S: Starting: Execute command in pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:30:54,481 - SANDBOX - K8S: Completed: Execute command in pod. {"result": "ExecResult(success=False, returncode ...
2024-11-05 22:30:54,481 - SANDBOX - K8S: Starting: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:30:54,588 - SANDBOX - K8S: Completed: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:31:23,680 - SANDBOX - K8S: Starting: Execute command in pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:31:26,082 - SANDBOX - K8S: Completed: Execute command in pod. {"result": "ExecResult(success=True, returncode ...
2024-11-05 22:31:26,082 - SANDBOX - K8S: Starting: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:31:26,204 - SANDBOX - K8S: Completed: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:32:31,732 - SANDBOX - K8S: Starting: Execute command in pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:32:31,976 - SANDBOX - K8S: Completed: Execute command in pod. {"result": "ExecResult(success=True, returncode ...
2024-11-05 22:33:22,228 - SANDBOX - K8S: Starting: Execute command in pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:33:22,633 - SANDBOX - K8S: Completed: Execute command in pod. {"result": "ExecResult(success=False, returncode ...
2024-11-05 22:34:08,098 - SANDBOX - K8S: Starting: Execute command in pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:34:08,285 - SANDBOX - K8S: Completed: Execute command in pod. {"result": "ExecResult(success=True, returncode ...
2024-11-05 22:34:52,291 - SANDBOX - K8S: Starting: Execute command in pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:34:52,587 - SANDBOX - K8S: Completed: Execute command in pod. {"result": "ExecResult(success=True, returncode ...
2024-11-05 22:34:52,587 - SANDBOX - K8S: Starting: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:34:52,731 - SANDBOX - K8S: Completed: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:35:52,309 - SANDBOX - K8S: Starting: Execute command in pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:36:06,006 - SANDBOX - K8S: Completed: Execute command in pod. {"result": "ExecResult(success=False, returncode ...
2024-11-05 22:36:06,006 - SANDBOX - K8S: Starting: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:36:06,116 - SANDBOX - K8S: Completed: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:37:14,616 - SANDBOX - K8S: Starting: Execute command in pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:37:14,772 - SANDBOX - K8S: Completed: Execute command in pod. {"result": "ExecResult(success=True, returncode ...
2024-11-05 22:37:14,773 - SANDBOX - K8S: Starting: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:37:14,900 - SANDBOX - K8S: Completed: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:39:34,692 - SANDBOX - K8S: Starting: Execute command in pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:39:34,884 - SANDBOX - K8S: Completed: Execute command in pod. {"result": "ExecResult(success=True, returncode ...
2024-11-05 22:40:39,403 - SANDBOX - K8S: Starting: Execute command in pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:40:39,624 - SANDBOX - K8S: Error during: Execute command in pod. {"cause": "'utf-8' ...
2024-11-05 22:42:07,421 - SANDBOX - K8S: Starting: Execute command in pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:42:07,722 - SANDBOX - K8S: Completed: Execute command in pod. {"result": "ExecResult(success=True, returncode ...
2024-11-05 22:42:07,722 - SANDBOX - K8S: Starting: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:42:07,859 - SANDBOX - K8S: Completed: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:43:32,065 - SANDBOX - K8S: Starting: Execute command in pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:43:32,350 - SANDBOX - K8S: Completed: Execute command in pod. {"result": "ExecResult(success=False, returncode ...
2024-11-05 22:43:32,350 - SANDBOX - K8S: Starting: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:43:32,733 - SANDBOX - K8S: Completed: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:45:06,100 - SANDBOX - K8S: Starting: Execute command in pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:45:06,709 - SANDBOX - K8S: Completed: Execute command in pod. {"result": "ExecResult(success=False, returncode ...
2024-11-05 22:45:06,709 - SANDBOX - K8S: Starting: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:45:06,834 - SANDBOX - K8S: Completed: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:46:01,742 - SANDBOX - K8S: Starting: Execute command in pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:46:02,621 - SANDBOX - K8S: Completed: Execute command in pod. {"result": "ExecResult(success=True, returncode ...
2024-11-05 22:46:02,621 - SANDBOX - K8S: Starting: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:46:02,791 - SANDBOX - K8S: Completed: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:47:30,587 - SANDBOX - K8S: Starting: Execute command in pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:47:30,767 - SANDBOX - K8S: Completed: Execute command in pod. {"result": "ExecResult(success=True, returncode ...
2024-11-05 22:48:29,348 - SANDBOX - K8S: Starting: Execute command in pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:48:29,485 - SANDBOX - K8S: Error during: Execute command in pod. {"cause": "'utf-8' ...
2024-11-05 22:50:08,606 - SANDBOX - K8S: Starting: Execute command in pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:50:08,999 - SANDBOX - K8S: Completed: Execute command in pod. {"result": "ExecResult(success=True, returncode ...
2024-11-05 22:50:08,999 - SANDBOX - K8S: Starting: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:50:09,321 - SANDBOX - K8S: Completed: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:51:44,178 - SANDBOX - K8S: Starting: Execute command in pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:51:44,363 - SANDBOX - K8S: Completed: Execute command in pod. {"result": "ExecResult(success=False, returncode ...
2024-11-05 22:51:44,363 - SANDBOX - K8S: Starting: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:51:44,480 - SANDBOX - K8S: Completed: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:53:17,418 - SANDBOX - K8S: Starting: Execute command in pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:53:17,571 - SANDBOX - K8S: Completed: Execute command in pod. {"result": "ExecResult(success=True, returncode ...
2024-11-05 22:53:17,571 - SANDBOX - K8S: Starting: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:53:17,705 - SANDBOX - K8S: Completed: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:54:56,659 - SANDBOX - K8S: Starting: Execute command in pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:54:56,986 - SANDBOX - K8S: Completed: Execute command in pod. {"result": "ExecResult(success=True, returncode ...
2024-11-05 22:56:11,922 - SANDBOX - K8S: Starting: Execute command in pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:56:13,731 - SANDBOX - K8S: Completed: Execute command in pod. {"result": "ExecResult(success=True, returncode ...
2024-11-05 22:56:13,731 - SANDBOX - K8S: Starting: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:56:13,871 - SANDBOX - K8S: Completed: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:57:57,959 - SANDBOX - K8S: Starting: Execute command in pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 23:15:49,220 - ERROR - K8S: Error during: Execute command in pod. {"cause": "(0)\nReason ...
2024-11-05T22:28:42Z Warning agent-env-akqvhzlv-default-0 FailedScheduling 0/46 nodes are available: 2 node(s) had untolerated taint {CriticalAddonsOnly: true}, 2 node(s) had untolerated taint {aisi.gov.uk/dev: true}, 2 node(s) had untolerated taint {aisi.gov.uk/devpods: true}, 2 node(s) had untolerated taint {node.kubernetes.io/unreachable: }, 38 Insufficient memory. preemption: 0/46 nodes are available: 38 No preemption victims found for incoming pod, 8 Preemption is not helpful for scheduling.
2024-11-05T22:28:45Z Normal agent-env-akqvhzlv-default-0 Scheduled Successfully assigned agent/agent-env-akqvhzlv-default-0 to ip-192-168-174-135.eu-west-2.compute.internal
2024-11-05T22:28:46Z Normal agent-env-akqvhzlv-default-0 Pulled Container image "redacted" already present on machine
2024-11-05T22:28:46Z Normal agent-env-akqvhzlv-default-0 Started Started container resolve-coredns-ip
2024-11-05T22:28:46Z Normal agent-env-akqvhzlv-default-0 Created Created container resolve-coredns-ip
2024-11-05T22:28:46Z Normal agent-env-akqvhzlv-default-0 Pulled Container image "toolbelt/dig:2024-09-23" already present on machine
2024-11-05T22:28:47Z Normal agent-env-akqvhzlv-default-0 Started Started container default
2024-11-05T22:28:47Z Normal agent-env-akqvhzlv-default-0 Created Created container default
2024-11-05T22:57:21Z Warning agent-env-akqvhzlv-default-0 NodeNotReady Node is not ready
2024-11-05T23:02:26Z Normal agent-env-akqvhzlv-default-0 TaintManagerEviction Marking for deletion Pod agent/agent-env-akqvhzlv-default-0
2024-11-05T23:02:27Z Normal agent-env-akqvhzlv-default SuccessfulCreate create Pod agent-env-akqvhzlv-default-0 in StatefulSet agent-env-akqvhzlv-default successful
2024-11-05T23:07:09Z Warning agent-env-akqvhzlv-default-0 FailedScheduling 0/46 nodes are available: 2 node(s) had untolerated taint {CriticalAddonsOnly: true}, 2 node(s) had untolerated taint {aisi.gov.uk/dev: true}, 2 node(s) had untolerated taint {aisi.gov.uk/devpods: true}, 36 Insufficient memory, 4 node(s) had untolerated taint {node.kubernetes.io/unreachable: }. preemption: 0/46 nodes are available: 10 Preemption is not helpful for scheduling, 36 No preemption victims found for incoming pod.
2024-11-05T23:07:19Z Normal agent-env-akqvhzlv-default-0 Scheduled Successfully assigned agent/agent-env-akqvhzlv-default-0 to ip-192-168-189-169.eu-west-2.compute.internal
2024-11-05T23:07:20Z Normal agent-env-akqvhzlv-default-0 Pulled Container image "toolbelt/dig:2024-09-23" already present on machine
2024-11-05T23:07:20Z Normal agent-env-akqvhzlv-default-0 Created Created container resolve-coredns-ip
2024-11-05T23:07:21Z Normal agent-env-akqvhzlv-default-0 Created Created container default
2024-11-05T23:07:21Z Normal agent-env-akqvhzlv-default-0 Pulled Container image "redacted" already present on machine
2024-11-05T23:07:21Z Normal agent-env-akqvhzlv-default-0 Started Started container resolve-coredns-ip
2024-11-05T23:07:22Z Normal agent-env-akqvhzlv-default-0 Started Started container default
2024-11-05T23:15:49Z Normal agent-env-akqvhzlv-default-0 SandboxChanged Pod sandbox changed, it will be killed and re-created.
2024-11-05T23:15:50Z Normal agent-env-akqvhzlv-default-0 Killing Stopping container default
2024-11-05T23:15:51Z Normal agent-env-akqvhzlv-default-0 Pulled Container image "toolbelt/dig:2024-09-23" already present on machine
2024-11-05T23:15:51Z Normal agent-env-akqvhzlv-default-0 Created Created container resolve-coredns-ip
2024-11-05T23:15:51Z Normal agent-env-akqvhzlv-default-0 Started Started container resolve-coredns-ip
2024-11-05T23:15:52Z Normal agent-env-akqvhzlv-default-0 Killing Stopping container resolve-coredns-ip