Client sometimes fails with httpx.LocalProtocolError
Β #7442
Closed
Description
First check
- I added a descriptive title to this issue.
- I used the GitHub search to find a similar issue and didn't find it.
- I searched the Prefect documentation for this issue.
- I checked that this issue is related to Prefect and not one of its dependencies.
Bug summary
Occasionally it seems agent is not able to retrieve the jobs to run from Prefect Cloud. It may be hours until the issue appears.
The problem had been discovered, when we received a "Late" scheduled flow notification and consequently by checking the agent logs.
The error appeared the same day we upgraded from 2.4.0 to 2.6.5 via pip.
Prefect slack thread for reference
Reproduction
1. Prefect agent 2.6.5 (or 2.6.6) running on AWS EC2 with 22.04.1 LTS (Jammy Jellyfish) x86_64
2. `pip install prefect==2.6.6`
3. `prefect agent start -q 'my_queue'`
4. tail the logs until one sees the failed scheduled flows
Error
22:24:31.719 | ERROR | prefect.agent - Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
Traceback (most recent call last):
File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/h2/connection.py", line 224, in process_input
func, target_state = self._transitions[(self.state, input_)]
KeyError: (<ConnectionState.CLOSED: 3>, <ConnectionInputs.SEND_HEADERS: 0>)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpcore/_async/http2.py", line 108, in handle_async_request
await self._send_request_headers(request=request, stream_id=stream_id)
File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpcore/_async/http2.py", line 205, in _send_request_headers
self._h2_state.send_headers(stream_id, headers, end_stream=end_stream)
File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/h2/connection.py", line 766, in send_headers
self.state_machine.process_input(ConnectionInputs.SEND_HEADERS)
File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/h2/connection.py", line 228, in process_input
raise ProtocolError(
h2.exceptions.ProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
yield
File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
resp = await self._pool.handle_async_request(req)
File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
raise exc
File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
response = await connection.handle_async_request(request)
File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpcore/_async/connection.py", line 90, in handle_async_request
return await self._connection.handle_async_request(request)
File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpcore/_async/http2.py", line 144, in handle_async_request
raise LocalProtocolError(exc) # pragma: nocover
httpcore.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/prefect/agent.py", line 154, in get_and_submit_flow_runs
queue_runs = await self.client.get_runs_in_work_queue(
File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/prefect/client/orion.py", line 759, in get_runs_in_work_queue
response = await self._client.post(
File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_client.py", line 1842, in post
return await self.request(
File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_client.py", line 1527, in request
return await self.send(request, auth=auth, follow_redirects=follow_redirects)
File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/prefect/client/base.py", line 159, in send
await super().send(*args, **kwargs)
File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_client.py", line 1614, in send
response = await self._send_handling_auth(
File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_client.py", line 1642, in _send_handling_auth
response = await self._send_handling_redirects(
File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_client.py", line 1679, in _send_handling_redirects
response = await self._send_single_request(request)
File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_client.py", line 1716, in _send_single_request
response = await transport.handle_async_request(request)
File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 352, in handle_async_request
with map_httpcore_exceptions():
File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
self.gen.throw(typ, value, traceback)
File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
22:24:31.800 | ERROR | prefect.agent - Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
Traceback (most recent call last):
File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/h2/connection.py", line 224, in process_input
func, target_state = self._transitions[(self.state, input_)]
KeyError: (<ConnectionState.CLOSED: 3>, <ConnectionInputs.SEND_HEADERS: 0>)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpcore/_async/http2.py", line 108, in handle_async_request
await self._send_request_headers(request=request, stream_id=stream_id)
File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpcore/_async/http2.py", line 205, in _send_request_headers
self._h2_state.send_headers(stream_id, headers, end_stream=end_stream)
File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/h2/connection.py", line 766, in send_headers
self.state_machine.process_input(ConnectionInputs.SEND_HEADERS)
File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/h2/connection.py", line 228, in process_input
raise ProtocolError(
h2.exceptions.ProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
yield
File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
resp = await self._pool.handle_async_request(req)
File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
raise exc
File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
response = await connection.handle_async_request(request)
File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpcore/_async/connection.py", line 90, in handle_async_request
return await self._connection.handle_async_request(request)
File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpcore/_async/http2.py", line 144, in handle_async_request
raise LocalProtocolError(exc) # pragma: nocover
httpcore.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/prefect/agent.py", line 154, in get_and_submit_flow_runs
queue_runs = await self.client.get_runs_in_work_queue(
File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/prefect/client/orion.py", line 759, in get_runs_in_work_queue
response = await self._client.post(
File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_client.py", line 1842, in post
return await self.request(
File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_client.py", line 1527, in request
return await self.send(request, auth=auth, follow_redirects=follow_redirects)
File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/prefect/client/base.py", line 159, in send
await super().send(*args, **kwargs)
File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_client.py", line 1614, in send
response = await self._send_handling_auth(
File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_client.py", line 1642, in _send_handling_auth
response = await self._send_handling_redirects(
File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_client.py", line 1679, in _send_handling_redirects
response = await self._send_single_request(request)
File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_client.py", line 1716, in _send_single_request
response = await transport.handle_async_request(request)
File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 352, in handle_async_request
with map_httpcore_exceptions():
File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
self.gen.throw(typ, value, traceback)
File "/home/prefect/insights-prefect-flows/venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
Versions
Version: 2.6.6
API version: 0.8.3
Python version: 3.10.6
Git commit: 87767cda
Built: Thu, Nov 3, 2022 1:15 PM
OS/Arch: linux/x86_64
Profile: production
Server type: cloud
Additional context
output of pip install prefect==2.6.6
$ pip install prefect==2.6.6
Requirement already satisfied: prefect==2.6.6 in ./venv/lib/python3.10/site-packages (2.6.6)
Requirement already satisfied: click<8.2,>=8.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (8.1.3)
Requirement already satisfied: pendulum>=2.1.2 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (2.1.2)
Requirement already satisfied: aiofiles>=0.7.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (22.1.0)
Requirement already satisfied: pytz>=2021.1 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (2022.2.1)
Requirement already satisfied: apprise>=1.0.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (1.0.0)
Requirement already satisfied: kubernetes>=24.2.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (24.2.0)
Requirement already satisfied: pydantic>=1.10.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (1.10.2)
Requirement already satisfied: rich>=11.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (12.5.1)
Requirement already satisfied: anyio>=3.4.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (3.6.1)
Requirement already satisfied: asgi-lifespan>=1.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (1.0.1)
Requirement already satisfied: cloudpickle>=2.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (2.2.0)
Requirement already satisfied: asyncpg>=0.23 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (0.26.0)
Requirement already satisfied: aiohttp>=3.8.1 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (3.8.3)
Requirement already satisfied: sqlalchemy[asyncio]!=1.4.33,>=1.4.20 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (1.4.41)
Requirement already satisfied: cryptography>=36.0.1 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (38.0.1)
Requirement already satisfied: pathspec>=0.8.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (0.10.1)
Requirement already satisfied: toml>=0.10.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (0.10.2)
Requirement already satisfied: coolname>=1.0.4 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (1.1.0)
Requirement already satisfied: uvicorn>=0.14.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (0.18.3)
Requirement already satisfied: python-slugify>=5.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (6.1.2)
Requirement already satisfied: readchar>=3.0.6 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (4.0.3)
Requirement already satisfied: orjson>=3.7 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (3.8.0)
Requirement already satisfied: httpx[http2]>=0.23 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (0.23.0)
Requirement already satisfied: docker>=4.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (6.0.0)
Requirement already satisfied: fsspec>=2022.5.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (2022.8.2)
Requirement already satisfied: typer>=0.4.1 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (0.6.1)
Requirement already satisfied: alembic>=1.7.5 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (1.8.1)
Requirement already satisfied: aiosqlite>=0.17.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (0.17.0)
Requirement already satisfied: pyyaml>=5.4.1 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (6.0)
Requirement already satisfied: croniter>=1.0.12 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (1.3.7)
Requirement already satisfied: packaging>=21.3 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (21.3)
Requirement already satisfied: jsonpatch>=1.32 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (1.32)
Requirement already satisfied: griffe>=0.20.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (0.22.2)
Requirement already satisfied: fastapi>=0.70 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (0.85.0)
Requirement already satisfied: typing-extensions>=4.1.0 in ./venv/lib/python3.10/site-packages (from prefect==2.6.6) (4.3.0)
Requirement already satisfied: aiosignal>=1.1.2 in ./venv/lib/python3.10/site-packages (from aiohttp>=3.8.1->prefect==2.6.6) (1.2.0)
Requirement already satisfied: frozenlist>=1.1.1 in ./venv/lib/python3.10/site-packages (from aiohttp>=3.8.1->prefect==2.6.6) (1.3.1)
Requirement already satisfied: charset-normalizer<3.0,>=2.0 in ./venv/lib/python3.10/site-packages (from aiohttp>=3.8.1->prefect==2.6.6) (2.1.1)
Requirement already satisfied: multidict<7.0,>=4.5 in ./venv/lib/python3.10/site-packages (from aiohttp>=3.8.1->prefect==2.6.6) (6.0.2)
Requirement already satisfied: attrs>=17.3.0 in ./venv/lib/python3.10/site-packages (from aiohttp>=3.8.1->prefect==2.6.6) (22.1.0)
Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in ./venv/lib/python3.10/site-packages (from aiohttp>=3.8.1->prefect==2.6.6) (4.0.2)
Requirement already satisfied: yarl<2.0,>=1.0 in ./venv/lib/python3.10/site-packages (from aiohttp>=3.8.1->prefect==2.6.6) (1.8.1)
Requirement already satisfied: Mako in ./venv/lib/python3.10/site-packages (from alembic>=1.7.5->prefect==2.6.6) (1.2.3)
Requirement already satisfied: idna>=2.8 in ./venv/lib/python3.10/site-packages (from anyio>=3.4.0->prefect==2.6.6) (3.4)
Requirement already satisfied: sniffio>=1.1 in ./venv/lib/python3.10/site-packages (from anyio>=3.4.0->prefect==2.6.6) (1.3.0)
Requirement already satisfied: six in ./venv/lib/python3.10/site-packages (from apprise>=1.0.0->prefect==2.6.6) (1.16.0)
Requirement already satisfied: requests-oauthlib in ./venv/lib/python3.10/site-packages (from apprise>=1.0.0->prefect==2.6.6) (1.3.1)
Requirement already satisfied: requests in ./venv/lib/python3.10/site-packages (from apprise>=1.0.0->prefect==2.6.6) (2.28.1)
Requirement already satisfied: markdown in ./venv/lib/python3.10/site-packages (from apprise>=1.0.0->prefect==2.6.6) (3.4.1)
Requirement already satisfied: python-dateutil in ./venv/lib/python3.10/site-packages (from croniter>=1.0.12->prefect==2.6.6) (2.8.2)
Requirement already satisfied: cffi>=1.12 in ./venv/lib/python3.10/site-packages (from cryptography>=36.0.1->prefect==2.6.6) (1.15.1)
Requirement already satisfied: websocket-client>=0.32.0 in ./venv/lib/python3.10/site-packages (from docker>=4.0->prefect==2.6.6) (1.4.1)
Requirement already satisfied: urllib3>=1.26.0 in ./venv/lib/python3.10/site-packages (from docker>=4.0->prefect==2.6.6) (1.26.12)
Requirement already satisfied: starlette==0.20.4 in ./venv/lib/python3.10/site-packages (from fastapi>=0.70->prefect==2.6.6) (0.20.4)
Requirement already satisfied: certifi in ./venv/lib/python3.10/site-packages (from httpx[http2]>=0.23->prefect==2.6.6) (2022.9.24)
Requirement already satisfied: httpcore<0.16.0,>=0.15.0 in ./venv/lib/python3.10/site-packages (from httpx[http2]>=0.23->prefect==2.6.6) (0.15.0)
Requirement already satisfied: rfc3986[idna2008]<2,>=1.3 in ./venv/lib/python3.10/site-packages (from httpx[http2]>=0.23->prefect==2.6.6) (1.5.0)
Requirement already satisfied: h2<5,>=3 in ./venv/lib/python3.10/site-packages (from httpx[http2]>=0.23->prefect==2.6.6) (4.1.0)
Requirement already satisfied: jsonpointer>=1.9 in ./venv/lib/python3.10/site-packages (from jsonpatch>=1.32->prefect==2.6.6) (2.3)
Requirement already satisfied: google-auth>=1.0.1 in ./venv/lib/python3.10/site-packages (from kubernetes>=24.2.0->prefect==2.6.6) (2.11.1)
Requirement already satisfied: setuptools>=21.0.0 in ./venv/lib/python3.10/site-packages (from kubernetes>=24.2.0->prefect==2.6.6) (59.6.0)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in ./venv/lib/python3.10/site-packages (from packaging>=21.3->prefect==2.6.6) (3.0.9)
Requirement already satisfied: pytzdata>=2020.1 in ./venv/lib/python3.10/site-packages (from pendulum>=2.1.2->prefect==2.6.6) (2020.1)
Requirement already satisfied: text-unidecode>=1.3 in ./venv/lib/python3.10/site-packages (from python-slugify>=5.0->prefect==2.6.6) (1.3)
Requirement already satisfied: pygments<3.0.0,>=2.6.0 in ./venv/lib/python3.10/site-packages (from rich>=11.0->prefect==2.6.6) (2.13.0)
Requirement already satisfied: commonmark<0.10.0,>=0.9.0 in ./venv/lib/python3.10/site-packages (from rich>=11.0->prefect==2.6.6) (0.9.1)
Requirement already satisfied: greenlet!=0.4.17 in ./venv/lib/python3.10/site-packages (from sqlalchemy[asyncio]!=1.4.33,>=1.4.20->prefect==2.6.6) (1.1.3)
Requirement already satisfied: h11>=0.8 in ./venv/lib/python3.10/site-packages (from uvicorn>=0.14.0->prefect==2.6.6) (0.12.0)
Requirement already satisfied: pycparser in ./venv/lib/python3.10/site-packages (from cffi>=1.12->cryptography>=36.0.1->prefect==2.6.6) (2.21)
Requirement already satisfied: pyasn1-modules>=0.2.1 in ./venv/lib/python3.10/site-packages (from google-auth>=1.0.1->kubernetes>=24.2.0->prefect==2.6.6) (0.2.8)
Requirement already satisfied: rsa<5,>=3.1.4 in ./venv/lib/python3.10/site-packages (from google-auth>=1.0.1->kubernetes>=24.2.0->prefect==2.6.6) (4.9)
Requirement already satisfied: cachetools<6.0,>=2.0.0 in ./venv/lib/python3.10/site-packages (from google-auth>=1.0.1->kubernetes>=24.2.0->prefect==2.6.6) (5.2.0)
Requirement already satisfied: hpack<5,>=4.0 in ./venv/lib/python3.10/site-packages (from h2<5,>=3->httpx[http2]>=0.23->prefect==2.6.6) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in ./venv/lib/python3.10/site-packages (from h2<5,>=3->httpx[http2]>=0.23->prefect==2.6.6) (6.0.1)
Requirement already satisfied: MarkupSafe>=0.9.2 in ./venv/lib/python3.10/site-packages (from Mako->alembic>=1.7.5->prefect==2.6.6) (2.1.1)
Requirement already satisfied: oauthlib>=3.0.0 in ./venv/lib/python3.10/site-packages (from requests-oauthlib->apprise>=1.0.0->prefect==2.6.6) (3.2.1)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in ./venv/lib/python3.10/site-packages (from pyasn1-modules>=0.2.1->google-auth>=1.0.1->kubernetes>=24.2.0->prefect==2.6.6) (0.4.8)