Intermittent httpcore.RemoteProtocolError: Server disconnected without sending a response.

[jogu]

Hi

I've been trying to switch from requests to httpx for all kinds of reasons, but I'm getting a weird and kind of reproducible problem.

About 1 in 5000 or so requests I'll get:

httpcore.RemoteProtocolError: Server disconnected without sending a response.

I'm running the latest httpx on the latest python:3.10-alpine3.15 docker container.

"By kind of reproducible" I mean "generally happens upto 2 or 3 times inside a 25 minute job in our CI".

This happens with both the normal and async client.

The client is talking to apache (as a reverse proxy for a java spring app) on a Debian container, and I've tried the apaches across 3 major versions of Debian (stretch, buster, bullseye) and the problem happens on all 3.

I've had some success adding a custom transport that retries in this case, though I'd prefer to try and fix the root cause if possible as I don't think this should happen.

I can't figure out how to diagnose this further. I tried HTTPX_LOG_LEVEL=trace, but as noted in other discussions that's not really doing much currently.

The backtrace looks like this:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 218, in handle_request
    resp = self._pool.handle_request(req)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 244, in handle_request
    raise exc
  File "/usr/local/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 228, in handle_request
    response = connection.handle_request(request)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 90, in handle_request
    return self._connection.handle_request(request)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_sync/http11.py", line 102, in handle_request
    raise exc
  File "/usr/local/lib/python3.10/site-packages/httpcore/_sync/http11.py", line 81, in handle_request
    ) = self._receive_response_headers(**kwargs)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_sync/http11.py", line 143, in _receive_response_headers
    event = self._receive_event(timeout=timeout)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_sync/http11.py", line 186, in _receive_event
    raise RemoteProtocolError(msg)
httpcore.RemoteProtocolError: Server disconnected without sending a response.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/builds/openid/conformance-suite-private/../conformance-suite/scripts/run-test-plan.py", line 285, in run_test_module
    test_module_info = await conformance.create_test_from_plan_with_variant(plan_id, module, moduledict.get('variant'))
  File "/builds/openid/conformance-suite/scripts/conformance.py", line 131, in create_test_from_plan_with_variant
    response = self.requests_session.post(api_url, params=payload)
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1116, in post
    return self.request(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 802, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 889, in send
    response = self._send_handling_auth(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 917, in _send_handling_auth
    response = self._send_handling_redirects(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 954, in _send_handling_redirects
    response = self._send_single_request(request)
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 990, in _send_single_request
    response = transport.handle_request(request)
  File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 217, in handle_request
    with map_httpcore_exceptions():
  File "/usr/local/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.RemoteProtocolError: Server disconnected without sending a response.

[johtso]

I'm having the same issue with about 1 in 10 requests..

[tomchristie]

Okay, so I can tell ya (almost certainly) why this occurs, but what we do need is to narrow down a bit more how we can get it to reproduce, and see if there's anything we can do to have more resilience to it.

Here's the deal...

• Make a request to a server, get a response.
• The connection returns to the connection pool.
• If the connection hasn't expired, and hasn't been closed by the server then we can re-use the connection on the next request.
• Except that sometimes the connection has actually been dropped by the server, without the connection having been properly closed. In which case we successfully send the request over the connection, and only "see" the connection as having been dropped when we attempt to read the response.

I'd really like to get a nice simple example of how we can reliably replicate this, so we can have a proper dig into if our connection-has-expired logic is meaningfully different from urllib3 or aiohttp, and figure out if there's a more robust way we can check for connection aliveness when re-acquiring connections from the pool.

[tomchristie]

Our is-connection-alive check...

https://github.com/encode/httpcore/blob/44b8eedfe3bc3c4b15d54810c36f05e7c5fd600d/httpcore/_utils.py#L7-L36

Which we're copying from trio...

https://github.com/python-trio/trio/blob/2f81928babefcb55915a5ec9042c764121e92a0d/trio/_socket.py#L471-L478

What urllib3 use...

https://github.com/urllib3/urllib3/blob/fab33936db9281e96b6325b4fafe2fbb852fa458/src/urllib3/util/connection.py#L11-L22

I can't figure out what aiohttp use for determining connection aliveness. Possibly structured differently because they're using asyncio protocols, rather than a Stream API. (?) ... https://github.com/aio-libs/aiohttp/blob/936e682d1ab6c833b3e5f0cc3596882cb9cb2444/aiohttp/client_proto.py#L75

[tomchristie]

@johtso I'd be interested to know what you get from the following on your system...

import select

p = select.poll()
p.poll(0)

[johtso]

@tomchristie

Python 3.10.2 (main, Feb  2 2022, 05:51:25) [Clang 13.0.0 (clang-1300.0.29.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import select
>>>
>>> p = select.poll()
>>> p.poll(0)
[]
>>>

[tomchristie]

@jogu - Could you start by updating httpx and httpcore to the lastest versions? (And then show the traceback once you've done that)

For example looking at this line in the traceback...

  File "/usr/local/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 228, in handle_request
    response = connection.handle_request(request)

I can see that doesn't match up with the latest codebase...

Line 228 is here...

https://github.com/encode/httpcore/blob/44b8eedfe3bc3c4b15d54810c36f05e7c5fd600d/httpcore/_sync/connection_pool.py#L228

Whereas the connection.handle_request line is actually on line 237...

https://github.com/encode/httpcore/blob/44b8eedfe3bc3c4b15d54810c36f05e7c5fd600d/httpcore/_sync/connection_pool.py#L237

[jogu]

@tomchristie I should probably add an update to this ticket. As I was never able to get any useful diagnostics out of httpx I never discovered exactly what was going wrong, but the problem seems like it was maybe related to timeouts, either on the httpx side or the server side (we had one spring hosted API that sometimes failed to complete before spring's timeout hit). Either way once I increased both sets of timeout (i.e. self.httpclient = httpx.Client(verify=verify_ssl, transport=transport, timeout=20) and increasing the spring timeout so it didn't give up either) things started behaving more sensibly. It also turned out I wasn't calling httpclient.close() when something went wrong, which may have somehow contributed (httpcore 0.14.6 made it obvious that I'd got that wrong).

I could have another go at reproducing if it's helpful, let me know if johtso's example script doesn't help.

[tomchristie]

Thanks. So based on looking into this with the reproduction shown by @johtso, I think you'd likely see exactly the same behaviour with either requests or httpx or aiohttp here.

We can't guard against this completely. For example as seen in @johtso's case, some server setups just outright fail to handle keep-alive connections reliably. However I think we could do a much better job of being resilient to these cases...

• We could default to limiting the maximum overall time that we keep connections open for. For example, don't re-use a connection if it's been alive for more than 5 minutes.
• We could default to limiting the maximum number of requests that connection will service before treating it as expired. For example, don't re-use a connection if it's sent more than 50 requests.
• We could improve our error messaging in this case. Possibly including more information about the connection would be helpful. https://github.com/encode/httpcore/blob/master/httpcore/_sync/http11.py#L189-L191
• We could also consider automatic retries for idempotent HTTP methods in this case...

See RFC 2616 Sec 8.1.4...

A client, server, or proxy MAY close the transport connection at any time. For example, a client might have started to send a new request at the same time that the server has decided to close the "idle" connection. From the server's point of view, the connection is being closed while it was idle, but from the client's point of view, a request is in progress.

This means that clients, servers, and proxies MUST be able to recover from asynchronous close events. Client software SHOULD reopen the transport connection and retransmit the aborted sequence of requests without user interaction so long as the request sequence is idempotent (see section 9.1.2). Non-idempotent methods or sequences MUST NOT be automatically retried, although user agents MAY offer a human operator the choice of retrying the request(s). Confirmation by user-agent software with semantic understanding of the application MAY substitute for user confirmation. The automatic retry SHOULD NOT be repeated if the second sequence of requests fails.

[jogu]

We could also consider automatic retries for idempotent HTTP methods in this case...

I'd say this is probably the most important one to go for.

It would be great if the application got some say as to what is/isn't idempotent too; there are quite a few cases where it's basically okay to retry a POST.

[johtso]

@tomchristie I've made a fairly minimal reproduction:

import httpx

urls = """\
https://archive.org/download/cu31924003160540/page/n0_x0_y0_w1283_h2099_s2.jpg
https://archive.org/download/cu31924003160540/page/n1_x532_y904_w220_h222_s1.jpg
https://archive.org/download/cu31924003160540/page/n2_x404_y1804_w326_h92_s1.jpg
https://archive.org/download/cu31924003160540/page/n3_x130_y270_w1024_h368_s2.jpg
https://archive.org/download/cu31924003160540/page/n7_x54_y214_w1184_h1342_s2.jpg
https://archive.org/download/cu31924003160540/page/n39_x174_y450_w902_h904_s2.jpg
https://archive.org/download/cu31924003160540/page/n50_x172_y452_w958_h968_s2.jpg
https://archive.org/download/cu31924003160540/page/n55_x210_y424_w450_h214_s1.jpg
https://archive.org/download/cu31924003160540/page/n55_x572_y628_w526_h238_s2.jpg
https://archive.org/download/cu31924003160540/page/n56_x220_y488_w872_h856_s2.jpg
https://archive.org/download/cu31924003160540/page/n58_x232_y730_w844_h344_s2.jpg
https://archive.org/download/cu31924003160540/page/n59_x168_y740_w966_h390_s2.jpg
https://archive.org/download/cu31924003160540/page/n60_x176_y868_w994_h458_s2.jpg
https://archive.org/download/cu31924003160540/page/n66_x236_y682_w802_h252_s2.jpg
https://archive.org/download/cu31924003160540/page/n80_x368_y672_w618_h528_s2.jpg
https://archive.org/download/cu31924003160540/page/n88_x152_y292_w1020_h1298_s2.jpg
https://archive.org/download/cu31924003160540/page/n92_x304_y116_w708_h1702_s2.jpg
https://archive.org/download/cu31924003160540/page/n98_x870_y412_w284_h958_s1.jpg
https://archive.org/download/cu31924003160540/page/n103_x260_y570_w776_h814_s2.jpg
https://archive.org/download/cu31924003160540/page/n119_x210_y592_w884_h536_s2.jpg
https://archive.org/download/cu31924003160540/page/n120_x562_y966_w586_h604_s2.jpg
https://archive.org/download/cu31924003160540/page/n147_x140_y342_w894_h1266_s2.jpg
https://archive.org/download/cu31924003160540/page/n152_x504_y498_w646_h586_s2.jpg
""".split("\n")

async def get_image(url, client):
    print('fetching', url)
    response = await client.get(url, follow_redirects=True)
    print(repr(response))
    

async def gather_with_concurrency(n, *tasks):
    semaphore = asyncio.Semaphore(n)

    async def sem_task(task):
        async with semaphore:
            return await task

    return await asyncio.gather(*(sem_task(task) for task in tasks))


async def main():
    client = httpx.AsyncClient(timeout=20.0)
    await gather_with_concurrency(2, *[get_image(url, client) for url in urls])


if __name__ == "__main__":
    import asyncio
    asyncio.run(main())

[tomchristie]

I think the follow_redirects=True is needed? I don't get the issue if we don't follow the redirect.

Yes correct. I believe the application server which handles requests on archive.org:443 is operating just fine, but the media servers which handle requests on domains such as ia800209.us.archive.org:443 are not consistently dealing with keep-alive connections properly.

You can resolve the issue for your case by switching off keep-alive...

with httpx.Client(headers={"Connection": "close"}) as client:
    ...

[johtso]

@tomchristie awesome! was about to ask if there was a way to switch off connection pooling while still using the Client api, will give that a go.

[jogu]

Yes correct. I believe the application server which handles requests on archive.org:443 is operating just fine, but the media servers which handle requests on domains such as ia800209.us.archive.org:443 are not consistently dealing with keep-alive connections properly.

I think they are actually very consistently dealing with keep-alive :-) They seem to be running a relatively modern nginx. The advantage of these servers is they are exposed on http too (i.e. no TLS) so it's easy to Wireshark and see exactly what's going on, which I just did.

They appear to have a pretty aggressive idle timeout - I make it to be around 800ms. I think that's what's causing the issue.

[adnaanbheda]

I've also been facing this issue but the magnitude at which this happened was much larger, around 300-400 requests died out of 50K requests. The server I'm accessing is running uvicorn, so my first solution was to increase the keep-alive config in uvicorn significantly, but that didn't alleviate the problem, I finally resorted to disable connection pooling i.e. recreating the client every request, that dipped the number significantly to about 2-3 requests out of 50K.

The thing that I've noticed is that the error has a turnaround time of around 5-10ms in my case, so maybe a retry mechanism can help solve the issue.

[dmakhervaks]

@tomchristie apologies for bringing up an old post.

I am dealing with a service which receives many requests and I got issues similar to the ones discussed in this thread.

This issue persisted when I had a constant number of httpx clients or a global client, however when I switched to the paradigm below:

with httpx.Client(headers={"Connection": "close"}) as client:
    ...

it seemed to have alleviated the issue. Is using the excerpt above bad practice (as opposed to having a single global client or a few clients)? In particular I am concerned with the additional overhead in instantiating a new client for every single request.

[jonathanslenders]

I'm facing the same issue (I think it's the same). It happens when talking to a Hypercorn ASGI server over HTTP/2. It doesn't happen always but very regularly. A few requests is all that's needed to reproduce it.
The main problem is that once the "Server disconnected" happens, the connection pool seems to become unusable and every following request will fail with the same error.

This is the traceback:

  Traceback (most recent call last):
    File ".../lib/python3.9/site-packages/httpx/_transports/default.py", line 66, in map_httpcore_exceptions
      yield
    File ".../lib/python3.9/site-packages/httpx/_transports/default.py", line 366, in handle_async_request
      resp = await self._pool.handle_async_request(req)
    File ".../lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 268, in handle_async_request
      raise exc
    File ".../lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 251, in handle_async_request
      response = await connection.handle_async_request(request)
    File ".../lib/python3.9/site-packages/httpcore/_async/connection.py", line 103, in handle_async_request
      return await self._connection.handle_async_request(request)
    File ".../lib/python3.9/site-packages/httpcore/_async/http2.py", line 185, in handle_async_request
      raise exc
    File ".../lib/python3.9/site-packages/httpcore/_async/http2.py", line 148, in handle_async_request
      status, headers = await self._receive_response(
    File ".../lib/python3.9/site-packages/httpcore/_async/http2.py", line 292, in _receive_response
      event = await self._receive_stream_event(request, stream_id)
    File ".../lib/python3.9/site-packages/httpcore/_async/http2.py", line 333, in _receive_stream_event
      await self._receive_events(request, stream_id)
    File ".../lib/python3.9/site-packages/httpcore/_async/http2.py", line 361, in _receive_events
      events = await self._read_incoming_data(request)
    File ".../lib/python3.9/site-packages/httpcore/_async/http2.py", line 452, in _read_incoming_data
      raise exc
    File ".../lib/python3.9/site-packages/httpcore/_async/http2.py", line 440, in _read_incoming_data
      raise RemoteProtocolError("Server disconnected")
  httpcore.RemoteProtocolError: Server disconnected

  The above exception was the direct cause of the following exception:

  Traceback (most recent call last):
    ...
    File ".../our-code.py", line 191, in make_stream_request
      async with client.stream(
    File "/home/jonathan/.pyenv/versions/3.9.16/lib/python3.9/contextlib.py", line 181, in __aenter__
      return await self.gen.__anext__()
    File ".../lib/python3.9/site-packages/httpx/_client.py", line 1573, in stream
      response = await self.send(
    File ".../lib/python3.9/site-packages/httpx/_client.py", line 1617, in send
      response = await self._send_handling_auth(
    File ".../lib/python3.9/site-packages/httpx/_client.py", line 1645, in _send_handling_auth
      response = await self._send_handling_redirects(
    File ".../lib/python3.9/site-packages/httpx/_client.py", line 1682, in _send_handling_redirects
      response = await self._send_single_request(request)
    File ".../lib/python3.9/site-packages/httpx/_client.py", line 1719, in _send_single_request
      response = await transport.handle_async_request(request)
    File ".../lib/python3.9/site-packages/httpx/_transports/default.py", line 366, in handle_async_request
      resp = await self._pool.handle_async_request(req)
    File "/home/jonathan/.pyenv/versions/3.9.16/lib/python3.9/contextlib.py", line 137, in __exit__
      self.gen.throw(typ, value, traceback)
    File ".../lib/python3.9/site-packages/httpx/_transports/default.py", line 83, in map_httpcore_exceptions
      raise mapped_exc(message) from exc
  httpx.RemoteProtocolError: Server disconnected

The client is created like this:

async with httpx.AsyncClient(
                  verify=ssl_context,
                  timeout=10.0,
                  trust_env=False,
                  proxies=None,
                  http2=True,
                  limits=httpx.Limits(
                      max_connections=100,
                      max_keepalive_connections=20,
                      keepalive_expiry=None,
                  ),
              ) as client: ...

edit: I think, but really not sure at all, that it happens after a timeout, possibly faster with max_keepalive_connections=1. It doesn't seem to happen if I keep firing requests in a loop.

[jonathanslenders]

I created this issue with a simple example to replicate it: #2983

[facundopadilla]

Hi guys, I had the same issue, but, I found some things related about this:

• If you use a VPN, try turn off it
• It's possible that you IP is banned/black listed, this happened to me.
• The server is off

I was use tenacity for retry and sometimes fixed the problem, but it come back likewise 🙃

[yks0000]

This issue in http2 still exits. We saw this issue after running a load of 20 request per second for 10 minutes and we are kind of reproduce it where we are seeing this issue after certain number of request are completed.

ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File ".../lib/python3.12/site-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions
    yield
  File ".../lib/python3.12/site-packages/httpx/_transports/default.py", line 127, in __iter__
    for part in self._httpcore_stream:
                ^^^^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.12/site-packages/httpcore/_sync/connection_pool.py", line 407, in __iter__
    raise exc from None
  File ".../lib/python3.12/site-packages/httpcore/_sync/connection_pool.py", line 403, in __iter__
    for part in self._stream:
                ^^^^^^^^^^^^
  File ".../lib/python3.12/site-packages/httpcore/_sync/http2.py", line 585, in __iter__
    raise exc
  File ".../lib/python3.12/site-packages/httpcore/_sync/http2.py", line 575, in __iter__
    for chunk in self._connection._receive_response_body(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.12/site-packages/httpcore/_sync/http2.py", line 316, in _receive_response_body
    event = self._receive_stream_event(request, stream_id)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.12/site-packages/httpcore/_sync/http2.py", line 336, in _receive_stream_event
    self._receive_events(request, stream_id)
  File ".../lib/python3.12/site-packages/httpcore/_sync/http2.py", line 355, in _receive_events
    raise RemoteProtocolError(self._connection_terminated)
httpcore.RemoteProtocolError: <ConnectionTerminated error_code:0, last_stream_id:19999, additional_data:None>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File ".../lib/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File ".../lib/python3.12/site-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File ".../lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File ".../lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File ".../lib/python3.12/site-packages/starlette/middleware/base.py", line 176, in __call__
    with recv_stream, send_stream, collapse_excgroups():
                                   ^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.12.6/Frameworks/Python.framework/Versions/3.12/lib/python3.12/contextlib.py", line 158, in __exit__
    self.gen.throw(value)
  File ".../lib/python3.12/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
    raise exc
  File ".../lib/python3.12/site-packages/starlette/middleware/base.py", line 178, in __call__
    response = await self.dispatch_func(request, call_next)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../main.py", line 64, in profile_request
    response = await call_next(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.12/site-packages/starlette/middleware/base.py", line 156, in call_next
    raise app_exc
  File ".../lib/python3.12/site-packages/starlette/middleware/base.py", line 141, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
  File ".../lib/python3.12/site-packages/opentelemetry/instrumentation/asgi/__init__.py", line 743, in __call__
    await self.app(scope, otel_receive, otel_send)
  File ".../lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File ".../lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File ".../lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File ".../lib/python3.12/site-packages/starlette/routing.py", line 714, in __call__
    await self.middleware_stack(scope, receive, send)
  File ".../lib/python3.12/site-packages/starlette/routing.py", line 734, in app
    await route.handle(scope, receive, send)
  File ".../lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File ".../lib/python3.12/site-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File ".../lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File ".../lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File ".../lib/python3.12/site-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File ".../lib/python3.12/site-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.12/site-packages/fastapi/routing.py", line 214, in run_endpoint_function
    return await run_in_threadpool(dependant.call, **values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.12/site-packages/starlette/concurrency.py", line 37, in run_in_threadpool
    return await anyio.to_thread.run_sync(func)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.12/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2470, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File ".../lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 967, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../routes/authorize.py", line 33, in authorize
    response: AuthzResponse = client.authorize(request=authz_request)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../client/client.py", line 194, in authorize
    response = self.http_sync_client.post(url=self.authorize_url, payload=payload, headers=headers)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../transport/sync_http_session_manager.py", line 172, in post
    response = self._session.post(url, content=b_payload, headers=headers, timeout=timeout)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.12/site-packages/httpx/_client.py", line 1144, in post
    return self.request(
           ^^^^^^^^^^^^^
  File ".../lib/python3.12/site-packages/httpx/_client.py", line 825, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.12/site-packages/httpx/_client.py", line 928, in send
    raise exc
  File ".../lib/python3.12/site-packages/httpx/_client.py", line 922, in send
    response.read()
  File ".../lib/python3.12/site-packages/httpx/_models.py", line 881, in read
    self._content = b"".join(self.iter_bytes())
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.12/site-packages/httpx/_models.py", line 897, in iter_bytes
    for raw_bytes in self.iter_raw():
                     ^^^^^^^^^^^^^^^
  File ".../lib/python3.12/site-packages/httpx/_models.py", line 951, in iter_raw
    for raw_stream_bytes in self.stream:
                            ^^^^^^^^^^^
  File ".../lib/python3.12/site-packages/httpx/_client.py", line 153, in __iter__
    for chunk in self._stream:
                 ^^^^^^^^^^^^
  File ".../lib/python3.12/site-packages/httpx/_transports/default.py", line 126, in __iter__
    with map_httpcore_exceptions():
         ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.12.6/Frameworks/Python.framework/Versions/3.12/lib/python3.12/contextlib.py", line 158, in __exit__
    self.gen.throw(value)
  File ".../lib/python3.12/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.RemoteProtocolError: <ConnectionTerminated error_code:0, last_stream_id:19999, additional_data:None>

[gh-andre]

I'm experiencing the same issue. Worth noting that keep-alive is not mandatory in TCP implementations and used to be frowned upon because it wastes bandwidth. When implemented, the keep-alive interval would be a couple of hours (although I did see 15 minutes being used as well), so it's not likely something that occurs within seconds or even a couple of minutes, as I seem to be observing in my case for these errors.

Also, servers may disconnect in a couple of ways. For a graceful disconnect, they will send a FIN packet, so the client will find the socket readable, but with zero data, which means the connection was gracefully closed. Some less-friendly implementations may use RST packets,, which typically come through as errors. I don't have experience with Python's HTTP libraries, but at the socket level this is handled by detecting socket readability via file descriptor sets and a subsequent zero-length read, so if Python exposes this behavior in some way, it could be used to detect closed connections before sending more data. RST packets work similarly, except that they maybe handled via readability and recv returning an error or via errors in file descriptors.

Lastly, worth noting that HTTP/3 works over QUIC/UDP, so none of the above applies.