[bug] ha/client: only `TcpStream::connect` has a timeout — any hung HA reply blocks chat/voice/dashboard tool calls indefinitely

[galuis116]

Summary

HaClient::http_request in crates/genie-core/src/ha/client.rs:131-185 enforces a 5-second cap on TcpStream::connect only. None of the subsequent HTTP I/O has any timeout — writer.write_all, the status-line read_line, header read_line loop, body read_exact / read_to_string / read_chunked_body, and the per-chunk reads inside read_chunked_body can all hang indefinitely if Home Assistant pauses mid-response (Python GC, integration restart, Supervisor self-update, slow custom AppDaemon script, dropped network packets after handshake). The read-to-EOF fallback at line 228 also has no size cap, so a malformed HA response can accumulate unbounded body bytes in memory. Every chat/voice/dashboard task that holds an in-flight HA call hangs forever in those scenarios. This is the most production-frequent reliability bug I've found in this repo: it fires in any household with HA whenever HA hiccups, which for active households is daily-to-weekly.

Steps to reproduce

1. Build and start genie-core against a working Home Assistant. The configuration just needs HA_TOKEN and [services.homeassistant].url:

   cargo build --release -p genie-core
   HA_TOKEN=… GENIEPOD_CONFIG=deploy/config/geniepod.dev.toml ./target/release/genie-core

2. Wait for the first chat turn or voice cycle that invokes a Home Assistant tool (home_status, home_control, home_undo, or any path that ends in HaClient::get_state / call_service / render_template). It runs against HA fine; nothing observable yet.

3. Make HA stop responding to in-flight requests. Easiest realistic ways:

   • supervisorctl stop homeassistant while a request is mid-flight,
   • or pause the HA process: sudo kill -STOP $(pgrep -f homeassistant) (the TCP listener stays open; new connects succeed, but the accept-loop never reads the request, so writes succeed but no response ever arrives),
   • or pull the HA host off the network at the switch.

4. From a separate shell, issue a chat turn that needs HA:

   curl -s -m 120 -X POST http://127.0.0.1:3000/api/chat \
     -H 'Content-Type: application/json' \
     -d '{"message":"is the kitchen light on?"}'

   genie-core's tokio task that's invoking HaClient::get_state(…) hangs forever (the 5-second connect timeout doesn't apply — the connection is established, the read is what's blocked). The curl client gives up after 120 s; genie-core itself never times out. journalctl -u genie-core shows the request line was logged but no response ever follows.

5. Confirmation via standalone reproducer (no real HA needed). A 30-line Rust program spawns a tokio::net::TcpListener, accepts one connection, reads the request, then does nothing else for ~5 minutes. Run HaClient::test_connection() against it: it hangs for the full 5 minutes without surfacing any error. (Tests in this PR's fix reproduce this exact pattern in mod tests.)

Expected behavior

• HaClient::http_request enforces a TOTAL deadline on the request (default 30 s), not just on TcpStream::connect.
• A hung HA — TCP-accepted but never replying — surfaces as anyhow::anyhow!("Home Assistant {method} {path} timed out after {N}s") within the budget, NOT as an indefinite hang.
• Response body reads have a maximum size (default 8 MiB) so a malformed HA response can never consume unbounded memory via the read-to-EOF fallback at line 228.
• Chunked-body reads enforce the same cumulative-size cap so a long stream of chunks is bounded.
• The connect_timeout, request_timeout, and max_response_bytes are named constants (today the 5 is a magic literal at line 140), set on the HaClient so tests can override them without waiting 30 s per case.
• All other behaviour — REST endpoints, JSON serialisation, status-code handling, authorisation header — is byte-identical to today.

Actual behavior

• The connect at lines 139-141 is the only awaited operation behind a timeout. After connect succeeds, every read is unbounded:

  // ha/client.rs:139-141   (connect — has 5s timeout)
  let stream =
      tokio::time::timeout(std::time::Duration::from_secs(5), TcpStream::connect(&addr))
          .await??;
  
  // ha/client.rs:166-167   (write + read — NO timeout)
  writer.write_all(request.as_bytes()).await?;
  let response = read_http_response(reader).await?;

  // ha/client.rs:188-260   (read paths — every one unbounded)
  buf_reader.read_line(&mut status_line).await?;           // unbounded
  buf_reader.read_line(&mut line).await?;                  // unbounded (in header loop)
  buf_reader.read_exact(&mut buf).await?;                  // unbounded (Content-Length branch)
  buf_reader.read_to_string(&mut body).await?;             // unbounded (no-Content-Length branch — also unbounded MEMORY)
  reader.read_line(&mut size_line).await?;                 // unbounded (in read_chunked_body)
  tokio::io::AsyncReadExt::read_exact(reader, &mut chunk).await?;  // unbounded (per chunk)

• Grep confirms crates/genie-core/src/ha/client.rs:140 is the only tokio::time::timeout in the entire crates/genie-core/src/ha/ tree.

• Callers (get_states, get_state, call_service, render_template, test_connection) have no compensating timeout of their own. The genie-core HTTP server (server.rs) doesn't enforce a per-request deadline on the inner tool dispatch either. So the chat request's tokio task hangs as long as HA doesn't reply.

Hardware

Non-Jetson (x86_64 dev / cross-build host) and Jetson alike — the bug is in pure tokio I/O scheduling, no hardware path. Triggered by anything that pauses Home Assistant after the TCP handshake but before the HTTP response flush. Jetson households with HA running on the same Jetson hit it on every HA reload (HACS, integration update, HA Supervisor self-update). Households with HA on a separate machine hit it any time the LAN drops packets after handshake.

JetPack / L4T version

No response — bug is hardware-independent.

GenieClaw version / commit

v1.0.0-alpha.9 / main @ crates/genie-core/src/ha/client.rs:131-260 (the http_request, read_http_response, and read_chunked_body functions).

Relevant logs

# genie-core logs while HA is paused (kill -STOP) and a chat request is in flight
$ journalctl -u genie-core --since '5 minutes ago' | tail -30
… genie-core[12345]: INFO  starting HTTP chat API port=3000
… genie-core[12345]: INFO  genie-core HTTP server listening addr=127.0.0.1:3000
… genie-core[12345]: INFO  POST /api/chat origin=Api status=in_progress
…
# (no further output from genie-core until kill -CONT is sent to HA, OR until the
#  client closes the TCP connection from the other side, whichever happens first.
#  The task simply never times out. No error log, no telemetry, no diagnostic.)

# Standalone reproducer: spawn a TcpListener that accepts then sleeps forever,
# point HaClient at it, observe the indefinite hang. The fix's `mod tests`
# implements exactly this pattern in `hung_server_after_connect_times_out_cleanly`.
let listener = TcpListener::bind("127.0.0.1:0").await?;
let addr = listener.local_addr()?;
tokio::spawn(async move {
    let (_conn, _) = listener.accept().await.unwrap();
    // Never write a response. Sleep forever — mimics HA in a GC pause.
    tokio::time::sleep(Duration::from_secs(600)).await;
});
let client = HaClient::new(&addr.ip().to_string(), addr.port(), "token");
// On main: this call NEVER returns. With the fix: returns Err("… timed out after 30s") at ~30s.
let _ = client.test_connection().await;

Additional context

Symptom users see:

1. "The assistant froze." Voice cycle or chat turn that touches HA never replies. User asks again — the second turn might land on a different worker and succeed (if HA is back), or might also hang. Eventually the user power-cycles the appliance or restarts the service.
2. No diagnostic for the operator. journalctl -u genie-core shows the request line was logged but no error and no completion. The operator has no way to know whether genie-core is alive, HA is alive, or both.
3. Memory growth on malformed responses. On the read-to-EOF fallback path (no Content-Length and no Transfer-Encoding: chunked), the daemon happily accumulates the entire body in memory. If HA sends partial bytes indefinitely, RSS grows without bound until OOM-killer fires. Rare but real for misbehaving HA versions or proxies.

Why this fires in production at all — every household with HA exercises this code path on essentially every tool call (home_status, home_control, home_undo, resolve_target, governor/health pollers). Triggers:

Trigger	How often in real households
HA Supervisor self-update / HACS reload / integration update	weekly+
HA Python GC pause during a slow custom AppDaemon / Node-RED automation	daily under load
HA host Wi-Fi drop, switch reboot, ethernet renegotiation	monthly+
HA process alive but slow custom integration blocks its event loop	sporadic but recurrent
HA crashes mid-response after TCP handshake	rare but realistic

Why the existing tests don't catch it:

• crates/genie-core/src/ha/client.rs has tests only for parse_http_url (lines 297-322). No tests exist for http_request, read_http_response, or read_chunked_body against any kind of failure mode — no mock TCP server, no timeout coverage.
• mod tests in this file currently has no tokio::test-flavored coverage at all.

Files to touch:

• crates/genie-core/src/ha/client.rs:
  • Add connect_timeout: Duration, request_timeout: Duration, max_response_bytes: usize fields to HaClient; default 5s, 30s, 8 MiB.
  • Add named constants for the defaults so they appear in the public surface and are not magic literals.
  • Wrap the post-connect block of http_request in tokio::time::timeout(self.request_timeout, …). Map Elapsed to an anyhow!("Home Assistant {method} {path} timed out after {N}s").
  • Extend read_http_response and read_chunked_body to take max_bytes; cap Content-Length, chunked-body, and read-to-EOF reads at that ceiling; surface a clear anyhow!("Home Assistant response exceeded N bytes") rather than OOM'ing.
  • Add pub(crate) fn with_test_timeouts(mut self, connect: Duration, request: Duration, max_bytes: usize) -> Self so tests can drive the timeouts in millisecond-scale.
• crates/genie-core/src/ha/client.rs tests — five new regressions, all hardware-free:
  • connect_timeout_to_unroutable_address — HaClient pointed at 192.0.2.1 (RFC 5737 documentation range) with a 100 ms test connect timeout; assert test_connection() fails inside the budget with a "connect" error string.
  • hung_server_after_connect_times_out_cleanly — TcpListener that accepts but never replies; assert get_states() fails with a "timed out" error inside the test request budget (e.g. 200 ms).
  • slow_server_within_budget_succeeds — TcpListener that accepts, waits ~50 ms, then writes a valid HTTP/1.1 200 response with a tiny JSON array; assert success.
  • read_to_eof_response_is_size_capped — TcpListener that writes a response with no Content-Length and no chunked encoding, then streams more bytes than max_response_bytes; assert the client bails with a "response too large" error rather than consuming memory.
  • chunked_body_with_oversize_aggregate_is_capped — TcpListener that returns a chunked body whose cumulative payload exceeds the cap; assert the client bails before allocating it all.

Acceptance (PR):

• cargo test -p genie-core --lib ha::client — existing 3 parse_http_url_* tests still pass, 5 new regression tests pass.
• Building genie-core in --release and pointing it at a kill -STOPped HA (using the steps under "Reproduce") returns an explicit Err("Home Assistant GET /api/states timed out after 30s") from the tool dispatch path within ~30 s. No daemon hang.
• Default geniepod.toml and geniepod.dev.toml see no behaviour change for healthy HA.

Timing: Standalone — no upstream PRs touch ha/client.rs (one commit ever on this file: scaffold + an unrelated rename). Lands independently of #163, #164/165, #169 (mine), or #171 (mine). No merge-conflict surface with any of those.

Real Behavior Proof: cargo test -p genie-core --lib ha::client showing the 5 new tests pass alongside the 3 existing parse_http_url_* tests. Plus the standalone-reproducer behaviour: pre-fix the hung-server test would hang forever; post-fix it returns Err("…timed out…") in ~200 ms.

Related: #124 / PR #125 fixed cancellation of the LLM stream producer when the client disconnects — adjacent in spirit ("don't hang on a remote that stops responding"), different module. #109 / PR #118 ("100 consecutive voice cycles with zero stalls and zero silent drops") chased voice stalls but never touched HA HTTP I/O. No upstream issue mentions HA HTTP read timeout. The shared "every household with HA hits this on every HA hiccup" framing puts it in the same severity class as #168 / PR #169 (mine) and #170 / PR #171 (mine), each of which the maintainers have accepted as a real bug.