fix(pi_rpc): detect a dead pi process immediately, not at the turn deadline
When pi died mid-turn, _read_stdout saw EOF and logged it, but nothing woke _consume -- it sat on the event queue until asyncio.timeout(budget) fired at the turn deadline (180 s). A dead process at second 3 was therefore reported as "no reply" at second 180, and the fallback that launched a fresh pi hid the death entirely. To the user this read as a hard hang. _read_stdout now enqueues a "_process_exited" sentinel on EOF, and _consume raises PiRpcError on it, so the failure path (same-session fallback model, then rotation, then a fresh process) starts within milliseconds. Smoke-tested: SIGKILL the child during a command and the client reports it in 0.0 s instead of the full budget, with no orphan left behind.
This commit is contained in:
+14
-1
@@ -445,7 +445,13 @@ class PiRpcClient:
|
||||
await result
|
||||
except Exception:
|
||||
LOG.exception("on_event callback failed")
|
||||
LOG.warning("pi RPC stdout closed")
|
||||
# EOF: the child died or closed its pipe. Without a sentinel, _consume
|
||||
# would sit on _events.get() until the turn deadline -- a process that
|
||||
# exited at second 3 would be reported as "no reply" at second 180, and a
|
||||
# fallback that launches a fresh pi would hide the death entirely. Put a
|
||||
# sentinel so the waiter fails immediately instead.
|
||||
LOG.warning("pi RPC stdout closed pid=%s rc=%s", self.pid, self._process.returncode)
|
||||
await self._events.put({"type": "_process_exited"})
|
||||
|
||||
async def _read_stderr(self) -> None:
|
||||
assert self._process and self._process.stderr
|
||||
@@ -528,6 +534,13 @@ class PiRpcClient:
|
||||
event = await self._events.get()
|
||||
kind = event.get("type")
|
||||
|
||||
if kind == "_process_exited":
|
||||
rc = self._process.returncode if self._process else None
|
||||
raise PiRpcError(
|
||||
f"pi process exited (pid={self.pid}, returncode={rc}) while "
|
||||
f"awaiting {request_id}; stderr tail: {' | '.join(self._stderr_tail[-3:])}"
|
||||
)
|
||||
|
||||
if kind == "response" and event.get("id") == request_id:
|
||||
response = event
|
||||
if not wait_for_settled or not event.get("success"):
|
||||
|
||||
Reference in New Issue
Block a user