pi-agent-config/docs/gateway-patterns.md

Gateway Patterns

Patterns for hosting a Pi agent behind a long-running service (Telegram, HTTP, queue). Distilled from the memo-inbox scenario, which has run these in production since 2026-07, and from the defects found in the curator scenario, which reinvented the same problems worse.

Mechanism references: pi-runtime-notes.md §§13–18.

---

Pattern 1 — One long-lived RPC process, not one process per message

Do: start pi --mode rpc once, drive it turn by turn over stdin.

Don't: spawn pi --print per message.

Per-message spawning pays node startup, resource discovery, skill loading and session deserialization on every single turn. The curator scenario spawned 2–4 processes per Telegram message; with a 120 s per-invocation timeout its worst case was ~480 s of model time for one message, with no overall budget.

An RPC process additionally gives you abort, set_model, set_thinking_level, compact and get_session_stats at runtime — none of which are reachable from --print.

Framing is strict JSONL

Split on \n only, strip a trailing \r. Do not use a generic line reader: Node's readline also splits on U+2028/U+2029, which are legal inside JSON strings. The reference implementation:

buffer = b""
while chunk := await proc.stdout.read(65536):
    buffer += chunk
    while b"\n" in buffer:
        line, buffer = buffer.split(b"\n", 1)
        line = line.rstrip(b"\r")
        if not line:
            continue
        await events.put(json.loads(line))

Correlate by id, and understand what success means

Every command may carry an id; the matching response echoes it. But success: true only means accepted. Failures after acceptance never produce a second response for that id — they arrive as events. So a request is complete when you have seen both the response and the terminal event you care about (agent_settled, or message_end for a reply).

---

Pattern 2 — Rotate sessions on a counter

Pi has no session TTL, rotation or size cap, and auto-compaction only fires near the context window — with a 1.05 M-token model that is effectively never.

memo-inbox rotates after a fixed number of prompts:

PI_SESSION_ROTATE_AFTER_PROMPTS=24

The curator scenario did not. Its per-chat session reached 174 KB / 74 messages, growing ~4.7 KB per turn because every turn embedded the full backend fact payload of that turn. Latency, cost and timeout risk degrade monotonically.

Rotate on whichever bound you can measure: prompt count, session bytes, or get_session_stats().contextUsage.percent. Prefer starting a fresh --session-id suffix over relying on compact, so that history is auditable.

---

Pattern 3 — Derive write receipts from tool_execution_end, never from prose

This is the single most valuable pattern in memo-inbox. The gateway subscribes to tool completion events and extracts the receipt from the tool result:

elif (event.get("type") == "tool_execution_end"
      and event.get("toolName") in MUTATION_TOOLS):
    result = event.get("result") or {}
    receipt = "\n".join(b.get("text", "") for b in result.get("content") or [])

The model's prose is then a summary, not the record of what happened.

Compare the curator scenario, which asked the model to phrase the outcome and relied on prompt rules to keep the distinction. Observed production output when has_file was false:

《My Brilliant Career》4K 版已成功加入库中，但目前还没有 4K 文件。

"加入库中" reads as in the library while the item had only been added to the tracker. Prose guardrails did not hold. Deterministic receipts do.

Rule: anything a user might act on — created, added, deleted, scheduled — must be rendered by code from a tool result, not generated by the model.

---

Pattern 4 — Structured extraction via a terminating tool

For classification and extraction there is no need for session state, skills, or prose parsing. Define a tool whose parameters is the output schema:

{
  name: "emit_extraction",
  parameters: <generated JSON Schema>,
  constrainedSampling: { type: "json_schema", strict: "prefer" },
  async execute(_id, params) {
    return { content: [{ type: "text", text: "recorded" }],
             details: params, terminate: true };
  }
}

Run these calls with --no-session so they cannot pollute the conversational history, and at a lower thinking level than the conversational turn.

This removes an entire failure class. The curator scenario extracted JSON with re.compile(r"\{.*\}", re.DOTALL) — a greedy match from the first { to the last } — and reused one post-processor across three different schemas, which injected an items: [] key into every intent plan.

---

Pattern 5 — Separate the stateful and stateless roles

Role	Session	Skills	Thinking	Process
Conversation / answering	persistent, rotated	yes	high	long-lived RPC
Classification / extraction / synthesis	--no-session	no	low–medium	short-lived --print --mode json

Sharing one session across both roles causes mode confusion: the curator session alternated "只输出 JSON，不回答问题" and "直接回答用户，不要输出 JSON" for 37 consecutive turns, and each turn carried the previous turn's full fact payload.

---

Pattern 6 — Project a fact pack; do not hand over raw backend responses

Everything you put in a prompt is context you pay for and surface you expose. Project backend results through an explicit allowlist before they reach the model.

Observed in a single production curator prompt: filesystem paths (/mnt/truenas/multimedia/tv/…, /mnt/unRaid/tv4k/…), size_on_disk, quality_profile_id, internal database ids, a static capabilities block resent every turn, and an action_result.regular_matches object that duplicated verbatim an object already present under library.matches.

None of it was needed to answer "is there a 4K version".

Budget by estimated tokens, not characters — for CJK text the ratio is close to 1:1, so an 80 000-character cap is an 80 000-token cap.

---

Pattern 7 — Close the timeout budget, and kill the process group

Per-invocation timeouts do not compose. Define one deadline per user message, allocate it across stages, and enforce it with RPC abort rather than by killing the process.

When you do have to kill: pi is a node CLI that may spawn children. Launch with start_new_session=True and terminate with killpg, otherwise orphaned node processes keep consuming provider quota.

---

Pattern 8 — Keep provider and backend credentials out of the node process

subprocess inherits the parent environment by default. A gateway unit that carries *_API_KEY for Radarr, Sonarr, Plex, Telegram and a search provider hands all of them to pi and to every extension it loads.

Pass an explicit minimal env= (PATH, HOME, LANG, plus the few PI_*/bridge variables the extension genuinely needs). The provider key belongs in ~/.pi/agent/models.json, which pi reads itself.

Both scenarios in this repository currently inherit the full environment. This is the highest-value shared fix.

---

Pattern 9 — Bridge host capabilities over loopback HTTP

There is no RPC command to inject a tool result from the host, so host-side capabilities must be exposed as extension tools that call back out. memo-inbox does this for document parsing (127.0.0.1:8090) and OCR.

Recommended shape:

• backend listens on 127.0.0.1 only, with a per-start shared secret passed to the extension through env;
• backend serves both the tool definitions (JSON Schema) and the tool invocations, so the schema has one source of truth;
• the extension is a generic proxy — registerTool accepts a plain JSON Schema object (verified), so tool definitions need not be duplicated in TypeScript;
• every tool truncates its own output before returning.

---

Pattern 10 — Observability is already in the stream

Do not build separate instrumentation. message_update carries usage (input / output / cacheRead / cacheWrite / cost) and get_session_stats returns totals plus contextUsage. Persist per-turn model, thinking level, latency, tokens, cost and fallback into the service's own event ledger.

The curator scenario discarded stderr on success and kept only the last line on failure, so "how often does the primary model time out" was answerable only by grepping journald for a Chinese notice string.

---

Conformance

Pattern	memo-inbox	curator (as found)
1 long-lived RPC	yes	no — 2–4 spawns/message
2 session rotation	yes — 24 prompts	no — unbounded
3 deterministic receipts	yes	no — model prose, observed failure
4 terminating-tool schema	n/a (tools are the output)	no — greedy regex
5 role separation	n/a (single role)	no — shared session
6 fact-pack projection	n/a	no — raw responses
7 closed timeout budget	partial (600 s event wait)	no
8 minimal env	no	no
9 loopback bridge	yes	n/a (no tools at all)
10 usage observability	not persisted	not persisted

Shared implementations live in shared/lib/py/ and shared/extensions/.