ACP sessions become permanently wedged with 502 "The model is currently unreachable" while fresh sessions succeed

Summary
Long-running Antigravity ACP sessions eventually enter an unrecoverable state where every subsequent session/prompt fails server-side with error_code: 502 (“The model is currently unreachable.”). Once triggered, the session never recovers, even though freshly created sessions (session/new) on the same Google account and model (gemini-4-argon-high) work immediately.

Environment

  • Runtime: agy_acp_server 1.3.0
  • OS / Arch: macOS (arm64)
  • Model: gemini-4-argon-high
  • Accounts affected: Reproduced across two separate Google accounts

Observed Evidence (3 sessions on a single day, same Google account)

  • Session 7039f6f1: Worked for 469 steps, then began returning 502 on every subsequent prompt.
  • Session 679f9922: Worked for ~2 hours (140 steps), experienced intermittent 502 errors, then became permanently dead.
  • Session c8db93c2: Permanently died with 502 after only 69 steps.
  • Control check: Concurrent fresh session probes to gemini-4-argon-high on the same account succeeded immediately every time.
  • Transcript log: The failure is recorded server-side in the session transcript as a SYSTEM ERROR_MESSAGE step with error_code: 502 (“The model is currently unreachable.”).

Impact

  • Because 502 looks like a transient upstream error, ACP clients keep retrying the same warm session indefinitely.
  • Prompts hang without streaming events until a client-side 10-minute idle watchdog terminates the request.
  • All accumulated in-session context and work are lost once the session handle becomes poisoned.

Root-Cause Hypothesis
The backend appears to evict or orphan the session’s underlying model binding/worker (due to capacity pressure or TTL) without migrating the binding or invalidating the session handle. Subsequent prompts route to the dead binding and return 502 indefinitely instead of re-binding or failing the session cleanly.

Requested Fix

  1. Automatically re-bind active sessions when their backend model worker is evicted, or
  2. Prevent eviction of actively used sessions, or
  3. Return a distinct, terminal JSON-RPC/ACP error code (e.g., SESSION_EXPIRED / SESSION_EVICTED) instead of a generic 502 so clients can fail fast and recover via session/new.

Hi @Markus_Liang,

Welcome to the Forum.

Thank you for the detailed report on agy_acp_server 1.3.0. When a long-running ACP session accumulates a large transcript or encounters an error during checkpointing, subsequent session/prompt calls on that session’s history can repeatedly return 502 ("The model is currently unreachable.") from the local ACP proxy, while a fresh session/new succeeds immediately because it starts with a clean history.

To work around this and help us investigate further:

  • Update agy_acp_server to the latest release (v1.4.1+): Newer ACP server releases include reliability fixes and add the /usage slash command (v1.4.0) alongside real-time usage_update notifications (v1.3.0) to track cumulative session tokens (used vs. context window size).
  • Rotate sessions proactively: Monitor token growth via usage_update or /usage and start a fresh session/new when long multi-step runs approach context limits or first return a 502 error, rather than continuing to retry the wedged session handle.

If the issue persists on the latest agy_acp_server release, please reply here with the stderr logs around the first 502 failure (for example, running agy_acp_server with --debug --alsologtostderr) so we can pass this along to our engineering team.

That side-by-side breakdown of the three sessions (469 steps, 140 steps, and 69 steps) is super clear, @Markus_Liang. To build on @Rebbanpalli_Naveen’s recommendation to upgrade agy_acp_server to v1.4.1+ and monitor /usage, the reason session c8db93c2 wedged permanently at just 69 steps (well below the total context window limit) comes down to how agy_acp_server serializes in-flight tool and thought state:
Why a Single Interrupted Step Permanently Wedges a Session ID with 502: On gemini-4-argon-high, every turn appends signed thinking blocks and tool_call / tool_response pairs into the session’s protobuf transcript. If a single step times out mid-stream, gets cancelled, or records an oversized tool output / dangling tool_call without a matching response block, every subsequent session/prompt on that session ID replays the malformed history to the upstream endpoint. The upstream validator rejects the payload, and agy_acp_server surfaces that rejection as error_code: 502 ("The model is currently unreachable") indefinitely, which is why session/new on the exact same account works one second later.
Proactive /compact Checkpointing: Sending /compact as a session/prompt every ~45-50 steps (or right after any transient 502 retry) forces the ACP server to summarize the conversation history into a clean text checkpoint and prune the accumulated raw thinking and tool-call blobs before they can wedge the session.
Automatic State Hand-Off Wrapper for Long Autonomous Runs: In our ACP client loop, we wrap session/prompt so that if two consecutive calls on an active session return 502 while a lightweight health check passes, the client automatically calls session/new and injects the last saved progress summary plus git status --short and git diff --stat.