Skip to content

fix(strategy): bound turn checkpoint finalization - #2188

Open
MuskanPaliwal wants to merge 19 commits into
entireio:mainfrom
MuskanPaliwal:fix-turn-checkpoint-finalize-deadline-2148
Open

MuskanPaliwal wants to merge 19 commits into
entireio:mainfrom
MuskanPaliwal:fix-turn-checkpoint-finalize-deadline-2148

Conversation

@MuskanPaliwal

@MuskanPaliwal MuskanPaliwal commented Aug 31, 2026 •

Copy link
Copy Markdown
Contributor

Closes #2148

At turn end, finalize rewrites every checkpoint the turn created with the full transcript. Each rewrite can fetch refs and blobs from the remote. Every fetch has its own limit, and the dead-network memos keep a remote that is down from costing that limit once per checkpoint. Nothing limited the pass as a whole, though. A slow remote that still answers finishes each fetch just under its limit, trips neither memo, and the total outlasts the host's hook timeout. The host kills the hook before Entire saves session state. The checkpoints keep their provisional transcripts, and the only trace is a few warning logs.

The fix

The network phase of finalizeAllTurnCheckpoints, meaning the checkpoint store open and the write loop, now runs under a deadline. The deadline comes from the real limit the agent's host puts on the turn-end hook.

Finalize checks the deadline at entry and in the network phase, never in the middle of local work. If the deadline has already passed at entry, finalize doesn't start. Nothing it prepared could be written by then, and the hook needs that time to save state. Reading, sanitizing, externalizing and redacting the transcript is local work that never looks at the context, so nothing cuts it off halfway. Its time still counts against the budget. If it runs long, finalize stops at the first network step instead of starting fetches it can't finish. Finalize checks again after the store opens, so precompute doesn't hash the transcript for writes that will never start. A write that fails as the deadline passes logs its error, and finalize then counts it as an interruption.

The deadline is processStart plus a per-agent budget, the same pattern sessionEndCondenseDeadline uses. The host starts its clock when it spawns the hook, so Entire starts its clock at process start too. The budget comes from the agent running the hook, not from state.AgentType, which can be empty.

Each agent declares its host's turn-end hook timeout through a new agent.TurnEndHookTimeouter capability. Entire derives the budget from it in one place, so changing the timeout changes the budget too. #1237 just changed Claude Code's. agent.TurnEndBudgetWithin keeps a fifth of the timeout, between 2s and 10s, for saving state after finalize. A timeout of zero or less falls back to the default.

Agent Turn-end hook limit Source Budget
Claude Code 120s checkpointHookTimeoutSec 110s
Antigravity 300s stopHookTimeoutSeconds 290s
Factory AI Droid 60s Host default (docs) 50s
Codex 30s defaultHookTimeoutSec 24s
Copilot CLI 30s Host default (docs) 24s
Pi 10s The extension's execFile timeout, pinned by a test 8s
Cursor, OpenCode, Vogon, external agents Unknown or none defaultTurnEndHookTimeout (30s) 24s

TestTurnEndHookTimeoutLedger makes every built-in agent either declare a timeout or appear in turnEndHookTimeoutFallbacks with a reason, and it fails on a stale entry. A new agent can't pick up the default by accident. External agents always get the default, because the protocol has no field for a hook timeout.

When finalize is cut short

  • Checkpoints finalize already rewrote keep the full transcript. The rest keep their provisional one.
  • The unfinalized IDs stay in TurnCheckpointIDs, as they already do for the late-transcript and degraded-scanner exits. A second Stop in the same turn retries them with a fresh budget. Otherwise the next turn start clears them and the provisional transcripts stay for good. I expect the retry to be rare. Claude Code, Codex, Copilot CLI and most other hosts fire Stop once per turn, so for them a cut-off checkpoint keeps its provisional transcript. That's the price of finishing inside the host's limit, and I think it's the right trade.
  • Finalize sets CaptureDegradedAt, so entire status reports the problem. This now covers the host cancelling the hook too (SIGTERM ends the root context), which used to leave no trace.
  • Finalize records the interruption where it sees it rather than guessing afterwards. An ordinary write error doesn't count as degradation. One older path also keeps the IDs; see "Not included".
  • A cut-off checkpoint shows up once, as capture degradation, and not again as a finalize error.

Older Antigravity installs

The Antigravity budget assumes the 300s Stop timeout Entire installs. Installs from before bd1e2ff have no timeout, so agy's 30s default applies and kills the hook long before the 290s budget runs out. AntigravityAgent.RepairHookTimeouts lets entire doctor add the missing timeout without asking, the same repair Claude Code already has. It edits the raw JSON, so keys Entire doesn't model survive. It only touches Entire's own Stop handler, which it finds by command. It keeps a timeout that's already set under any key case, skips null entries, and leaves a disabled entry alone. The doctor message now covers an agent default that's too short as well as one that's too long.

Doctor only helps people who run it. So Antigravity now also reports a missing Stop timeout through CheckHookConfig, and entire status shows "Antigravity hooks out of date · run 'entire doctor'". --json lists the agent under hooks_outdated. The check and the repair share one function, so status flags an install exactly when doctor has a timeout to add. Running doctor always clears the warning. A user's own handlers, a timeout they set, or a disabled entry never trigger it.

Decision for reviewers

The 30s fallback is a judgment call. Cursor documents only a "platform default". OpenCode has no limit at all. Entire's plugin runs the hook through spawnSync, which blocks OpenCode's UI until entire exits, so the fallback is the only limit OpenCode gets. 30s matches what Codex and Copilot enforce. It's one constant in lifecycle.go.

Verification

Finalize deadline tests:

  • With no deadline, a future deadline, or slow fetches and no deadline, every checkpoint finalizes.
  • A deadline that passed before the network phase, a deadline that expires mid-loop while the parent context lives, a context that ends mid-loop, and a parent cancelled before start each leave the right checkpoints provisional. Each records degradation and counts no error.
  • An expired deadline at entry skips preparation, keeps the IDs and records degradation.
  • A deadline that passes while finalize prepares the transcript stops it before any fetch or precompute and keeps the IDs.
  • When the host cancels the hook during preparation, the repository open fails. Finalize counts that as an interruption, not an error, and keeps the IDs.
  • A second pass over kept IDs finalizes them and clears the list.
  • An invalid checkpoint ID, or a write that fails while the context is alive, counts as an error and not as degradation.
  • The degraded-scanner and late-transcript exits keep their IDs under an expired deadline.
  • The logs give the deadline as its own cause, so a host cancellation never looks like the budget running out.

Agent and budget tests:

  • Every agent that declares a timeout gets a budget above zero and below that timeout.
  • Pi's Go constant matches the .ts extension.
  • The ledger test above.

Antigravity tests:

  • The repair adds a missing Stop timeout. It keeps an existing timeout, including "Timeout" in another case, and keeps unmodeled keys, foreign entries, a user's handler in the Stop list, null entries and a disabled entry. It leaves the file alone when there's nothing to add.
  • The status check reports absent with no Entire entry, and current for a fresh install, an existing timeout in either case, a user's handler, or a disabled entry. It reports outdated only for Entire's Stop handler with no timeout.

How the timing tests stay deterministic:

  • In the parent-cancel mid-loop case, the fake fetcher ends the context itself.
  • In the deadline mid-loop case, the second fetch blocks until the deadline fires. A 10s fallback makes a regression fail instead of hang. The deadline is 3s because it also covers transcript preparation and the store open, which leaves a slow runner time to reach the first fetch.
  • The two preparation tests block the transcript read on a FIFO, so the deadline or the cancellation lands while preparation runs. Each test also fails if finalize exits before it reaches the read, so neither can pass on the entry check. They run on Unix only, since they need a FIFO.

I checked each new test against a planted regression, and each one failed. The regressions were the deadline check moved to the top, the checks around the store open removed, the repository-open interruption check removed, writes ignoring the deadline, cancellation not flagged, a write error flagged as degradation, interrupted IDs cleared or kept off by one, Pi's constant drifting, and a missing or stale ledger entry.

I also ran it end to end with a built binary. An Antigravity install with the Stop timeout removed shows the entire status warning. entire doctor adds the 300s timeout and the warning goes away. A fresh install shows no warning.

On the final commit, format, lint (0 issues), and the unit, integration and canary tests all pass.

Not included

  • No durable retry queue for cut-off checkpoints.
  • No change to the per-fetch budgets, and no batching of missing-blob fetches.
  • Entire doesn't read a hook timeout the user lowered themselves in Claude Code, Codex, Antigravity or Copilot CLI. The budget comes from the timeout Entire installs or the host default.
  • Entire doesn't write an explicit timeout into Cursor's hook config. That would swap the fallback for a known limit. But existing installs only get a hook config change through a doctor repair, so it belongs in its own PR.
  • Local turn-end work counts against the budget, because the clock starts at process start like the host's. For an agent with no host limit, such as OpenCode, finalize can now give up on checkpoints that used to finish when local work runs long. A longer default for agents we know have no limit would fix that. Follow-up.
  • Local preparation has no time limit once it starts. Finalize checks the deadline before and after it, so a long preparation never starts fetches, but preparation itself can eat into the headroom. Cursor's sidecar image read can take up to sqlite3Timeout (30s) against the assumed 30s hook limit. Pi has 2s of headroom to redact what may be a large transcript. The cheap fix is to cap the sidecar read at whatever deadline is left. Follow-up.
  • If a turn start is missed because its hook failed or was killed, the kept IDs carry into the next turn. That turn's Stop then rewrites the older checkpoints with the newer turn's transcript and prompts, and skips the prompt backfill. The late-transcript and degraded-scanner exits could already do this. A slow remote makes it more likely. Follow-up.
  • If the checkpoint store fails to open for any reason other than the deadline, finalize still returns with TurnCheckpointIDs set and doesn't record capture degradation. That predates this PR. Follow-up.

Entire-Checkpoint: 01M14QBVM11XQ1ZD7QCVZXMBST
Entire-Checkpoint: 01M1BDT4T6N6HDTEJCXM5WM0QW
@MuskanPaliwal
MuskanPaliwal marked this pull request as ready for review August 31, 2026 08:45
@MuskanPaliwal
MuskanPaliwal requested a review from a team as a code owner August 31, 2026 08:45
Copilot AI balanced review requested due to automatic review settings October 14, 2026 05:15

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

MuskanPaliwal and others added 12 commits October 14, 2026 16:23
Entire-Checkpoint: 01M4X26AHSD70GXQT6VXNAZT3N
…ast the deadline

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Entire-Checkpoint: 01M4X70BFCP0FFQTJ20Y7V0DVX
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Entire-Checkpoint: 01M4X789CH3XGK8PYK7ZSH7FNH
The deadline can pass while checkpoint.Open runs, and precompute would
then hash the whole transcript for writes that never start. Also log a
write error that coincides with the deadline instead of hiding it behind
the interruption, and widen the mid-loop deadline test so a slow runner
still reaches the first fetch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Entire-Checkpoint: 01M4X8H5085N6527PBH3HV9JK2
…eouts

An interrupted turn-end finalize now keeps the unfinalized
TurnCheckpointIDs, as the late-transcript and degraded-scanner exits
already do, so a repeated Stop in the same turn retries them with a
fresh budget. The five copied interruption checks collapse into
finalizeInterrupted.

The finalize deadline now comes from the agent running the hook rather
than state.AgentType, which can be empty.

Antigravity installs from before bd1e2ff have no Stop timeout, so
agy's 30s default applies while the budget assumes 300s.
RepairHookTimeouts lets `entire doctor` add the missing timeout,
editing raw JSON so unmodeled keys survive and set timeouts are kept.
The doctor message now covers a default that is too short as well as
too long.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Entire-Checkpoint: 01M4XZ6B1C1SQFVWK7TVHE6KST
…dline-2148' into fix-turn-checkpoint-finalize-deadline-2148
… repair

Finalize now checks the turn-end deadline before it starts. When the
lock wait and SaveStep have already spent the budget, the transcript is
no longer read and redacted for writes that can never start, so that
time stays available for saving session state. Local work that has
started is still never interrupted. The deadline carries its own cause,
so logs tell an exhausted budget apart from the host cancelling the
hook.

The Antigravity hook-timeout repair that doctor runs without asking now
touches only Entire's own Stop handler, matched by command. It treats a
timeout key in any case as already set, steps over null entries instead
of panicking on them, and leaves a disabled entry alone.

Agents now declare their host's turn-end hook timeout through
agent.TurnEndHookTimeouter, and Entire derives the finalize budget from
it in one place. A non-positive value falls back to the default instead
of meaning "skip", which matched neither the sibling SessionEndBudgeter
nor any agent's intent. The per-agent tests that restated the
implementation are replaced by one check over every declaring agent.

entire status names both causes of capture degradation again, and the
checkpoint scenario docs show that interrupted IDs are kept for retry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Entire-Checkpoint: 01M4ZBSJDK749JX37VPRSE974Q
Entire-Checkpoint: 01M4ZTEZZ6MXES3ZD7VQQRM78Z
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

finalizeAllTurnCheckpoints has no total deadline: a slow remote can outlast the agent's hook timeout

2 participants