Repository navigation
Conversation
|
Important Review skippedToo many files! This PR contains 315 files, which is 165 over the limit of 150. To get a review, reduce the PR to 150 files or fewer by splitting it into smaller PRs or changing its base branch. Upgrade to a paid plan to raise the limit. Usage-priced reviews support at most 300 files.
 |
Add requirements, system design, plan and work orders for opt-in host capabilities (workspace descriptor, repository intent, host-owned tunnel, drain receipt, destroy/recover checkpoints, durable bootstrap, crash barrier) so a worker-pool provider can ship as a plugin. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ty versioning first Add capability negotiation and versioning, shared environments, admission, pending placement, provider operations, launch policy, extensible preparation, and migration requirements and designs. Reorder the plan with M7 first and add work orders 01 and 09-15. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…n spec Add the profile schema, typed values, and secret references requirements and design, the min_kandev_version rule, the state budget, and the refused provision outcome. Add work order 17 and update the plan and fixture proof. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…in capabilities Map state schema onto the existing state-version mechanism, add the envelope byte budget and state_too_large rule, define shared-owner rows, admission order, cancel versus completion, migration source and routes, secret references on main, and align the work orders. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…in capabilities Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…in capabilities Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…pecs Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…in capabilities Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…le (round 5) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…findings Cut M8 (MIGRATION-001, work order 15) to Out of scope, follow-up card a8546b99. Resume keeps its reservation and goes through attachRetainedPluginExecutor with the same gates; deny refuses, hook outage resumes. Close L1, L2, L4, L6-L8, L12-L16. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…a follow-up card Cut resource classes, labels and provider-reported admission values to card 11c92feb. Resume never takes the scope lock; placement_failed/timeout recovery specified against recoverPluginExecutor; designs and work orders aligned. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…sume fence findings Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…yntax Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…tion echo Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…utor state The lifecycle confirms the manifest, registry and accepted echo before any provider call on first launch and on retained attach, pins the negotiated set into the plugin_executor envelope, refuses oversized provider state with state_too_large (cleaning the resource), and reports the affected environment count when an upgrade would strand recorded state. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…admission Add the per-installation readability lock shared by upgrade activation and the admission checkpoint, the writable checker wired from the plugins service, and the admin read of the host capability registry. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…eferences Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… environment Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…th sequence fencing A provider that negotiates destroy_checkpoints can checkpoint under the active cleanup or recovery claim. The repository validates the claim, the per-claim sequence and the envelope revision in one transaction, so a released claim, a successor claim or a competing checkpoint changes nothing. Settlement to absent now also requires a stored checkpoint of the claim. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…lugin environment Add the agentctl runtime seal-and-drain endpoint with claim-keyed receipts, the drain_receipt capability with drain_outcome and a receipt on the destroy request, destroy_timeout_seconds in the manifest, and an idle_timeout_seconds bootstrap field so a drainable runtime keeps its idle reaper off. Only task teardown cleanup drains; ordinary stop, backend shutdown and plugin disable never seal the runtime. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ry claim Guarded, idempotent flip of a creating environment to ready on both dialects: the held claim, session, generation, runtime identity, phase, workspace descriptor, repository inventory and revision are re-checked in the publishing transaction, and a mismatch publishes nothing. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…er the recovery claim Repeat the agentctl handshake with the stored one-time nonce, adopt and publish the environment when it is accepted, and clean it up as bootstrap_consumed when agentctl rejects the nonce. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…API URLs Adds the runtime_api_url manifest key (host_public default, provider_forwarded with api_url_forward), the unauthenticated read-only runtime ping route backed by a one-time probe token, the agentctl probe-host-api control route, and a forwarded-launch probe that rolls the environment back as api_unreachable. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…uild record Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A lease transport of host_tunnel makes Kandev open bidirectional streams to the provider, bound to installation, environment, ownership generation and one allowed port, acknowledged before any request byte flows. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…agent launch Includes the tests, which were written and seen failing first but cannot pass the per-commit package lint without the implementation.
…ugh runner closed
…nd passthrough runner closed Tests trust t.TempDir() through an unexported, test-only boundary seam; the production boundary is the filesystem root.
… legacy remote execution A recovered execution on an agentctl without retained storage pins no stream, so the replay preflight asked for the session ID while the peer names its stream <incarnation>:g<generation>. The peer answered OWNER_MISMATCH, the attach failed on every retry and the session stayed behind the recovery guard. Ask the peer for its current stream and verify the owner identity instead.
…l under the environment workspace Without a journal the peer is a legacy delivery peer, so a recovered execution stays Running and never takes the durable Ready route after a backend restart. Place the journal under the provider-declared workspace, as the SSH and Kubernetes executors do, keyed by the session owner.
Adds the worker_api_url capability and an optional worker_api_url field on the provision resource descriptor. A remote worker cannot reach the host's own listener, so the reachability probe uses the loopback forward the provider reports when it opted in.
…tored phase A provider checkpoints artifact_staging and bootstrapping from inside its provision call, so the launch's own copy of the inventory phase can be behind the stored row. The rollback of a failed launch recorded its drain outcome with that stale phase, the store refused the move back to allocating, and the rollback gave up with the cleanup pending. The environment reset that retried it later had no handshake to prove nothing ran, reported the runtime as unreachable, and the provider quarantined a healthy node. Record the drain outcome as cleanup_pending, which every live phase may reach.
…ied allocation directory
…o the process owner
…ion directory tests
…dering Add evidence tests for the launch-failure work, with no behaviour change: - every agentctl control and instance route outside the documented exemptions rejects a request without the token - no instance, process manager or launched process exists before an authenticated create, and startup launches exactly the orphan sweep - loading a node launch wrapper reads it and never executes it - the agentctl token reference is checkpointed before CreateInstance on the launch and durable-bootstrap recovery paths
Version 2 reuses not_attempted for 'the host never held an agentctl credential for this runtime'. The registry accepts versions 1 to 2, a manifest names exactly one, and version 1 is unchanged. The proto comment and the contract documents carry the stated preconditions and the risks.
…eceipt version 2 When no credential is available and no handshake was ever started, a record that pinned drain_receipt@2 sends not_attempted for every cleanup reason. The version is read from the stored record. Version 1 is unchanged, and the fixture provider accepts the wider meaning only when it declares version 2.
…tial Adds the sticky envelope fields credential_recorded, set in the checkpoint that names the token secret in both the launch and the recovery path, and handshake_rejected. Under drain_receipt@2 with durable_bootstrap a started handshake that never produced a recorded credential and was not rejected is not_attempted; a rejected handshake, a recorded credential with an unreadable secret, a non-durable provider and a record without the field stay unreachable.
…ider checkpoint keeps it
…restart A plugin_remote session waiting in the pending recovery queue whose agentctl is gone returned an empty recovery outcome, so the archive stop failed and retried forever and DestroyTaskEnvironment never ran. A cleanup stop now completes and leaves the stored record to the environment cleanup, which drains from the record (unreachable when a credential was recorded).
# Conflicts: # apps/backend/internal/agentctl/server/process/manager.go # pluginsdk/proto/kandev/plugin/v1/plugin.pb.go
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tip
PR walkthrough: Open the visual walkthrough
Remote executor plugins cannot own an environment safely today: the host gives them no workspace path, no repository intent, no proof the agent stopped before destroy and no durable record of a half-finished cleanup, so a crash or an archive mid-launch can orphan the remote resource. This adds those host capabilities behind negotiated, versioned capabilities, so a plugin that opts in can provision, connect, drain, destroy and recover without leaking, and plugins that opt into nothing behave as before.
Important Changes
cleanup_progressandrecovery_progresscheckpoints fence stale writers.not_attemptedis sent only withlaunch_failedand only when no agentctl handshake ever started; each cleanup claim acquisition has a unique id that the provider echoes on checkpoints; a bounded periodic sweep retries handoffs and retainedcleanup_pendingrows.unrecorded_plugin_executorsand destroyed by the cleanup under its own claim, so archiving during provisioning does not orphan it. That destroy carries the launch cleanup reasonlaunch_failedanddrain_outcome: not_attempted, the no-runtime evidence a launch rollback sends.cleanup_progresscheckpoint may omit the workspace descriptor while none is recorded, so a provider can checkpoint before destroying a resource a launch handed off before binding a node. The periodic sweep leaves a target whose task has an active cleanup job to that job.plugin_remoteonly): the graceful agentctl stop and its lifecycle-lock wait are bounded (15 s or a third of the remaining archive budget), so a frozen agentctl no longer uses up the archive budget. A busy lock reports the stop as unconfirmed and the existing cleanup runs: drainuncertain, then the provider retains and may quarantine the node. Other runtimes keep the old path; no SDK or contract change.file://repository remote is accepted, cloned without credentials and takes no Git credential lease, and agentctl's materialize endpoint accepts a host-lessfile://locator, and the host forwardsKANDEV_E2E_MOCKto the provider's agentctl; production is unchanged.docs/specs/executors/requirements/remote-executor-plugin-native-workers.mddocs/specs/executors/system-design/remote-executor-plugin-native-workers.mddocs/plans/executor-plugin-native-workers/plan.md(work orders 00-08, 17)Validation
go test -count=1(lifecycle with-race) over./internal/plugins/... ./cmd/plugin-fixture/... ./internal/task/... ./internal/agent/runtime/lifecycle/ ./internal/orchestrator/executor/ ./internal/gitcredentials/ ./internal/repoclone/.8962b8bdeb,./internal/agentctl/server/api/ ./internal/orchestrator/executor/ ./internal/task/service/ ./internal/repoclone/ ./internal/plugins/...: 7191 passed, 194 failed../internal/agent/runtime/lifecycle/with-race: 5083 passed, 28 failed (counts include subtests).173d1ef2d8(the same 194 and 28 tests, none new): host-dependent worktree, Kubernetes-prepare, alias and archive-reclaim, and repository-selection tests.cd pluginsdk && go test ./...: 164 passed, 0 failed.8962b8bdeb: smoke, archive during provisioning (5 runs), restart during provisioning, and loopback http andfile://repository runs all pass, in dev mode and with TLS, ACLs and a scoped token.f257db1d49. The node launch wrapper commit (2e9d23fad6) is not covered by those live runs; its checks are the unit tests listed in the node-wrapper section.f257db1d49(this PR), same rig in secure mode: that 9-scenario suite passes; a frozen-agentctl archive returns in about 40 s (it returned 500 after 180 s on8962b8bd), the provider quarantines the node about 70 s after the freeze, a second launch is refused, operator readmission works and the next task runs; one real Codex agent turn (codex-acp) passes end to end. The dev-mode suite passed one commit earlier, atc100d78791.8962b8bdeband pass on the head. The lifecycle package has the same 25 failing tests (host-dependent, e.g. macOS temp-dir and symlink paths) on8962b8bdeband on the head; none fail only on the head.Node launch wrapper (added on this branch)
An operator-owned launch wrapper, set per node and not per task, can be required for every agentctl agent and utility launch. It is generic: core knows only a wrapper argv prefix and a required flag, and no sandbox product name appears in core.
docs/specs/agents/requirements/node-launch-wrapper.mdanddocs/specs/agents/system-design/node-launch-wrapper.md. The "no executor-level wrapper" exclusion ingeneric-launch-wrappers.mdis narrowed to point at them.argv[0], which must be an absolute path. The owner is set by flag, default is the agentctl process uid.Restart fix (follow-up message after a backend restart).
After a survivable backend shutdown, a follow-up message to a running
plugin_remotetask got no reply. Two causes:409 OWNER_MISMATCHand recovery failed every 60 s. Fixed by reading the peer's own stream for a legacy execution with no pinned stream, and checking session, incarnation and generation match (114a85bc8b).plugin_remoteagentctl had no durable journal, so every recovered execution was legacy: published Running, never Ready, and a later launch hitcannot configure while agent is running. It now gets a journal under the provider-declared workspace, like Kubernetes (f7860bcaf9). The plugin's workspace removal deletes it with the environment. No SDK, proto or other-runtime change.Live (local attach, mock agent, branch build): message right after restart replied at about +61 s (was none in 90 s); message 70 s after restart replied in 1.0 s (was session FAILED); default suite 9/9.
Limits: environments created before this change stay on the legacy path until recreated; the first reply waits for the 60 s provider retry; the durable-bootstrap re-create has no journal; only the mock agent was tried.
Worker-side API URL for the readiness probe (
7530eb2735).After provisioning, core probes the host API through the worker. On a remote worker (ssh node) the host URL
127.0.0.1:<kandev port>is not the host, so the probe failed withapi_unreachablealthough the provider's forward was up. A provider can now return an optionalworker_api_url(proto field 11, additive) and core probes that URL when set, elsehost_api_urlas before.worker_api_urlv1, requiresapi_url_forward. A provider that did not negotiate it and sends the field is refused; one that does not opt in behaves exactly as today.http, loopback host (127.0.0.1,::1,localhost), a port, no userinfo, path only/, no query or fragment. Only forprovider_forwardedlaunches. Agentctl still forces the ping path.sdk-release.jsonneeds achange_kindbump.api_unreachable; after, smoke replied in 8.1 s; unblocked control passed; default suite 9/9; ssh-restart pass. A patched plugin on the old core is refused at install (unknown capability), an unpatched plugin on the new core behaves as before.Failed-launch drain fix (
f70b18657d). Found on a real ssh node:agentctldied at startup and the node was quarantined. A provider checkpointsartifact_stagingandbootstrappingfrom inside its provision call, so the launch's stored phase was ahead of the launch's local copy; the drain outcome was written with the stale phase and the store refused the move back toallocating, so the fallbackcleanup_pendingwrite ran and the provider never got thelaunch_faileddestroy. The drain outcome is now recorded ascleanup_pendingwith the retained resource from the start. Tests: a failed provision after the provider checkpointed rolls back as absent; no handshake givesnot_attempted, a handshake without a credential givesunreachable, a handshake with a credential givesdrained. Local rig (agentctl exits 1 at startup): node no longer quarantined with this fix and the plugin fix; default suite 9/9, restart-message pass. Open: core's own handshake failing because agentctl died still reportsunreachable.Allocation directory token for the node wrapper (
dd5007f704,9f3c1ceb2a,14853d2601). A node wrapper policy may use{{kandev.alloc_dir}}, a per-launch directory owned by the node's launcher and outside the agent's writable workspace. agentctl resolves it only from--node-launch-wrapper-alloc-dir(never from the environment or a request), requires an absolute, resolved, existing directory owned by the process owner, not group or world writable and not inside the workdir, and refuses a policy that uses the token without a value. REQ-AGENTS-NODEWRAP-006 AC .1 to .6; thenode-launch-wrapper.v1capability is unchanged (an older agentctl rejects the unknown flag and the token, so it never starts unwrapped). Tests cover token placement, each refusal, the value never coming from a request or the environment, and the unused-value case. Live (local rig): a recording wrapper received the directory; default suite 9/9, restart-message pass. Not proven: the real sandbox runner on a real node.drain_receipt@2: a failed launch no longer quarantines a clean node (c7e889b40c,b815d79db3,f425b5a158,9653ec926f, plus pinning tests06755defc3).not_attemptednow means the host never held an agentctl credential for this runtime. Gated by a new negotiateddrain_receipt@2(registry Min 1, Max 2;@1hosts and records behave exactly as before; the version comes from the record's stored capabilities). Under@2: no handshake ever started plus no token givesnot_attemptedfor any cleanup reason (waslaunch_failedonly); a handshake that started but never produced a recorded credential (EOF, deadline, connection refused) also givesnot_attempted, via two new sticky envelope fieldscredential_recorded(set in the same checkpoint that names the token secret, in both the launch and the durable-recovery paths) andhandshake_rejected. A rejected handshake stays strict (unreachable), as do a recorded credential whose secret is unreadable, a non-durable provider after a started handshake, and any record without the new fields. The plugin still verifies the job, terminal allocations, node contract and worker state; only the drain is skipped. Preconditions are stated in the spec: agentctl starts no agent, shell or managed process before an authenticated request when a bootstrap nonce is configured (with none, authentication is off on loopback), and its only startup process is a code-server orphan sweep (pgrep, may SIGKILL an orphaned code-server host-wide; recorded as a known hazard, not changed). New spec filesremote-executor-plugin-native-workers-drain-receipt-v2.md(requirements and system-design); the D1 table, AC 005.4 and capability-versioning are updated; the proto change is comment-only. Pinning tests prove the premises: only documented routes accept requests without a token, spawning routes start nothing unauthenticated, the wrapper is validated and never executed at startup, and the token reference is checkpointed beforeCreateInstancein both paths. Lifecycle failures equal base (the same 24 environment tests).Decided on this branch:
argv[0]must be an absolute path, soPATHcannot redirect the wrapper.node-launch-wrapper.v1. Limit: a wrapper the backend was never told about is not detected, and that runner behaves as before (AC-AGENTS-NODEWRAP-005.4).Not covered, follow-ups:
node-launch-wrapper.v1before dispatch. The worker already fails closed, so this is a clearer error and defense in depth, not a bypass.Tests: new tests in
launchwrapper,process(node_wrapper_test.go),utility(node_wrapper_test.go),apiandcmd/agentctl, including the composition-order testTestNodeWrapperIsFirstArgvOfSessionCommand. Run locally:-racepasses for the new tests.TestGenericWrapperPreparefails on this branch and onf257db1d49alike (macOS/varvs/private/var), so it is pre-existing and not changed here.Commits: the spec is a separate commit. The code and tests are one commit (26 files), because the per-commit lint needs the implementation to compile with the tests. The PR is one logical change, but it is large; splitting is possible if reviewers ask.
Possible Improvements
Medium risk: persistence and lock-order changes. Recorded secrets are revealed to the provider on every attach, inspect, resolve and tunnel-open call, because the provider needs its cluster credential each time. A lease-scoped credential would narrow that.
Checklist
apps/web/), I have added or updated Playwright e2e tests inapps/web/e2e/and verified them withmake test-e2e.docs/public/**and updated them or noted why no docs change is needed.Preview Environment
f274c85