[oblinder]

Summary

Readiness-driven onboarding for UC-1, the first part of D35 (docs/specs/PRD.md). The onboarding no longer waits inside the NATS handler for the rollout, the AgentCard and the tool manifest. It reads each readiness fact, and a fact that is not ready yet is nak'd and redelivered, so a slow workload does not reach the dead-letter queue.

The PRD narrows D35 to what this branch builds. Each layer that is not built is marked deferred.

Sync bugs fixed on this branch

1. AgentCard race (5d21357): the card was accepted before the operator set Synced=True. A timing gap could give the agent a wrong role set, and OPA then denied every call. The card wait now waits for Synced=True, and a card that is not synced is nak'd.
2. Rollout gate (91a6d0a): the pod wait could end mid-rollout and raise a permanent error for an old pod. The pod wait now reads the Deployment first and lists pods only when the rollout is complete. A pending rollout is nak'd.
3. Not-ready errors share the real-failure budget (5d21357, 91a6d0a): not-ready errors were not nak'd, so a slow workload could reach the dead-letter queue with no retry. WorkloadNotReadyError and AgentCardNotReadyError are nak'd (_NAK_ERRORS).
4. Dead-letter log filter (3d76caa): the filter searched for DLQ, but the subject is aiac.apply.dlq, so dead-letter moves never reached the test message.
5. Harness gate on a stale pod (fd6998e): an agent gate could pass on a pod that had no AuthBridge yet, when the operator was late.
6. Synced card selection and the paused-deployment message (a9a377a): a stale legacy card no longer hides the operator's synced card, and a paused Deployment is named as the cause.

Known limits and deferred layers

The PRD's D35 layers 5, 6, 9 and 10 are deferred. In particular:

• The nak delay is constant (default 30 s), and MAX_DELIVER is 5, so a late workload has a fixed number of looks.
• The start scan (D35 layer 9) is not built.
• The in-handler waits (the client wait, the pod and rollout waits, the tool MCP wait) still run inside the handler. D35 layer 1 says so.
• The bootstrap CR is dropped on any failure after the bootstrap, including a not-ready nak. It stays absent until the redelivery writes it again (docs/specs/components/aiac-agent.md, the Ack contract).

Harness gates in this PR

The system-test harness gains staged gates (test/system/uc1_onboard.py): the final template, the rollout, the operator log, the AgentCard, the CR content, and the probe hints. These are test-side work under the same decision, with no runtime change. Their pure deciders are pinned by test/unit/agent/uc/onboarding/test_uc1_staged_gates.py, spec'd in docs/testing/staged-gates-unit.md.

Review

Nine review rounds (mattpocock-skills:code-review, Standards and Spec axes). Round 1 raised 18 findings, and round 1 narrowed D35 to the built layers. Later rounds fixed the spec cites, the Ack contract, the REPO_ROOT index (which round 8 had broken and round 9 confirmed), the staged-gates spec, and the no-client log case.

Accepted exceptions (decided by the author):

• test_uc1_staged_gates.py imports the system harness, which links to demo/assets/ at import time. Fourteen existing unit files do the same. docs/testing/staged-gates-unit.md records it.
• Ten branch commits carry Co-Authored-By: Claude …. The repo's ai-assisted-by-trailer hook would rewrite it to Assisted-By. The author chose to keep the trailers and not rewrite history.
• The harness gates stay in this PR while D35 is still marked proposed.

Declined, with reasons:

• Duplicated _synced and duplicated DeploymentComplete rule, the harness file size, the repeated type switch in consumer.py, the divergent nodes.py, feature envy in _rollout_pending, and the data clump in _check_probes: out of scope for this change, reported only.
• The comment-only "handoff" removals (a5c7fbc): out of scope, left in place.

Open at exit (judgement only, no hard findings):

• The Ack contract's failed-nak wording: the sequence is recorded before the nak runs, so a failed nak is not reflected in the text.
• The PRD traceability list omits the new harness helpers probe_hint, signals_report, bundle_service_probe, controller_onboarding_lines and curl_pod.
• The onboarding_lines keep-order wording should say "input order".
• The probe-hint test asserts only the probe-pod wording, not the service wording.
• Standards smells: an unused containers=None kwarg in kube_fakes.py, a misleading ready name in the staged-gates test, a name prefix in curl_pod, a repeated template-metadata chain in uc1_onboard.py, and two Deployment builders with the same counters.

Checks

• Unit lane (-k uc1_staged_gates): 43 passed on ce4e4f7.
• ruff check and ruff format --check (pinned v0.14.8): clean on the changed Python file.
• markdownlint-cli2: 0 errors on the changed docs.
• checks.sh --base origin/main: all gates pass on the branch.
• System lane (pytest -m system): 87 passed, 0 failed, 0 errors, 14 skipped (the skips are in test/system/disabled/, intentional), on a9a377a. The branch has changed since then only in docs and in test/unit/, and in comments in test/system/uc1_onboard.py. src/ is unchanged since a9a377a. The system lane was not rerun on ce4e4f7, per the author's instruction.

🤖 Generated with Claude Code

[anatolykoyfman]

Reviewed the runtime changes (consumer.py, preconditions.py, provision/nodes.py, provision/kube.py, the RBAC rule), their unit tests, and the D35 / UC-1 spec updates. The unit lane passes locally on ce4e4f7 (1952 passed), and CI is green.

The core logic looks right to me:

• _rollout_pending matches the Deployment controller's DeploymentComplete test: observed generation, updated == spec, replicas == updated, available == spec. It also covers spec.replicas defaulting to 1 and zero counters that the API omits.
• Reading the Deployment before listing pods closes the stale-pod window. A 404 (StatefulSet / Sandbox) falls back to the plain pod list, and any other read error is still a 502 at once.
• When the rollout never completes, running #6 on the pod template is a good call. An httpGet probe through the sidecar would otherwise look like a race forever and only surface as a DLQ'd not-ready error.
• An AgentCardNotReadyError is raised in analyze_agent before provision_service writes anything. The bootstrap drop applies only to tools. So a nak there leaves nothing behind to compensate.
• Choosing the synced card over cards[0] fixes the case where a stale legacy card hides the real one.
• _not_visible_seqs is still set only for ServiceNotVisibleError, so a workload or card nak redelivery still waits for the client. The test pins this.
• The new apps/deployments get RBAC rule is the only manifest change needed. k8s/agent-deployment.yaml is the only manifest with the Controller's rules.

Non-blocking notes:

1. Behavior change for card-less agents. The old fallback ("partial: no AgentCard found, default scope assigned") is gone. An agent workload with no AgentCard, for example one with no protocol.rossoctl.io/<name> label, is now nak'd until MAX_DELIVER and then DLQ'd, where before it was onboarded with a default scope. I think this is the right fail-closed choice, and the UC-1 spec documents it. Still, it's worth calling out in the release notes / demo docs so nobody is surprised by a DLQ entry for a misconfigured agent.
2. The env name. AIAC_NOT_VISIBLE_NAK_DELAY_SECONDS now sets the delay for all three not-ready classes. The comment explains why, but a follow-up rename to something like AIAC_NOT_READY_NAK_DELAY_SECONDS, keeping the old name as a fallback, would make the k8s config easier to read.
3. The shared budget. As the description says, the not-ready naks still use the same 5 deliveries as real failures, so a workload that is ready after ~4 min still lands in the DLQ with nothing to retry it. That is acknowledged as deferred D35 layers (5/9). Just flagging it as the main thing still open before D35 moves out of "proposed".
4. The "open at exit" smells in the description (the unused containers=None kwarg in kube_fakes.py, the ready name in the staged-gates test, and the rest) are fine to leave for a follow-up.

Approving.

[oblinder]

System test lane: results

Result: 87 passed, 14 skipped, 0 failed, 0 errors (pytest -m system -v, exit 0, 1895 s).

Commit under test: ce4e4f7 (branch agent-onboarding-sync-bugs-fix). Cluster: Kind rossoctl, namespace aiac-system, workloads in team1.

Per file

File	Passed	Skipped	Errors
test/system/test_uc1_onboard_agent_only.py	5	0	0
test/system/test_uc1_onboard_agent_then_tool.py	23	0	0
test/system/test_uc1_onboard_tool_then_agent.py	19	0	0
test/system/test_uc1_onboard_resync.py	6	0	0
test/system/test_uc1_onboard_failure_rollback.py	17	0	0
test/system/test_policy_pipeline.py	17	0	0
test/system/disabled/test_launcher_select_live_pod.py	0	5	0
test/system/disabled/test_uc1_onboard_cross_service_conflict.py	0	1	0
test/system/disabled/test_uc1_onboard_policy_agnostic.py	0	8	0

The 14 skips are all in test/system/disabled/. They are skipped by their own markers and did not run. They are not coverage.

Platform preconditions (checked before the run)

• k8s/opa-kind-verify.sh: all checks passed.
• k8s/keycloak-spi-verify.sh: all checks passed.

Redeploy before the run

• Rebuilt and loaded localhost/aiac-pdp-config:local, localhost/aiac-pdp-policy-opa:local, localhost/aiac-policy-model-store:local, localhost/aiac-agent:local.
• Ran the agent LLM reload block from k8s/aiac-deployment-guide.md. The .env key returned HTTP 200.
• Re-ran keycloak-spi-install.sh with KEYCLOAK_IMAGE_BASE=quay.io/keycloak/keycloak:26.5.2, because the StatefulSet had no base-image annotation.
• Rebuilt the demo images with demo/assets/kind-load.sh --rebuild. The demo deploy.sh was not run.
• Health: agent, interface, PDP policy, and policy model store return {"status":"ok"}.

Not checked

• Whether the demo team1 workloads were removed at teardown.
• Whether the Direct Access Grants setting on the rossoctl login client was reset at session end.