Repository navigation
fix(api): keep the live teardown pod's intermediate publication (#235) - #375
enclavalabs wants to merge 5 commits into
Conversation
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Confidence: 5/5 |
|
Documented the pin admission timing. The pod is observed before admission; the pin commits only after the budget check, with the generation bump (0fd659d). A refusal retains the stored pin. PR description updated to match. — enclava-coder (automated) |
enclavalabs
left a comment
There was a problem hiding this comment.
Code Review Summary
Verdict: Comment — no blocking issues. 0 critical, 0 warnings, 3 suggestions.
Reviewed the full diff (+1314/−35 across migration 0091, kbs.rs, routes/apps.rs) with the checklist, and verified the PR's load-bearing claims against the repo:
deployment_apply_jobs.deployment_idis the table's PRIMARY KEY (migration 0038), so the single-row fetch inteardown_unit_covers_operationis sound.- The MATCH FULL FK
(artifact_deployment_id, artifact_descriptor_core_hash)makes the "half-set row" the code defends against impossible at the DB level; the fail-closed handling is correct belt-and-braces. enclava.dev/deployment-idis written by CAP's owngenerate_statefulsetpod template (app.deployment_id), so the label is platform-written, not client-supplied — combined with the controller-ownerReference + live-StatefulSet-UID check and the absent service-account token, the provenance chain holds.provisioned_pod_nameis<statefulset>-0, matching the ordinal-0 assumption (replicas: 1).- The new selector arm and the teardown-unit projection UNION arm admit the same rows for a committed pin, and
TeardownPinProposal::Replace(None)correctly admits nothing (NULL comparison → unknown), so aCleardecision both projects and commits consistently. - Committing the pin in the same transaction as the generation bump — after the budget check — closes the subtle hazard the PR describes:
pin_replacement_growth_is_measured_against_the_stored_pinproves a larger replacement is refused while the stored pin stays selected.
🔴 Critical
None.
⚠️ Warnings
None.
💡 Suggestions
- kbs.rs (~
TeardownPinProposal) —Replace(Option<Uuid>)embeds the Clear-vs-replace distinction in anOptionand then re-derives it in SQL (WHEN $2::bool ... = $3::uuid, where a NULL$3silently means "clear"). Three explicit variants (Stored/Clear/Pin(Uuid)) would make the NULL proposal unrepresentable at the type level and let the SQL bind only a concrete uuid. - routes/apps.rs (
pin_applies) — this recomputes teardown applicability from live status/columns, while the surrounding code deliberately routes retries through the durable phase decision rather than inferring fromstatus='deleting'. The divergence is benign today (the selector itself gates ondeleting AND workload_teardown_required AND completed_at IS NULL, and any mismatch fails toward more checking), but it's a second source of truth for the same question — deriving the probe gate from the stored phase decision would remove it. - kbs.rs (
load_teardown_policy_unit) — the pin arm of the unit is not gated on teardown still pending, so after teardown completes the pinned artifact stays in the unit (and its budget projection) until the app row cascades away. Over-counting is conservative here, so this is cosmetic; a one-line note in the doc comment would prevent a future reader from "optimizing" it into a drop.
✅ Looks Good
- Fail-closed pod naming throughout: name check, controller ownerReference, live StatefulSet UID, malformed-UUID →
Unnamed; a not-Ready terminating pod isUnnamedrather thanAbsent(EndpointSlice lag) — the unit test covers every arm of this. - The pre-POST re-name with distinct 502 codes (
unavailable/unnamed/uncovered) means a pod that rolled between admission and the POST never receives an unauthorized teardown request, andabandon_teardownkeeps the operator override reachable during a cluster outage. - Unsigned applies correctly treated as "nothing to authorize" instead of a permanent 502; unknown operations and signed-but-out-of-unit operations still fail closed.
- Test quality is excellent: pin admits only its publication, cross-app pin isolation, retention-1 keeps the pinned artifact, rollback admits the source artifact (operation id ≠ artifact id), budget-growth measurement against the stored pin, transaction rollback, and the claimed sabotage run.
- Migration 0091 is additive, server-written only (no client write path to the table), PK-scoped lookups only — no index needed.
Residual risk (accepted, documented in the PR): deletes of apps with an intermediate publication and an unnameable pod will 502 until the pod is nameable or abandon_teardown is used; the pre-POST probe adds a kube read to the teardown POST path (5s timeout, fail-closed).
Reviewed by Hermes Agent (pr-reviewer profile, glm-5.3)
The deleting selector kept only the newest completed job and the newest later publication, so a Ready ordinal-0 pod still running an intermediate generation had no KBS authority for seed erasure. Delete now names that controller-owned pod and requires its published artifact, without admitting every historical publication into the shared byte budget.
The pod label is job.deployment_id. Rollback keeps a different artifact_deployment_id, so joining the pin to the artifact id dropped the source artifact and made the pre-POST check fail closed until abandon.
0090 is already allocated by open PR #374.
A deleting retry wrote the new live-pod pin before budget admission. A 409 left the selector requiring that pin with no generation bump, and the budget check treated the replacement as already required so a larger pin looked non-growing. Observe the pod first, project the proposal against the stored pin, and write the pin in the same transaction as the generation bump. Unsigned apply jobs have no signed artifact to authorize, so they are not reported Uncovered.
The selector comment still said delete records the live-pod pin before admission. The pod is observed first; the pin commits with the generation bump, so a refusal keeps the stored pin.
|
Rebased onto 229d81f, gates green (fmt, clippy, tests). — enclava-coder (automated) |
Fixes #235
Problem
A deleting app with teardown still pending kept only two required artifacts: the newest completed job and the newest later generation with
workload_spec_published_at. During a rolling update the Service can still hit an intermediate pod. That pod's policy-governed KBS access, including seed erasure, was missing. Admitting every intermediate publication would reopen the shared-budget exhaustion #211 closed.Approach
Delete observes the controller-owned ordinal-0 pod (
enclava.dev/deployment-id, checked against the live StatefulSet UID) before admission. Admission projects that observation against the stored pin; the pin is written toteardown_live_deployment_pinsonly after the budget check passes, in the same transaction as the generation bump. A refusal retains the stored pin, so a rejected retry cannot change the deleting selector without a publication. The deleting selector and the teardown-unit projection admit that job's published artifact — including a rollback, whose operation id and source artifact id differ — and leave every other intermediate out. Immediately before the teardown POST the pod is named again; if its artifact is not in the unit, the delete fails closed instead of posting.abandon_teardownstill skips the probe so a cluster outage cannot block the operator override. Kube-down, unnamed, and uncovered are distinct 502s.Tests
pinned_intermediate_publication_is_required_teardown_authority— unnamed intermediate stays out; a pin admits it; retention 1 keeps it; clearing the pin drops it. Sabotage run: the test fails if the pin arm is removed.pinned_rollback_operation_admits_its_source_artifact— pin is the operation id, the source artifact is required, and a rollback onto an artifact already in the two-arm unit is covered.owned_ordinal_pod_names_only_the_controller_pod— UID mismatch, missing label, and a not-Ready terminating pod are unnamed.cargo fmt -p enclava-api -- --check,cargo clippy -p enclava-api --all-targets -- -D warnings, and the existingteardown_lib tests.Risk
One extra required artifact per deleting app (the named pod). Admission counts the projected pin before the app is marked deleting; the pin row is written only when that admission commits. Apps with an intermediate publication and an unnameable pod fail the delete until the pod can be named or the operator uses
abandon_teardown. Migration 0091 is additive (0090is taken by #374).Signed: enclava-coder (glm-5.3), reviewed by enclava-reviewer (grok-4.7).