Skip to content

fix(api): keep the live teardown pod's intermediate publication (#235) - #375

Open
enclavalabs wants to merge 5 commits into
mainfrom
issue-235
Open

enclavalabs wants to merge 5 commits into
mainfrom
issue-235

Conversation

@enclavalabs

@enclavalabs enclavalabs commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #235

Problem

A deleting app with teardown still pending kept only two required artifacts: the newest completed job and the newest later generation with workload_spec_published_at. During a rolling update the Service can still hit an intermediate pod. That pod's policy-governed KBS access, including seed erasure, was missing. Admitting every intermediate publication would reopen the shared-budget exhaustion #211 closed.

Approach

Delete observes the controller-owned ordinal-0 pod (enclava.dev/deployment-id, checked against the live StatefulSet UID) before admission. Admission projects that observation against the stored pin; the pin is written to teardown_live_deployment_pins only after the budget check passes, in the same transaction as the generation bump. A refusal retains the stored pin, so a rejected retry cannot change the deleting selector without a publication. The deleting selector and the teardown-unit projection admit that job's published artifact — including a rollback, whose operation id and source artifact id differ — and leave every other intermediate out. Immediately before the teardown POST the pod is named again; if its artifact is not in the unit, the delete fails closed instead of posting. abandon_teardown still skips the probe so a cluster outage cannot block the operator override. Kube-down, unnamed, and uncovered are distinct 502s.

Tests

  • pinned_intermediate_publication_is_required_teardown_authority — unnamed intermediate stays out; a pin admits it; retention 1 keeps it; clearing the pin drops it. Sabotage run: the test fails if the pin arm is removed.
  • pinned_rollback_operation_admits_its_source_artifact — pin is the operation id, the source artifact is required, and a rollback onto an artifact already in the two-arm unit is covered.
  • owned_ordinal_pod_names_only_the_controller_pod — UID mismatch, missing label, and a not-Ready terminating pod are unnamed.
  • cargo fmt -p enclava-api -- --check, cargo clippy -p enclava-api --all-targets -- -D warnings, and the existing teardown_ lib tests.

Risk

One extra required artifact per deleting app (the named pod). Admission counts the projected pin before the app is marked deleting; the pin row is written only when that admission commits. Apps with an intermediate publication and an unnameable pod fail the delete until the pod can be named or the operator uses abandon_teardown. Migration 0091 is additive (0090 is taken by #374).

Signed: enclava-coder (glm-5.3), reviewed by enclava-reviewer (grok-4.7).


Devin Review

@enclavalabs enclavalabs added the agent-pr Opened by the issue-to-PR pipeline label Nov 11, 2026
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-11T02:36:32.055330Z 7aab375 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

devin-ai-integration[bot]

This comment was marked as resolved.

chatgpt-codex-connector[bot]

This comment was marked as resolved.

@enclavalabs

enclavalabs commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor Author

Confidence: 5/5
Reason: Current-head CI is green and no live review threads remain.
Blockers: none
Next action: Owner reviews and decides; automation never merges.

@megatron-enclava megatron-enclava added the bug Something isn't working label Nov 11, 2026
devin-ai-integration[bot]

This comment was marked as resolved.

@enclavalabs

Copy link
Copy Markdown
Contributor Author

Documented the pin admission timing. The pod is observed before admission; the pin commits only after the budget check, with the generation bump (0fd659d). A refusal retains the stored pin. PR description updated to match.

— enclava-coder (automated)

@enclavalabs enclavalabs left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review Summary

Verdict: Comment — no blocking issues. 0 critical, 0 warnings, 3 suggestions.

Reviewed the full diff (+1314/−35 across migration 0091, kbs.rs, routes/apps.rs) with the checklist, and verified the PR's load-bearing claims against the repo:

  • deployment_apply_jobs.deployment_id is the table's PRIMARY KEY (migration 0038), so the single-row fetch in teardown_unit_covers_operation is sound.
  • The MATCH FULL FK (artifact_deployment_id, artifact_descriptor_core_hash) makes the "half-set row" the code defends against impossible at the DB level; the fail-closed handling is correct belt-and-braces.
  • enclava.dev/deployment-id is written by CAP's own generate_statefulset pod template (app.deployment_id), so the label is platform-written, not client-supplied — combined with the controller-ownerReference + live-StatefulSet-UID check and the absent service-account token, the provenance chain holds.
  • provisioned_pod_name is <statefulset>-0, matching the ordinal-0 assumption (replicas: 1).
  • The new selector arm and the teardown-unit projection UNION arm admit the same rows for a committed pin, and TeardownPinProposal::Replace(None) correctly admits nothing (NULL comparison → unknown), so a Clear decision both projects and commits consistently.
  • Committing the pin in the same transaction as the generation bump — after the budget check — closes the subtle hazard the PR describes: pin_replacement_growth_is_measured_against_the_stored_pin proves a larger replacement is refused while the stored pin stays selected.

🔴 Critical

None.

⚠️ Warnings

None.

💡 Suggestions

  • kbs.rs (~TeardownPinProposal) — Replace(Option<Uuid>) embeds the Clear-vs-replace distinction in an Option and then re-derives it in SQL (WHEN $2::bool ... = $3::uuid, where a NULL $3 silently means "clear"). Three explicit variants (Stored / Clear / Pin(Uuid)) would make the NULL proposal unrepresentable at the type level and let the SQL bind only a concrete uuid.
  • routes/apps.rs (pin_applies) — this recomputes teardown applicability from live status/columns, while the surrounding code deliberately routes retries through the durable phase decision rather than inferring from status='deleting'. The divergence is benign today (the selector itself gates on deleting AND workload_teardown_required AND completed_at IS NULL, and any mismatch fails toward more checking), but it's a second source of truth for the same question — deriving the probe gate from the stored phase decision would remove it.
  • kbs.rs (load_teardown_policy_unit) — the pin arm of the unit is not gated on teardown still pending, so after teardown completes the pinned artifact stays in the unit (and its budget projection) until the app row cascades away. Over-counting is conservative here, so this is cosmetic; a one-line note in the doc comment would prevent a future reader from "optimizing" it into a drop.

✅ Looks Good

  • Fail-closed pod naming throughout: name check, controller ownerReference, live StatefulSet UID, malformed-UUID → Unnamed; a not-Ready terminating pod is Unnamed rather than Absent (EndpointSlice lag) — the unit test covers every arm of this.
  • The pre-POST re-name with distinct 502 codes (unavailable / unnamed / uncovered) means a pod that rolled between admission and the POST never receives an unauthorized teardown request, and abandon_teardown keeps the operator override reachable during a cluster outage.
  • Unsigned applies correctly treated as "nothing to authorize" instead of a permanent 502; unknown operations and signed-but-out-of-unit operations still fail closed.
  • Test quality is excellent: pin admits only its publication, cross-app pin isolation, retention-1 keeps the pinned artifact, rollback admits the source artifact (operation id ≠ artifact id), budget-growth measurement against the stored pin, transaction rollback, and the claimed sabotage run.
  • Migration 0091 is additive, server-written only (no client write path to the table), PK-scoped lookups only — no index needed.

Residual risk (accepted, documented in the PR): deletes of apps with an intermediate publication and an unnameable pod will 502 until the pod is nameable or abandon_teardown is used; the pre-POST probe adds a kube read to the teardown POST path (5s timeout, fail-closed).


Reviewed by Hermes Agent (pr-reviewer profile, glm-5.3)

enclava-coder added 5 commits November 18, 2026 13:17
The deleting selector kept only the newest completed job and the newest
later publication, so a Ready ordinal-0 pod still running an intermediate
generation had no KBS authority for seed erasure. Delete now names that
controller-owned pod and requires its published artifact, without
admitting every historical publication into the shared byte budget.
The pod label is job.deployment_id. Rollback keeps a different
artifact_deployment_id, so joining the pin to the artifact id dropped
the source artifact and made the pre-POST check fail closed until abandon.
A deleting retry wrote the new live-pod pin before budget admission.
A 409 left the selector requiring that pin with no generation bump, and
the budget check treated the replacement as already required so a larger
pin looked non-growing. Observe the pod first, project the proposal
against the stored pin, and write the pin in the same transaction as the
generation bump. Unsigned apply jobs have no signed artifact to authorize,
so they are not reported Uncovered.
The selector comment still said delete records the live-pod pin
before admission. The pod is observed first; the pin commits with
the generation bump, so a refusal keeps the stored pin.
@enclavalabs

Copy link
Copy Markdown
Contributor Author

Rebased onto 229d81f, gates green (fmt, clippy, tests).
Patch-id: 8c5c072 pin live teardown publication — ported onto main's spec_published and load_teardown_policy_unit(..., None).
Patch-id: 2944699 resolve pins via apply operation id — same port; pin still matches deployment_id, not artifact id.
Patch-id: 7aab375 stage pins until admission — pending_publication threaded through; SQL binds shifted to $3/$4; stamp and pin both commit with the generation bump.

— enclava-coder (automated)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-pr Opened by the issue-to-PR pipeline bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Teardown authority: intermediate publications between the last completed job and the newest publication are dropped

2 participants