Skip to content

fix: correct exclusive topology scheduling - #588

Open
QIW9 wants to merge 4 commits into
sgl-project:mainfrom
QIW9:fix/exclusive-topology
Open

QIW9 wants to merge 4 commits into
sgl-project:mainfrom
QIW9:fix/exclusive-topology

Conversation

@QIW9

@QIW9 QIW9 commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Ⅰ. Motivation

The default exclusive-topology path generates Pods with the group-uid label, but the required affinity selects group-unique-hash. The first Pod therefore cannot match its own affinity and remains Pending.

Ⅱ. Modifications

  • Use group-uid for exclusive-topology affinity and anti-affinity.
  • Preserve existing workload selectors and report conflicting group-uid values.
  • Reconcile RBGs when the exclusive-topology annotation changes.
  • Update the examples and documentation to use the current annotation and template structure.

Ⅲ. Does this pull request fix one issue?

Fixes #518

Ⅳ. List the added test cases

  • Unit tests for selector matching, disabled roles, legacy selectors, conflicting labels, and annotation changes.
  • End-to-end tests for same-group placement, anti-affinity against other groups, disabled roles, and annotation updates.

Ⅴ. Describe how to verify it

make fmt

go test ./pkg/reconciler ./internal/controller/workloads -count=1

# Set KUBEBUILDER_ASSETS to Kubernetes v1.31.0 envtest binaries.
go test ./test/envtest/testcase/rbg -count=1 -timeout=8m

go vet ./pkg/reconciler ./internal/controller/workloads \
  ./test/e2e/framework/workloads ./test/e2e/testcase/v1alpha2

The focused end-to-end suite passed 5 tests on an isolated single-node Kubernetes v1.35.1 cluster. It can be run against a dedicated cluster with the RBG controller installed:

# Set KUBECONFIG and E2E_DEFAULT_IMAGE for the test cluster.
go test -v ./test/e2e -run '^TestE2E$' -count=1 -timeout=8m -args \
  -ginkgo.v \
  -ginkgo.focus='rbg with exclusive-topology|schedules exclusive groups together|blocks an exclusive group from a node|uses a disabled role|reconciles exclusive topology annotation changes' \
  -ginkgo.label-filter='!v1alpha1 && !volcano && !scheduler-plugins'

VI. Special notes for reviews

  • Existing RBGs already configured for exclusive topology may receive new Pod-template revisions and replace Pods after the controller upgrade. In the local binary-swap test, a workload without exclusive topology kept the same Pod UID and revision.
  • Anti-affinity now matches other Pods carrying group-uid in the same namespace, including groups that do not enable exclusive topology. It does not exclude unlabeled Pods or Pods in other namespaces.

Existing workloads

On the broken default path, the first Pod normally remains Pending because it does not carry the label selected by its own affinity. For that path, this change does not introduce a dedicated migration gate for running Pods.
This does not mean that all existing workloads are unaffected. Pending Pods may be replaced, and users who supplied matching labels or custom affinity rules may have running Pods whose templates change and roll out after the upgrade.
In the local binary-swap test, an RBG without exclusive topology retained its Pod UID and revision.

Checklist

  • Format your code make fmt.
  • Add unit tests or integration tests.
  • Update the documentation related to the change.

QIW9 added 2 commits November 14, 2026 18:33
Signed-off-by: QIW9 <qiwang922337@gmail.com>
Signed-off-by: QIW9 <qiwang922337@gmail.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Existing exclusive-topology workloads can retain broken selectors after upgrade because podSpecEqual ignores affinity changes, allowing reconciliation to skip the corrective update.

2 open issues
What changed in this PR

This PR addresses #518 by aligning exclusive-topology scheduling rules with the group-uid label emitted by RBG, allowing the first Pod to match its own required affinity.

Changes:

  • Correct affinity selectors while preserving workload selectors and reporting conflicting group identities.
  • Reconcile exclusive-topology annotation changes and add unit and end-to-end coverage.
  • Update examples and documentation, including scheduling scope and upgrade effects.
File Description
test/​e2e/​testcase/​v1alpha2/​rbg.go Add scheduling, isolation, disabled-role, and annotation-update scenarios.
test/​e2e/​framework/​workloads/​workload_v2_expect.go Expect group-uid in v1alpha2 affinity rules.
test/​e2e/​framework/​workloads/​workload_expect.go Expect group-uid in legacy API affinity rules.
pkg/​reconciler/​pod_reconciler.go Correct affinity labels and detect selector conflicts.
pkg/​reconciler/​pod_reconciler_test.go Cover label matching, selector preservation, conflicts, and opt-out behavior.
internal/​controller/​workloads/​rolebasedgroup_controller.go Enqueue exclusive-topology annotation changes.
internal/​controller/​workloads/​rolebasedgroup_controller_test.go Test annotation-change filtering.
examples/​basic/​rbgs/​rbgs-exclusive-topology.yaml Correct group-template structure and annotation key.
examples/​basic/​rbg/​scheduling/​exclusive-topology.yaml Correct annotations and clarify co-location behavior.
doc/​reference/​variables.md Clarify current and legacy group identity labels.
doc/​features/​exclusive-topology.md Explain scheduling boundaries, disabled roles, and upgrade effects.
api/​workloads/​constants/​label.go Clarify label comments without changing constants.

Review effort: Balanced

Comment thread pkg/reconciler/pod_reconciler.go Resolved
Comment thread pkg/reconciler/pod_reconciler.go Resolved

@cheyang cheyang left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both reviewers independently reproduced the premise of this PR on the merge-base: the injected required podAffinity selects rbg.workloads.x-k8s.io/group-unique-hash, a label no v1alpha2 Pod carries (they carry the same hash under group-uid), so the first Pod of an exclusive group can never satisfy its own affinity and stays Pending — the bug from #518, introduced by the v1alpha2 storage migration. The fix is correct and verified on both reviewers' harnesses at unit and envtest level (red on base, green at head, and reverting the one-line selector change turns the harnesses red again), and the four new e2e specs ran and passed in the CI lifecycle slice at head. No blocking findings.

The one substantive follow-up, flagged by Copilot and confirmed by both debaters: the upgrade path repairs existing broken workloads only on the RoleInstanceSet path. Deployments, StatefulSets and LeaderWorkerSets keep the old broken affinity after upgrade, because the shared equality check (podSpecEqual) never compares Affinity and their reconcile is skipped as "unchanged". Those workloads were already permanently Pending before this PR, so nothing regresses — but the docs' claim that upgrading replaces Pods of already-configured RBGs should be scoped to RoleInstanceSet (or the equality check taught about affinity), and toggling the annotation or touching the spec forces the repair in the meantime.

The debate's one real disagreement resolved in Codex's favor: the new conflict-guard error is not invisible — a FailedReconcileWorkload Warning event fires per role — so the residual ask there is only to confirm the hard-wedge semantics and consider a status condition. Reviewer C's advisory position (upgrade-repair gap + missing upgrade e2e fixture) was confirmed on both points and is folded into the follow-ups below.

Next round

[MUST] none — no PR-introduced blocker/major findings; merge is not gated on any finding from this round.
[SHOULD] Scope the upgrade-repair docs claim to RoleInstanceSet, or include Affinity in podSpecEqual (+ a regression test from a stored group-unique-hash template).
[SHOULD] Confirm the conflict guard's hard-wedge semantics and consider a status condition for it (currently events-only).
[SHOULD] Uplift the envtest annotation-toggle case from the verification branches so clusterless CI guards the predicate/rewrite loop.
[SHOULD] Release-note/capacity note for the now-active namespace-wide mutual exclusion (it affects RBGs that never opted in).
[Filed] none.

Non-blocking observations (9) — not part of this round's ask; fix now, follow up, or file an issue as you see fit
  • pkg/reconciler/pod_reconciler.go:284 [minor] — Upgrade repair of existing exclusive-topology workloads holds only for RoleInstanceSet; Deployment/STS/LWS keep the broken affinity
    S1 (flagged by Reviewer C as C4573963141, confirmed by both debaters in cross-critique): the docs claim that upgrading replaces Pods of already-configured RBGs is true only for the RoleInstanceSet path, which patches unconditionally. For Deployment, StatefulSet and LeaderWorkerSet, the reconcile is skipped when semanticallyEqual && revisionHashEqual && controllerRef is present, and the shared podSpecEqual (pod_reconciler.go:284-306) compares schedulerName/initContainers/containers/volumes but never Affinity — so an existing exclusive-topology workload created in the #188→pre-588 window, whose only base→head diff is the affinity selector, is judged equal and keeps the broken group-unique-hash affinity after upgrade. Not a regression vs. base (those workloads are already permanently Pending pre-fix), and toggling the annotation or touching the spec forces the repair (the pod annotation is compared by objectMetaEqual and rbg.* keys are not filtered). Ask: scope the docs claim to RoleInstanceSet, or include Affinity in the equality check, plus a reconcile-level regression test starting from a stored group-unique-hash template. Follow-up, does not gate this PR.
  • pkg/reconciler/pod_reconciler.go:125 [minor] — New conflict-guard error wedges the role's reconcile; surfacing is events-only (merged Claude F2 / Codex F2; the 'invisible' sub-claim was rebutted)
    S2: ConstructPodTemplateSpecApplyConfiguration now returns a hard error when a workload's persisted selector pins a group-uid different from the group identity. The error propagates per role: a FailedReconcileWorkload Warning event is emitted on the RBG naming the role and carrying the full conflict message, the role retries with backoff until the foreign selector is fixed, later dependency layers are skipped for that pass, and no status condition carries it (constructAndUpdateRoleStatuses reports replica counts only). This matches the repo's existing convention for role reconcile errors and the PR body explicitly chose to 'report conflicting group-uid values', so it is defensible — please confirm a hard wedge (vs. event + skipping the affinity injection) is the intended semantics, and consider a status condition so kubectl get rbg shows it. Reachability is narrow (claimable deploy/sts workloads with a foreign group-uid in the selector; the RIS path always passes the group's own labels), hence minor. Note on the debate: Claude's phase-1 headline that the failure is invisible to users was rebutted by Codex — the Warning event already fires at rolebasedgroup_controller.go:609-612 — and Claude conceded; the residual ask is the status-condition/semantics question above.
  • pkg/reconciler/pod_reconciler.go:123 [minor] — pre-existing, not introduced by this PR — Empty group-exclusive-topology value hard-fails the reconcile loop (pre-existing)
    S3: GetExclusiveKey returns found=true for an empty annotation value, so 'group-exclusive-topology: ""' reaches setExclusiveAffinities with an empty topology key and errors ('topology key can't be nil'), leaving the RBG in an error/requeue loop. Identical on the merge-base, so not a regression — but the PR's new predicate test explicitly enqueues a reconcile for an empty-value change ({name: 'empty value added', new: {key: ''}, want: true}), knowingly scheduling a reconcile that then hard-fails. Follow-up: treat empty as absent or reject it at admission.
  • pkg/reconciler/pod_reconciler.go:219 [minor] — Anti-affinity is now namespace-wide mutual exclusion for all RBGs — intended, operationally consequential
    S4: The required anti-affinity (Exists group-uid + NotIn , own namespace only, required-during-scheduling) means enabling exclusive topology on one group (a) blocks it from topology domains occupied by any other RBG's pods and (b) blocks other RBGs' new pods — including groups that never opted in — from the exclusive domain (the scheduler checks incoming pods against existing pods' anti-affinity terms). Before this PR the term was inert because its selector matched nothing. This is documented in doc/features/exclusive-topology.md and asserted by the PR's own e2e ('blocks an exclusive group from a node occupied by an ordinary group'), so it is an accepted product decision, not a defect. Flagging the operational hazard: an ordinary RBG that lands first on the last free domain permanently wedges an exclusive RBG (required anti-affinity, no preemption/eviction). Worth a release-note-level mention and a capacity statement (free domains must be at least the number of concurrent exclusive groups plus squatters).
  • internal/controller/workloads/rolebasedgroup_controller.go:1735 [minor] — pre-existing, not introduced by this PR — RBGPredicate stays annotation-selective: other behavioral annotations (e.g. group-gang-scheduling) still do not requeue (pre-existing design question)
    S5: RBGPredicate.UpdateFunc fires on spec changes and now on group-exclusive-topology annotation changes — the right surgical fix for this feature. But other behavior-bearing RBG annotations, e.g. rbg.workloads.x-k8s.io/group-gang-scheduling (read by common.GetGangStrategy during reconcile, called from pod_reconciler.go:157 and roleinstanceset_reconciler.go:251), still do not enqueue when toggled on RBG metadata, leaving a stale PodGroup until an unrelated spec change lands. Pre-existing and the PR is not obligated to fix it, but the one-key comparison now codifies the pattern: is per-key annotation comparison the intended long-term shape, or should the predicate compare metadata.annotations wholesale (skip only when spec AND annotations are both unchanged)? Answering here avoids a slow accretion of one-off keys.
  • test/envtest/testcase/rbg [minor] — No envtest (integration) coverage of the annotation-toggle reconcile loop in the PR itself
    S6: The PR covers template construction with unit tests and the full loop with live e2e, but test/envtest (which runs on every PR via envtest.yml and needs no cluster) has zero exclusive-topology coverage — the predicate -> reconcile -> workload-template rewrite path is only exercised on a real cluster. Both verification branches add exactly this: create with annotation -> RIS template gains the group-uid affinity; remove annotation -> affinity dropped (Claude: test/envtest/testcase/rbg/verify_exclusive_topology_claude_test.go; Codex: an equivalent envtest spec exercising the toggle through the real controller watch). Consider uplifting one of them into the PR or a follow-up so clusterless CI catches predicate/rewrite regressions cheaply.
  • test/e2e/upgrade/fixtures.go:125 [minor] — pre-existing, not introduced by this PR — No exclusive-topology fixture or recorded migration in test/e2e/upgrade (companion to S1)
    S7 (flagged by Reviewer C as C4573963174, factually confirmed): the upgrade e2e suite has no exclusive-topology fixture and no recorded migration for it, so the upgrade behavior this PR changes (RIS affinity rewrite, deprecated-type retention — see S1) is unguarded at the live layer. Follow-up: add a running old-release fixture with the matching group-unique-hash template, record the accepted release-specific rewrite in profile.go, and verify that only the intended templates/revisions change, non-exclusive workloads remain unchanged, and subsequent reconciles cause no further rollout.
  • test/e2e/testcase/v1alpha2/rbg.go:497 [nit] — expectGroupBlockedByAntiAffinity greps scheduler event text
    S8: strings.Contains(event.Message, "anti-affinity") depends on kube-scheduler's message wording ('didn't satisfy existing pods anti-affinity rules' etc.). The helper is new in this PR, and the pod-stays-Pending check plus Reason == FailedScheduling already carry most of the signal — though the message grep does distinguish anti-affinity blocking from resource pressure and the wording has been stable across many releases. A comment noting the dependency (or matching on Reason + Pending alone) would do.
  • api/workloads/v1alpha2/helper.go:179 [nit] — pre-existing, not introduced by this PR — group-uid is a hash of namespace/name, not a UID (pre-existing semantics now load-bearing)
    S9: GenGroupUniqueKey is sha1(namespace/name), so deleting and recreating an RBG with the same name yields the same 'uid'; during overlap the old generation's terminating pods satisfy the new group's affinity and share its domain. Pre-existing label semantics that this PR now leans on for mutual exclusion; the docs update explicitly disclaiming 'It is not the Kubernetes object UID' is the right mitigation. No action requested — recorded for the cross-review.
Verified review delta
  • S1: open (initial-pr)
  • S2: open (initial-pr)
  • S3: open (preexisting)
  • S4: open (initial-pr)
  • S5: open (preexisting)
  • S6: open (initial-pr)
  • S7: open (preexisting)
  • S8: open (initial-pr)
  • S9: open (preexisting)

Code: clear. CI: success; attribution: unknown. Required CI checks still apply.

Signed-off-by: QIW9 <qiwang922337@gmail.com>
@QIW9

QIW9 commented Nov 15, 2026

Copy link
Copy Markdown
Contributor Author

Both reviewers independently reproduced the premise of this PR on the merge-base: the injected required podAffinity selects rbg.workloads.x-k8s.io/group-unique-hash, a label no v1alpha2 Pod carries (they carry the same hash under group-uid), so the first Pod of an exclusive group can never satisfy its own affinity and stays Pending — the bug from #518, introduced by the v1alpha2 storage migration. The fix is correct and verified on both reviewers' harnesses at unit and envtest level (red on base, green at head, and reverting the one-line selector change turns the harnesses red again), and the four new e2e specs ran and passed in the CI lifecycle slice at head. No blocking findings.

The one substantive follow-up, flagged by Copilot and confirmed by both debaters: the upgrade path repairs existing broken workloads only on the RoleInstanceSet path. Deployments, StatefulSets and LeaderWorkerSets keep the old broken affinity after upgrade, because the shared equality check (podSpecEqual) never compares Affinity and their reconcile is skipped as "unchanged". Those workloads were already permanently Pending before this PR, so nothing regresses — but the docs' claim that upgrading replaces Pods of already-configured RBGs should be scoped to RoleInstanceSet (or the equality check taught about affinity), and toggling the annotation or touching the spec forces the repair in the meantime.

The debate's one real disagreement resolved in Codex's favor: the new conflict-guard error is not invisible — a FailedReconcileWorkload Warning event fires per role — so the residual ask there is only to confirm the hard-wedge semantics and consider a status condition. Reviewer C's advisory position (upgrade-repair gap + missing upgrade e2e fixture) was confirmed on both points and is folded into the follow-ups below.

Next round

[MUST] none — no PR-introduced blocker/major findings; merge is not gated on any finding from this round. [SHOULD] Scope the upgrade-repair docs claim to RoleInstanceSet, or include Affinity in podSpecEqual (+ a regression test from a stored group-unique-hash template). [SHOULD] Confirm the conflict guard's hard-wedge semantics and consider a status condition for it (currently events-only). [SHOULD] Uplift the envtest annotation-toggle case from the verification branches so clusterless CI guards the predicate/rewrite loop. [SHOULD] Release-note/capacity note for the now-active namespace-wide mutual exclusion (it affects RBGs that never opted in). [Filed] none.

Non-blocking observations (9) — not part of this round's ask; fix now, follow up, or file an issue as you see fit

  • pkg/reconciler/pod_reconciler.go:284 [minor] — Upgrade repair of existing exclusive-topology workloads holds only for RoleInstanceSet; Deployment/STS/LWS keep the broken affinity
    S1 (flagged by Reviewer C as C4573963141, confirmed by both debaters in cross-critique): the docs claim that upgrading replaces Pods of already-configured RBGs is true only for the RoleInstanceSet path, which patches unconditionally. For Deployment, StatefulSet and LeaderWorkerSet, the reconcile is skipped when semanticallyEqual && revisionHashEqual && controllerRef is present, and the shared podSpecEqual (pod_reconciler.go:284-306) compares schedulerName/initContainers/containers/volumes but never Affinity — so an existing exclusive-topology workload created in the chore(rbg): implement v1alpha2 as storage version #188→pre-588 window, whose only base→head diff is the affinity selector, is judged equal and keeps the broken group-unique-hash affinity after upgrade. Not a regression vs. base (those workloads are already permanently Pending pre-fix), and toggling the annotation or touching the spec forces the repair (the pod annotation is compared by objectMetaEqual and rbg.* keys are not filtered). Ask: scope the docs claim to RoleInstanceSet, or include Affinity in the equality check, plus a reconcile-level regression test starting from a stored group-unique-hash template. Follow-up, does not gate this PR.
  • pkg/reconciler/pod_reconciler.go:125 [minor] — New conflict-guard error wedges the role's reconcile; surfacing is events-only (merged Claude F2 / Codex F2; the 'invisible' sub-claim was rebutted)
    S2: ConstructPodTemplateSpecApplyConfiguration now returns a hard error when a workload's persisted selector pins a group-uid different from the group identity. The error propagates per role: a FailedReconcileWorkload Warning event is emitted on the RBG naming the role and carrying the full conflict message, the role retries with backoff until the foreign selector is fixed, later dependency layers are skipped for that pass, and no status condition carries it (constructAndUpdateRoleStatuses reports replica counts only). This matches the repo's existing convention for role reconcile errors and the PR body explicitly chose to 'report conflicting group-uid values', so it is defensible — please confirm a hard wedge (vs. event + skipping the affinity injection) is the intended semantics, and consider a status condition so kubectl get rbg shows it. Reachability is narrow (claimable deploy/sts workloads with a foreign group-uid in the selector; the RIS path always passes the group's own labels), hence minor. Note on the debate: Claude's phase-1 headline that the failure is invisible to users was rebutted by Codex — the Warning event already fires at rolebasedgroup_controller.go:609-612 — and Claude conceded; the residual ask is the status-condition/semantics question above.
  • pkg/reconciler/pod_reconciler.go:123 [minor] — pre-existing, not introduced by this PR — Empty group-exclusive-topology value hard-fails the reconcile loop (pre-existing)
    S3: GetExclusiveKey returns found=true for an empty annotation value, so 'group-exclusive-topology: ""' reaches setExclusiveAffinities with an empty topology key and errors ('topology key can't be nil'), leaving the RBG in an error/requeue loop. Identical on the merge-base, so not a regression — but the PR's new predicate test explicitly enqueues a reconcile for an empty-value change ({name: 'empty value added', new: {key: ''}, want: true}), knowingly scheduling a reconcile that then hard-fails. Follow-up: treat empty as absent or reject it at admission.
  • pkg/reconciler/pod_reconciler.go:219 [minor] — Anti-affinity is now namespace-wide mutual exclusion for all RBGs — intended, operationally consequential
    S4: The required anti-affinity (Exists group-uid + NotIn , own namespace only, required-during-scheduling) means enabling exclusive topology on one group (a) blocks it from topology domains occupied by any other RBG's pods and (b) blocks other RBGs' new pods — including groups that never opted in — from the exclusive domain (the scheduler checks incoming pods against existing pods' anti-affinity terms). Before this PR the term was inert because its selector matched nothing. This is documented in doc/features/exclusive-topology.md and asserted by the PR's own e2e ('blocks an exclusive group from a node occupied by an ordinary group'), so it is an accepted product decision, not a defect. Flagging the operational hazard: an ordinary RBG that lands first on the last free domain permanently wedges an exclusive RBG (required anti-affinity, no preemption/eviction). Worth a release-note-level mention and a capacity statement (free domains must be at least the number of concurrent exclusive groups plus squatters).
  • internal/controller/workloads/rolebasedgroup_controller.go:1735 [minor] — pre-existing, not introduced by this PR — RBGPredicate stays annotation-selective: other behavioral annotations (e.g. group-gang-scheduling) still do not requeue (pre-existing design question)
    S5: RBGPredicate.UpdateFunc fires on spec changes and now on group-exclusive-topology annotation changes — the right surgical fix for this feature. But other behavior-bearing RBG annotations, e.g. rbg.workloads.x-k8s.io/group-gang-scheduling (read by common.GetGangStrategy during reconcile, called from pod_reconciler.go:157 and roleinstanceset_reconciler.go:251), still do not enqueue when toggled on RBG metadata, leaving a stale PodGroup until an unrelated spec change lands. Pre-existing and the PR is not obligated to fix it, but the one-key comparison now codifies the pattern: is per-key annotation comparison the intended long-term shape, or should the predicate compare metadata.annotations wholesale (skip only when spec AND annotations are both unchanged)? Answering here avoids a slow accretion of one-off keys.
  • test/envtest/testcase/rbg [minor] — No envtest (integration) coverage of the annotation-toggle reconcile loop in the PR itself
    S6: The PR covers template construction with unit tests and the full loop with live e2e, but test/envtest (which runs on every PR via envtest.yml and needs no cluster) has zero exclusive-topology coverage — the predicate -> reconcile -> workload-template rewrite path is only exercised on a real cluster. Both verification branches add exactly this: create with annotation -> RIS template gains the group-uid affinity; remove annotation -> affinity dropped (Claude: test/envtest/testcase/rbg/verify_exclusive_topology_claude_test.go; Codex: an equivalent envtest spec exercising the toggle through the real controller watch). Consider uplifting one of them into the PR or a follow-up so clusterless CI catches predicate/rewrite regressions cheaply.
  • test/e2e/upgrade/fixtures.go:125 [minor] — pre-existing, not introduced by this PR — No exclusive-topology fixture or recorded migration in test/e2e/upgrade (companion to S1)
    S7 (flagged by Reviewer C as C4573963174, factually confirmed): the upgrade e2e suite has no exclusive-topology fixture and no recorded migration for it, so the upgrade behavior this PR changes (RIS affinity rewrite, deprecated-type retention — see S1) is unguarded at the live layer. Follow-up: add a running old-release fixture with the matching group-unique-hash template, record the accepted release-specific rewrite in profile.go, and verify that only the intended templates/revisions change, non-exclusive workloads remain unchanged, and subsequent reconciles cause no further rollout.
  • test/e2e/testcase/v1alpha2/rbg.go:497 [nit] — expectGroupBlockedByAntiAffinity greps scheduler event text
    S8: strings.Contains(event.Message, "anti-affinity") depends on kube-scheduler's message wording ('didn't satisfy existing pods anti-affinity rules' etc.). The helper is new in this PR, and the pod-stays-Pending check plus Reason == FailedScheduling already carry most of the signal — though the message grep does distinguish anti-affinity blocking from resource pressure and the wording has been stable across many releases. A comment noting the dependency (or matching on Reason + Pending alone) would do.
  • api/workloads/v1alpha2/helper.go:179 [nit] — pre-existing, not introduced by this PR — group-uid is a hash of namespace/name, not a UID (pre-existing semantics now load-bearing)
    S9: GenGroupUniqueKey is sha1(namespace/name), so deleting and recreating an RBG with the same name yields the same 'uid'; during overlap the old generation's terminating pods satisfy the new group's affinity and share its domain. Pre-existing label semantics that this PR now leans on for mutual exclusion; the docs update explicitly disclaiming 'It is not the Kubernetes object UID' is the right mitigation. No action requested — recorded for the cross-review.

Verified review delta

@cheyang Addressed in 29fa5fa. Deployment, StatefulSet, and LeaderWorkerSet now reconcile affinity-only changes for exclusive topology. And I added tests for annotation changes, migration from the legacy selector, and rollout stability after a controller restart.All the ci tests pass.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The migration can disrupt running workloads and activate namespace-wide exclusion, requiring maintainer acceptance of the operational impact and validation of the new release-upgrade scenarios.

1 open issue
1 resolved since last review

Review effort: Balanced

cheyang pushed a commit to cheyang/rbg that referenced this pull request Nov 15, 2026
Additive verification harness only; production code untouched.

- L1 unit contract/canary tests for the exclusive-topology affinity fix
- L2 envtest integration test incl. the annotation-toggle reconcile loop
- premise (P0) reproduced against the merge-base at both layers
- harness-bites check recorded (un-fixing goes red at L1+L2)
- docs/verification/exclusive-topology-claude: README, manifest, re-verify.sh

Co-Authored-By: Claude Code <noreply@anthropic.com>
cheyang pushed a commit to cheyang/rbg that referenced this pull request Nov 15, 2026
…viewer A)

Re-verified all round-1 claims against the new head 29fa5fa and added the
S1 upgrade-repair claims: legacy group-unique-hash affinity on a stored
Deployment/StatefulSet is migrated to group-uid by the new affinity guard in
podTemplateSpecEqual (RED on previous head 3ae9d70, GREEN on 29fa5fa),
idempotently; annotation removal still drops the affinity. re-verify.sh now
fails on a failed graft and on no-tests-matched false greens. Production
code untouched.

Co-Authored-By: Claude Code <noreply@anthropic.com>

@cheyang cheyang left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Round 2 at head 29fa5faa — one commit, "migrate exclusive topology affinity on upgrade" — and it closes essentially the whole round-1 follow-up list. The fixes check out.

The headline: podTemplateSpecEqual now compares Pod-template affinity when the desired template carries group-exclusive-topology, so a Deployment/StatefulSet/LeaderWorkerSet stored by the pre-fix controller with the broken group-unique-hash affinity is no longer judged "unchanged" — the reconcile rewrites it to group-uid. Both of us verified this independently with different harnesses: tests driving the real reconcilers are red on the previous head (the skip path keeps the legacy affinity, exactly the round-1 finding) and green on this head, idempotent afterwards, with the gate correctly scoped and all four call sites passing the desired template on the right side of the comparison. The PR's own Deployment repair test agrees, and the docs' upgrade claim is now accurate for all four workload types.

Also landed: the PR's own envtest toggle spec (executed green by both of us), the upgrade e2e fixture with recorded migrations for v0.7.0/v0.8.0 (premises verified against the tags; not yet executed — below), documented conflict-guard semantics and capacity statement, and the scheduler-message comment. Round-1 regression suites fully green on both harnesses at the new head; all CI checks green, with the four lifecycle e2e specs demonstrably executed and passing.

One new follow-up worth acting on before merge: the new upgrade migration specs live in test/e2e/upgrade, which required CI never runs — that suite only executes via a maintainer /release-test comment. Nothing on this head has run them. One /release-test on this PR executes them before the release is the first place they ever do.

No disagreements this round — the cross-critiques confirmed every disposition 10-for-10, and Copilot raised no findings at this head. Remaining open items are pre-existing and minor: the empty annotation value hard-fail (S3) and the annotation-selective predicate design question (S5).

Next round

[MUST] none — no PR-introduced blocker/major findings; merge is not gated on any finding from this round.
[SHOULD] Trigger /release-test on head 29fa5faa so the new upgrade e2e specs execute once before merge (S10).
[SHOULD] Follow-up, not this PR: treat an empty group-exclusive-topology value as absent or reject it at admission (S3, pre-existing).
[SHOULD] Maintainer call on the predicate design question: per-key annotation comparison vs. wholesale (S5, pre-existing).
[Filed] none.

Non-blocking observations (1) — not part of this round's ask; fix now, follow up, or file an issue as you see fit
  • test/e2e/upgrade/exclusive_topology.go:26 [minor] — New upgrade e2e specs are compile-clean but have never executed: the upgrade suite runs only via the comment-triggered Release Test workflow (NEW in this round, mutual discovery by both reviewers)
    S10 (found independently by both reviewers in this round's delta review): test/e2e/upgrade is a separate test binary invoked only by 'make test-e2e-upgrade', which appears exclusively in .github/workflows/release-test.yml (line 535) — a workflow triggered by an issue comment containing '/release-test' from an OWNER/MEMBER/COLLABORATOR. The per-PR e2e matrix runs 'make test-e2e' = 'go test ./test/e2e/' (the e2e package only, not the upgrade subpackage) with label filters, so the two new exclusive-topology migration specs are compiled and vet-checked at 29fa5fa but have never executed: no 'Release Test / e2e-test-upgrade' status exists on the head (gh pr checks 588 lists 22 green checks, none of them the release test). The new assertions are strict (exact affinity equality, empty storedSpecDiff, pod-UID and revision turnover) and have never run against the real v0.7.0/v0.8.0 hops, so the first execution could still surface a legitimately-recorded rewrite or a real defect. This follows the repo's established convention for the whole upgrade suite (pre-existing specs run the same way) and the migration mechanism is independently proven at the unit layer (S1 differential on both harnesses plus the PR's own Deployment test) — hence minor, not a coverage blocker. Ask: trigger /release-test once on this head before merging so the first execution of the new specs is not the release itself, or explicitly accept release-time shakedown. A late discovery about new code in the delta, not a regression in it (no previous-head-pass/current-head-fail differential exists — the specs did not exist before).
Verified review delta
  • S1: fixed (initial-pr)
  • S2: fixed (initial-pr)
  • S4: fixed (initial-pr)
  • S6: fixed (initial-pr)
  • S7: fixed (initial-pr)
  • S10: open (initial-pr)
  • S8: fixed (initial-pr)

Code: clear. CI: success; attribution: unknown. Required CI checks still apply.

@cheyang cheyang left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review at head 29fa5faa (base main 1fc69f70).

Verdict: approve once /release-test is green. The fix is correct and restores the original design from #32, where every Pod carried the key that the exclusive affinity selects. #184 renamed only the label side. I found no blocking issues.

# Severity Location Problem → suggested fix Evidence
1 🟡 SHOULD test/e2e/upgrade/exclusive_topology.go The new upgrade specs have not executed yet, because the upgrade suite only runs through /release-test. Run it on this head before merge. This agrees with S10 from the previous round. CI check list on 29fa5fa
2 🟡 SHOULD doc/features/exclusive-topology.md ("Existing workloads") and the release note A group that is Running because it used the group-unique-hash label workaround rolls on upgrade. Its replacement Pods are anti-affine to every group-uid Pod in the namespace, whereas the workaround excluded only Pods that carried the label. If ordinary RBG Pods occupy the old domain, the replacement can stay Pending, which takes down a group that was Running before the upgrade. The upgrade fixture is isolated in its own namespace, so this case is not exercised. Suggest saying this explicitly in the release note: before upgrading, make sure each exclusive group has a free topology domain. Otherwise, a maintainer should explicitly accept the risk. code reading only
3 ⚪ nit pkg/reconciler/pod_reconciler.go:127-131 Controller-generated selectors can't trigger the conflict guard. Every selector comes from GetCommonLabelsFromRole, so it either carries the same hash or predates #184 and has no group-uid. Keeping the guard as a defensive check is fine; a one-line comment saying so would help. code reading and git history
4 ⚪ nit test/e2e/upgrade/profile.go exclusiveTopologyFromLabel bypasses recordedRewrites. That is acceptable because the fixture is namespace-isolated; worth stating in the field comment. code reading

Verified

  • go test ./pkg/reconciler -run 'Exclusive|PodTemplate|Deploy' -count=1 passes on this head.
  • The affinity check added to podTemplateSpecEqual keys off the desired template, and all four call sites pass it as template2.
  • extraLabels and the affinity are set only on the exclusive path, so templates of RBGs without the annotation are unchanged.
  • The RBGPredicate change is scoped to the single annotation.

Not run: envtest, e2e, upgrade e2e (requesting /release-test below).

@cheyang

cheyang commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator

/release-test

@cheyang

cheyang commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator

Correction to my review: the /release-test run on 29fa5faa (run 43791276444) failed as expected. The failure does not reflect a defect in this PR.

Why it failed. /release-test checks out the PR's test code, but it never builds the PR's controller. The helm and manifest jobs deploy the chart's default images, and the upgrade jobs upgrade to whatever the chart's default values point at (.github/workflows/release-test.yml). The chart pins rolebasedgroup/rbgs-controller:v0.9.0-73e05db4 (deploy/helm/rbgs/values.yaml:13), which was built from #524 on main, before this fix. The new assertions therefore ran against a controller that still renders group-unique-hash:

What this changes in my review. Finding 1 ("run /release-test before merge") was wrong, and so was the same assumption in the previous round's S10. A PR that changes controller behaviour can't pass /release-test until its image is published and pinned in the chart, so the run can't gate this PR.

How to validate before merge. Build the controller image from this head, load it into Kind, and run the upgrade suite against it:

make docker-build-controller TAG=e2e-upgrade
make docker-build-crd-upgrader TAG=e2e-upgrade
# on kind: kind load docker-image both images, plus the from-release images
RBGS_TO_TAG=e2e-upgrade make test-e2e-upgrade

These are the steps documented in test/e2e/upgrade/install.go. Run them on a dedicated cluster, not a shared one. The helm and manifest lifecycle specs already pass on this head in the PR's own e2e CI, which builds the PR image. Alternatively, a maintainer can accept that the upgrade specs first run in /release-test after the post-merge image bump. This failure is expected and should not block merge on its own.

Signed-off-by: QIW9 <qiwang922337@gmail.com>
@QIW9

QIW9 commented Nov 16, 2026

Copy link
Copy Markdown
Contributor Author

Independent review at head 29fa5faa (base main 1fc69f70).

Verdict: approve once /release-test is green. The fix is correct and restores the original design from #32, where every Pod carried the key that the exclusive affinity selects. #184 renamed only the label side. I found no blocking issues.

Severity Location Problem → suggested fix Evidence

1 🟡 SHOULD test/e2e/upgrade/exclusive_topology.go The new upgrade specs have not executed yet, because the upgrade suite only runs through /release-test. Run it on this head before merge. This agrees with S10 from the previous round. CI check list on 29fa5fa
2 🟡 SHOULD doc/features/exclusive-topology.md ("Existing workloads") and the release note A group that is Running because it used the group-unique-hash label workaround rolls on upgrade. Its replacement Pods are anti-affine to every group-uid Pod in the namespace, whereas the workaround excluded only Pods that carried the label. If ordinary RBG Pods occupy the old domain, the replacement can stay Pending, which takes down a group that was Running before the upgrade. The upgrade fixture is isolated in its own namespace, so this case is not exercised. Suggest saying this explicitly in the release note: before upgrading, make sure each exclusive group has a free topology domain. Otherwise, a maintainer should explicitly accept the risk. code reading only
3 ⚪ nit pkg/reconciler/pod_reconciler.go:127-131 Controller-generated selectors can't trigger the conflict guard. Every selector comes from GetCommonLabelsFromRole, so it either carries the same hash or predates #184 and has no group-uid. Keeping the guard as a defensive check is fine; a one-line comment saying so would help. code reading and git history
4 ⚪ nit test/e2e/upgrade/profile.go exclusiveTopologyFromLabel bypasses recordedRewrites. That is acceptable because the fixture is namespace-isolated; worth stating in the field comment. code reading
Verified

  • go test ./pkg/reconciler -run 'Exclusive|PodTemplate|Deploy' -count=1 passes on this head.
  • The affinity check added to podTemplateSpecEqual keys off the desired template, and all four call sites pass it as template2.
  • extraLabels and the affinity are set only on the exclusive path, so templates of RBGs without the annotation are unchanged.
  • The RBGPredicate change is scoped to the single annotation.

Not run: envtest, e2e, upgrade e2e (requesting /release-test below).

The scheduling risks of existing manually labeled pods are documented in the doc.

@github-actions

Copy link
Copy Markdown

/release-test is restricted to repository collaborators and org members. Ask a maintainer to run it for you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] group-exclusive-topology leaves Pods Pending: required podAffinity selects group-unique-hash, which Pods never carry

3 participants