Skip to content
Open
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 3 additions & 4 deletions api/workloads/constants/label.go
Original file line number Diff line number Diff line change
Expand Up @@ -32,9 +32,8 @@ const (
// GroupNameLabelKey identifies resources belonging to a specific RoleBasedGroup
GroupNameLabelKey = RBGPrefix + "group-name"

// GroupUIDLabelKey is set on every Pod and carries a short hash
// that identifies all Pods belonging to the same RoleBasedGroup instance.
// Used as the match label for topology affinity.
// GroupUIDLabelKey identifies an RBG by a hash of its namespace and name.
// It is used as the match label for topology affinity.
GroupUIDLabelKey = RBGPrefix + "group-uid"

// RBGOwnerUIDLabelKey carries the real Kubernetes UID of the owning
Expand All @@ -48,7 +47,7 @@ const (
// RoleBasedGroup Roles's template.
GroupRevisionLabelKey = RBGPrefix + "group-revision"

// GroupUniqueHashLabelKey is used for pod affinity rules in exclusive topology
// GroupUniqueHashLabelKey is the legacy exclusive-topology selector key.
GroupUniqueHashLabelKey = RBGPrefix + "group-unique-hash"
)

Expand Down
38 changes: 34 additions & 4 deletions doc/features/exclusive-topology.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,21 @@
# Exclusive Topology

Exclusive topology is a feature that allows you to specify a topology domain for a RoleBasedGroup or RoleBasedGroupSet, ensuring that all pods belonging to the same group are scheduled on the same topology domain (e.g., the same node, rack, or zone). This is particularly useful for distributed applications that require low-latency communication between roles or need to ensure data locality.
Exclusive topology lets a RoleBasedGroup or RoleBasedGroupSet constrain participating Pods to a topology domain, such as a node, rack, or zone. This is useful for distributed applications that require low-latency communication between roles or need to ensure data locality.

## How It Works

The exclusive topology feature uses Kubernetes topology keys to define scheduling constraints. When you specify an exclusive topology key, all pods within the same RoleBasedGroup or RoleBasedGroupSet will be scheduled on nodes that share the same value for that topology key. This ensures co-location of related workloads for better performance and reduced network latency.
The controller uses the Pod label `rbg.workloads.x-k8s.io/group-uid` to identify a group. For roles
with exclusive topology enabled, it adds required Pod affinity for the same group and required Pod
anti-affinity for other groups with that label. In the same namespace, this prevents another
RoleBasedGroup from sharing the topology domain, even if that group does not enable exclusive topology.
Plan for one eligible topology domain per concurrent exclusive group, in addition to domains
already occupied by other groups in the namespace. An ordinary group that schedules first can
leave an exclusive group Pending if it occupies the last eligible domain.

The rules do not exclude Pods without `group-uid` or Pods in other namespaces. They apply when Pods
are scheduled and do not evict already running Pods if conditions later change. A topology domain can
be a node, zone, or other domain defined by a node label; some clusters restrict required Pod
anti-affinity to `kubernetes.io/hostname`.

## Annotation

Expand All @@ -28,6 +39,10 @@ To skip exclusive-topology affinity injection for a specific role, add this anno
rbg.workloads.x-k8s.io/role-disable-exclusive: "true"
```

That role keeps its group label, so its Pods can still be affinity targets for enabled roles in the
same group. If same-group Pods already occupy multiple topology domains, enabled roles may follow
any of those domains.

## Example: RoleBasedGroup with Exclusive Topology

The [rbg-with-exclusive-topology example](../../examples/basic/rbg/scheduling/exclusive-topology.yaml) demonstrates how to use exclusive topology with a RoleBasedGroup, ensuring all roles are scheduled on the same node.
Expand All @@ -40,10 +55,25 @@ RoleBasedGroupSet extends the exclusive topology feature to multiple groups, whe

In this example:

- 2 RoleBasedGroups will be created (as specified by `replicas: 2`)
- 3 RoleBasedGroups will be created (as specified by `replicas: 3`)
- Each RoleBasedGroup will have its pods scheduled on a separate node
- All pods within each individual RoleBasedGroup will be co-located on the same node

## Existing workloads

Enabling or changing exclusive topology on a running RBG changes its Pod templates and may replace
Pods. The controller upgrade replaces the generated `group-unique-hash` affinity selector with
`group-uid` on already-configured RoleInstanceSet, Deployment, StatefulSet, and LeaderWorkerSet
workloads. This may replace their Pods, including running Pods whose templates carry the old
label as a workaround. The upgrade activates the namespace-scoped mutual exclusion for newly
scheduled Pods, including Pods of groups without the annotation.

Existing workload selectors are preserved. If an existing selector pins a conflicting `group-uid`,
reconciliation for that role stops before applying the workload and the controller reports the
conflict in a `FailedReconcileWorkload` Warning event on the RoleBasedGroup. Disable exclusive
topology for the affected role to resume reconciliation with that selector. When both required Pod
affinity and anti-affinity already use the topology key, the controller uses those rules as written.

## Use Cases

1. **Disaggregated Inference**: Deploy Prefill and Decode on the same node to reduce inter-PD communication overhead
Expand All @@ -55,4 +85,4 @@ In this example:
- **Reduced Network Latency**: Co-located pods can communicate with minimal network overhead
- **Improved Performance**: Related workloads benefit from data locality
- **Resource Efficiency**: Better utilization of node resources by grouping related workloads
- **Simplified Management**: Logical grouping of distributed applications
- **Simplified Management**: Logical grouping of distributed applications
4 changes: 2 additions & 2 deletions doc/reference/variables.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,9 @@
| Key | Description |
|-----|-------------|
| `rbg.workloads.x-k8s.io/group-name` | The name of the RoleBasedGroup to which these resources belong. |
| `rbg.workloads.x-k8s.io/group-uid` | A short hash identifying all Pods belonging to the same RoleBasedGroup instance. Used for topology affinity. |
| `rbg.workloads.x-k8s.io/group-uid` | A hash of the RoleBasedGroup's namespace and name, used for exclusive-topology affinity. It is not the Kubernetes object UID. |
| `rbg.workloads.x-k8s.io/group-revision` | The revision hash of the RoleBasedGroup, used to determine whether the RBG object has changed. |
| `rbg.workloads.x-k8s.io/group-unique-hash` | Used for pod affinity rules in exclusive topology. |
| `rbg.workloads.x-k8s.io/group-unique-hash` | Legacy exclusive-topology selector key. The constant and user-supplied labels are retained; new rules use `group-uid`. |

### Role Level Labels

Expand Down
11 changes: 5 additions & 6 deletions examples/basic/rbg/scheduling/exclusive-topology.yaml
Original file line number Diff line number Diff line change
@@ -1,23 +1,22 @@
# Example: RoleBasedGroup with exclusive topology (v1alpha2)
# Pods of different roles are scheduled on different nodes (or topology zones)
# by setting the exclusive-topology annotation.
# Pods of the same group are scheduled in the same topology domain.
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
name: nginx-exclusive
namespace: default
annotations:
rolebasedgroup.workloads.x-k8s.io/exclusive-topology: "kubernetes.io/hostname"
rbg.workloads.x-k8s.io/group-exclusive-topology: "kubernetes.io/hostname"
spec:
roles:
- name: frontend
replicas: 1
standalonePattern:
template:
# To disable exclusive topology for this specific role, add:
metadata:
annotations:
rolebasedgroup.workloads.x-k8s.io/disable-exclusive-topology: "true"
# metadata:
# annotations:
# rbg.workloads.x-k8s.io/role-disable-exclusive: "true"
spec:
containers:
- name: nginx-frontend
Expand Down
9 changes: 4 additions & 5 deletions examples/basic/rbgs/rbgs-exclusive-topology.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,10 @@ metadata:
spec:
replicas: 3
groupTemplate:
metadata:
labels:
app: inference-cluster
annotations:
rolebasedgroup.workloads.x-k8s.io/group-exclusive-topology: "kubernetes.io/hostname"
labels:
app: inference-cluster
annotations:
rbg.workloads.x-k8s.io/group-exclusive-topology: "kubernetes.io/hostname"
spec:
roles:
- name: prefill
Expand Down
5 changes: 4 additions & 1 deletion internal/controller/workloads/rolebasedgroup_controller.go
Original file line number Diff line number Diff line change
Expand Up @@ -1732,7 +1732,10 @@ func RBGPredicate() predicate.Funcs {
oldRbg, ok1 := e.ObjectOld.(*workloadsv1alpha2.RoleBasedGroup)
newRbg, ok2 := e.ObjectNew.(*workloadsv1alpha2.RoleBasedGroup)
if ok1 && ok2 {
if !reflect.DeepEqual(oldRbg.Spec, newRbg.Spec) {
oldTopology, oldHasTopology := oldRbg.Annotations[constants.GroupExclusiveTopologyKey]
newTopology, newHasTopology := newRbg.Annotations[constants.GroupExclusiveTopologyKey]
if !reflect.DeepEqual(oldRbg.Spec, newRbg.Spec) ||
oldHasTopology != newHasTopology || oldTopology != newTopology {
ctrl.Log.Info("enqueue: rbg update event", "rbg", klog.KObj(e.ObjectOld))
return true
}
Expand Down
25 changes: 25 additions & 0 deletions internal/controller/workloads/rolebasedgroup_controller_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ import (
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/client/fake"
"sigs.k8s.io/controller-runtime/pkg/client/interceptor"
"sigs.k8s.io/controller-runtime/pkg/event"
"sigs.k8s.io/controller-runtime/pkg/log/zap"
"sigs.k8s.io/controller-runtime/pkg/reconcile"
"sigs.k8s.io/rbgs/pkg/reconciler"
Expand All @@ -65,6 +66,30 @@ import (
wrappersv2 "sigs.k8s.io/rbgs/test/wrappers/v1alpha2"
)

func TestRBGPredicateExclusiveTopologyAnnotation(t *testing.T) {
key := constants.GroupExclusiveTopologyKey
for _, tt := range []struct {
name string
old map[string]string
new map[string]string
want bool
}{
{name: "added", new: map[string]string{key: "kubernetes.io/hostname"}, want: true},
{name: "removed", old: map[string]string{key: "kubernetes.io/hostname"}, want: true},
{name: "changed", old: map[string]string{key: "old"}, new: map[string]string{key: "new"}, want: true},
{name: "empty value added", new: map[string]string{key: ""}, want: true},
{name: "unchanged", old: map[string]string{key: "node"}, new: map[string]string{key: "node"}},
{name: "unrelated annotation", new: map[string]string{"example.com/key": "value"}},
} {
t.Run(tt.name, func(t *testing.T) {
old := &workloadsv1alpha2.RoleBasedGroup{ObjectMeta: metav1.ObjectMeta{Annotations: tt.old}}
new := &workloadsv1alpha2.RoleBasedGroup{ObjectMeta: metav1.ObjectMeta{Annotations: tt.new}}
got := RBGPredicate().Update(event.UpdateEvent{ObjectOld: old, ObjectNew: new})
assert.Equal(t, tt.want, got)
})
}
}

func Test_CalculateNextRollingTarget_WithNormalCases(t *testing.T) {
type testCase struct {
name string
Expand Down
40 changes: 40 additions & 0 deletions pkg/reconciler/deploy_reconciler_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ import (
"testing"

"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
appsv1 "k8s.io/api/apps/v1"
corev1 "k8s.io/api/core/v1"
apierrors "k8s.io/apimachinery/pkg/api/errors"
Expand All @@ -34,6 +35,7 @@ import (
"sigs.k8s.io/controller-runtime/pkg/client/interceptor"
"sigs.k8s.io/rbgs/api/workloads/constants"
workloadsv1alpha2 "sigs.k8s.io/rbgs/api/workloads/v1alpha2"
wrappersv2 "sigs.k8s.io/rbgs/test/wrappers/v1alpha2"
)

var expectedRevisionHash = "revision-hash-value"
Expand Down Expand Up @@ -194,6 +196,44 @@ func TestDeploymentReconciler_Reconciler(t *testing.T) {
}
}

func TestDeploymentReconciler_RepairsLegacyExclusiveTopology(t *testing.T) {
ctx := context.Background()
scheme := runtime.NewScheme()
require.NoError(t, appsv1.AddToScheme(scheme))
require.NoError(t, workloadsv1alpha2.AddToScheme(scheme))
fakeClient := fake.NewClientBuilder().WithScheme(scheme).Build()
reconciler := NewDeploymentReconciler(scheme, fakeClient)
rbg := wrappersv2.BuildBasicRoleBasedGroup("exclusive", "default").
WithAnnotations(map[string]string{constants.GroupExclusiveTopologyKey: corev1.LabelHostname}).Obj()
role := &rbg.Spec.Roles[0]
role.Annotations = map[string]string{constants.RoleWorkloadTypeAnnotationKey: "apps/v1/Deployment"}
key := types.NamespacedName{Namespace: rbg.Namespace, Name: rbg.GetWorkloadName(role)}
require.NoError(t, reconciler.Reconciler(ctx, rbg, role, nil, expectedRevisionHash))
deploy := &appsv1.Deployment{}
require.NoError(t, fakeClient.Get(ctx, key, deploy))

// Keep the stored owner, selector and role hash. Only affinity is stale.
deploy.UID = "stored-deployment-uid"
deploy.Spec.Template.Spec.Affinity = nil
require.NoError(t, setExclusiveAffinities(&deploy.Spec.Template, rbg.GenGroupUniqueKey(),
corev1.LabelHostname, constants.GroupUniqueHashLabelKey))
require.NoError(t, fakeClient.Update(ctx, deploy))

var wantTemplate corev1.PodTemplateSpec
require.NoError(t, setExclusiveAffinities(&wantTemplate, rbg.GenGroupUniqueKey(),
corev1.LabelHostname, constants.GroupUIDLabelKey))
require.NoError(t, reconciler.Reconciler(ctx, rbg, role, nil, expectedRevisionHash))
require.NoError(t, fakeClient.Get(ctx, key, deploy))
require.Equal(t, wantTemplate.Spec.Affinity, deploy.Spec.Template.Spec.Affinity)
require.Equal(t, rbg.GetCommonLabelsFromRole(role), deploy.Spec.Selector.MatchLabels)
require.Equal(t, expectedRevisionHash, deploy.Labels[fmt.Sprintf(constants.RoleRevisionLabelKeyFmt, role.Name)])
resourceVersion := deploy.ResourceVersion

require.NoError(t, reconciler.Reconciler(ctx, rbg, role, nil, expectedRevisionHash))
require.NoError(t, fakeClient.Get(ctx, key, deploy))
require.Equal(t, resourceVersion, deploy.ResourceVersion)
}

// TestDeploymentReconciler_CheckWorkloadReady tests the CheckWorkloadReady method of DeploymentReconciler
func TestDeploymentReconciler_CheckWorkloadReady(t *testing.T) {
scheme := runtime.NewScheme()
Expand Down
22 changes: 19 additions & 3 deletions pkg/reconciler/pod_reconciler.go
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ import (
"sort"

corev1 "k8s.io/api/core/v1"
apiequality "k8s.io/apimachinery/pkg/api/equality"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/util/strategicpatch"
Expand Down Expand Up @@ -119,12 +120,21 @@ func (r *PodReconciler) ConstructPodTemplateSpecApplyConfiguration(
}

// Set Exclusive topology
var extraLabels map[string]string
if topologyKey, found := rbg.GetExclusiveKey(); found {
if podAnnotations[constants.DisableExclusiveKeyAnnotationKey] == "" {
podAnnotations[constants.GroupExclusiveTopologyKey] = topologyKey
uniqueKey := rbg.GenGroupUniqueKey()
value, hasGroupUID := podLabels[constants.GroupUIDLabelKey]
if hasGroupUID && value != uniqueKey {
return nil, fmt.Errorf("exclusive topology for role %q: workload selector %s=%q conflicts with group identity %q",
role.Name, constants.GroupUIDLabelKey, value, uniqueKey)
}
if !hasGroupUID {
extraLabels = map[string]string{constants.GroupUIDLabelKey: uniqueKey}
}
podAnnotations[constants.GroupExclusiveTopologyKey] = topologyKey
err := setExclusiveAffinities(
&podTemplateSpec, uniqueKey, topologyKey, constants.GroupUniqueHashLabelKey,
&podTemplateSpec, uniqueKey, topologyKey, constants.GroupUIDLabelKey,
Comment thread
Copilot marked this conversation as resolved.
Resolved
Comment thread
cheyang marked this conversation as resolved.
Resolved
)
if err != nil {
return nil, err
Expand Down Expand Up @@ -152,7 +162,7 @@ func (r *PodReconciler) ConstructPodTemplateSpecApplyConfiguration(
r.gangScheduler.InjectPodSchedulingFields(rbg, role, gangStrategy, podTemplateApplyConfiguration)
}

podTemplateApplyConfiguration.WithLabels(podLabels).WithAnnotations(podAnnotations)
podTemplateApplyConfiguration.WithLabels(podLabels).WithLabels(extraLabels).WithAnnotations(podAnnotations)
return podTemplateApplyConfiguration, nil
}

Expand All @@ -161,6 +171,12 @@ func podTemplateSpecEqual(template1, template2 corev1.PodTemplateSpec) (bool, er
return false, fmt.Errorf("objectMeta not equal: %s", err.Error())
}

// The generated affinity can change without changing the role revision.
if template2.Annotations[constants.GroupExclusiveTopologyKey] != "" &&
!apiequality.Semantic.DeepEqual(template1.Spec.Affinity, template2.Spec.Affinity) {
return false, fmt.Errorf("podTemplate affinity not equal")
}

if equal, err := podSpecEqual(template1.Spec, template2.Spec); !equal {
return false, fmt.Errorf("spec not equal: %s", err.Error())
}
Expand Down
Loading
Loading