Skip to content
Open
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 3 additions & 4 deletions api/workloads/constants/label.go
Original file line number Diff line number Diff line change
Expand Up @@ -32,9 +32,8 @@ const (
// GroupNameLabelKey identifies resources belonging to a specific RoleBasedGroup
GroupNameLabelKey = RBGPrefix + "group-name"

// GroupUIDLabelKey is set on every Pod and carries a short hash
// that identifies all Pods belonging to the same RoleBasedGroup instance.
// Used as the match label for topology affinity.
// GroupUIDLabelKey identifies an RBG by a hash of its namespace and name.
// It is used as the match label for topology affinity.
GroupUIDLabelKey = RBGPrefix + "group-uid"

// RBGOwnerUIDLabelKey carries the real Kubernetes UID of the owning
Expand All @@ -48,7 +47,7 @@ const (
// RoleBasedGroup Roles's template.
GroupRevisionLabelKey = RBGPrefix + "group-revision"

// GroupUniqueHashLabelKey is used for pod affinity rules in exclusive topology
// GroupUniqueHashLabelKey is the legacy exclusive-topology selector key.
GroupUniqueHashLabelKey = RBGPrefix + "group-unique-hash"
)

Expand Down
31 changes: 27 additions & 4 deletions doc/features/exclusive-topology.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,19 @@
# Exclusive Topology

Exclusive topology is a feature that allows you to specify a topology domain for a RoleBasedGroup or RoleBasedGroupSet, ensuring that all pods belonging to the same group are scheduled on the same topology domain (e.g., the same node, rack, or zone). This is particularly useful for distributed applications that require low-latency communication between roles or need to ensure data locality.
Exclusive topology lets a RoleBasedGroup or RoleBasedGroupSet constrain participating Pods to a topology domain, such as a node, rack, or zone. This is useful for distributed applications that require low-latency communication between roles or need to ensure data locality.

## How It Works

The exclusive topology feature uses Kubernetes topology keys to define scheduling constraints. When you specify an exclusive topology key, all pods within the same RoleBasedGroup or RoleBasedGroupSet will be scheduled on nodes that share the same value for that topology key. This ensures co-location of related workloads for better performance and reduced network latency.
The controller uses the Pod label `rbg.workloads.x-k8s.io/group-uid` to identify a group. For roles
with exclusive topology enabled, it adds required Pod affinity for the same group and required Pod
anti-affinity for other groups with that label. In the same namespace, this prevents another
RoleBasedGroup from sharing the topology domain, even if that group does not enable exclusive topology.
The cluster needs enough topology domains for these constraints.

The rules do not exclude Pods without `group-uid` or Pods in other namespaces. They apply when Pods
are scheduled and do not evict already running Pods if conditions later change. A topology domain can
be a node, zone, or other domain defined by a node label; some clusters restrict required Pod
anti-affinity to `kubernetes.io/hostname`.

## Annotation

Expand All @@ -28,6 +37,10 @@ To skip exclusive-topology affinity injection for a specific role, add this anno
rbg.workloads.x-k8s.io/role-disable-exclusive: "true"
```

That role keeps its group label, so its Pods can still be affinity targets for enabled roles in the
same group. If same-group Pods already occupy multiple topology domains, enabled roles may follow
any of those domains.

## Example: RoleBasedGroup with Exclusive Topology

The [rbg-with-exclusive-topology example](../../examples/basic/rbg/scheduling/exclusive-topology.yaml) demonstrates how to use exclusive topology with a RoleBasedGroup, ensuring all roles are scheduled on the same node.
Expand All @@ -40,10 +53,20 @@ RoleBasedGroupSet extends the exclusive topology feature to multiple groups, whe

In this example:

- 2 RoleBasedGroups will be created (as specified by `replicas: 2`)
- 3 RoleBasedGroups will be created (as specified by `replicas: 3`)
- Each RoleBasedGroup will have its pods scheduled on a separate node
- All pods within each individual RoleBasedGroup will be co-located on the same node

## Existing workloads

Enabling or changing exclusive topology on a running RBG changes its Pod templates and may replace
Pods. Upgrading the controller can also replace Pods of RBGs already configured for exclusive
topology, because the affinity selector changes from `group-unique-hash` to `group-uid`.

Existing workload selectors are preserved. If a selector requires a different `group-uid`, the
controller reports a conflict. When both required Pod affinity and anti-affinity already use the
topology key, the controller keeps them without adding its own; check their selectors.

## Use Cases

1. **Disaggregated Inference**: Deploy Prefill and Decode on the same node to reduce inter-PD communication overhead
Expand All @@ -55,4 +78,4 @@ In this example:
- **Reduced Network Latency**: Co-located pods can communicate with minimal network overhead
- **Improved Performance**: Related workloads benefit from data locality
- **Resource Efficiency**: Better utilization of node resources by grouping related workloads
- **Simplified Management**: Logical grouping of distributed applications
- **Simplified Management**: Logical grouping of distributed applications
4 changes: 2 additions & 2 deletions doc/reference/variables.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,9 @@
| Key | Description |
|-----|-------------|
| `rbg.workloads.x-k8s.io/group-name` | The name of the RoleBasedGroup to which these resources belong. |
| `rbg.workloads.x-k8s.io/group-uid` | A short hash identifying all Pods belonging to the same RoleBasedGroup instance. Used for topology affinity. |
| `rbg.workloads.x-k8s.io/group-uid` | A hash of the RoleBasedGroup's namespace and name, used for exclusive-topology affinity. It is not the Kubernetes object UID. |
| `rbg.workloads.x-k8s.io/group-revision` | The revision hash of the RoleBasedGroup, used to determine whether the RBG object has changed. |
| `rbg.workloads.x-k8s.io/group-unique-hash` | Used for pod affinity rules in exclusive topology. |
| `rbg.workloads.x-k8s.io/group-unique-hash` | Legacy exclusive-topology selector key. The constant and user-supplied labels are retained; new rules use `group-uid`. |

### Role Level Labels

Expand Down
11 changes: 5 additions & 6 deletions examples/basic/rbg/scheduling/exclusive-topology.yaml
Original file line number Diff line number Diff line change
@@ -1,23 +1,22 @@
# Example: RoleBasedGroup with exclusive topology (v1alpha2)
# Pods of different roles are scheduled on different nodes (or topology zones)
# by setting the exclusive-topology annotation.
# Pods of the same group are scheduled in the same topology domain.
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
name: nginx-exclusive
namespace: default
annotations:
rolebasedgroup.workloads.x-k8s.io/exclusive-topology: "kubernetes.io/hostname"
rbg.workloads.x-k8s.io/group-exclusive-topology: "kubernetes.io/hostname"
spec:
roles:
- name: frontend
replicas: 1
standalonePattern:
template:
# To disable exclusive topology for this specific role, add:
metadata:
annotations:
rolebasedgroup.workloads.x-k8s.io/disable-exclusive-topology: "true"
# metadata:
# annotations:
# rbg.workloads.x-k8s.io/role-disable-exclusive: "true"
spec:
containers:
- name: nginx-frontend
Expand Down
9 changes: 4 additions & 5 deletions examples/basic/rbgs/rbgs-exclusive-topology.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,10 @@ metadata:
spec:
replicas: 3
groupTemplate:
metadata:
labels:
app: inference-cluster
annotations:
rolebasedgroup.workloads.x-k8s.io/group-exclusive-topology: "kubernetes.io/hostname"
labels:
app: inference-cluster
annotations:
rbg.workloads.x-k8s.io/group-exclusive-topology: "kubernetes.io/hostname"
spec:
roles:
- name: prefill
Expand Down
5 changes: 4 additions & 1 deletion internal/controller/workloads/rolebasedgroup_controller.go
Original file line number Diff line number Diff line change
Expand Up @@ -1732,7 +1732,10 @@ func RBGPredicate() predicate.Funcs {
oldRbg, ok1 := e.ObjectOld.(*workloadsv1alpha2.RoleBasedGroup)
newRbg, ok2 := e.ObjectNew.(*workloadsv1alpha2.RoleBasedGroup)
if ok1 && ok2 {
if !reflect.DeepEqual(oldRbg.Spec, newRbg.Spec) {
oldTopology, oldHasTopology := oldRbg.Annotations[constants.GroupExclusiveTopologyKey]
newTopology, newHasTopology := newRbg.Annotations[constants.GroupExclusiveTopologyKey]
if !reflect.DeepEqual(oldRbg.Spec, newRbg.Spec) ||
oldHasTopology != newHasTopology || oldTopology != newTopology {
ctrl.Log.Info("enqueue: rbg update event", "rbg", klog.KObj(e.ObjectOld))
return true
}
Expand Down
25 changes: 25 additions & 0 deletions internal/controller/workloads/rolebasedgroup_controller_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ import (
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/client/fake"
"sigs.k8s.io/controller-runtime/pkg/client/interceptor"
"sigs.k8s.io/controller-runtime/pkg/event"
"sigs.k8s.io/controller-runtime/pkg/log/zap"
"sigs.k8s.io/controller-runtime/pkg/reconcile"
"sigs.k8s.io/rbgs/pkg/reconciler"
Expand All @@ -65,6 +66,30 @@ import (
wrappersv2 "sigs.k8s.io/rbgs/test/wrappers/v1alpha2"
)

func TestRBGPredicateExclusiveTopologyAnnotation(t *testing.T) {
key := constants.GroupExclusiveTopologyKey
for _, tt := range []struct {
name string
old map[string]string
new map[string]string
want bool
}{
{name: "added", new: map[string]string{key: "kubernetes.io/hostname"}, want: true},
{name: "removed", old: map[string]string{key: "kubernetes.io/hostname"}, want: true},
{name: "changed", old: map[string]string{key: "old"}, new: map[string]string{key: "new"}, want: true},
{name: "empty value added", new: map[string]string{key: ""}, want: true},
{name: "unchanged", old: map[string]string{key: "node"}, new: map[string]string{key: "node"}},
{name: "unrelated annotation", new: map[string]string{"example.com/key": "value"}},
} {
t.Run(tt.name, func(t *testing.T) {
old := &workloadsv1alpha2.RoleBasedGroup{ObjectMeta: metav1.ObjectMeta{Annotations: tt.old}}
new := &workloadsv1alpha2.RoleBasedGroup{ObjectMeta: metav1.ObjectMeta{Annotations: tt.new}}
got := RBGPredicate().Update(event.UpdateEvent{ObjectOld: old, ObjectNew: new})
assert.Equal(t, tt.want, got)
})
}
}

func Test_CalculateNextRollingTarget_WithNormalCases(t *testing.T) {
type testCase struct {
name string
Expand Down
15 changes: 12 additions & 3 deletions pkg/reconciler/pod_reconciler.go
Original file line number Diff line number Diff line change
Expand Up @@ -119,12 +119,21 @@ func (r *PodReconciler) ConstructPodTemplateSpecApplyConfiguration(
}

// Set Exclusive topology
var extraLabels map[string]string
if topologyKey, found := rbg.GetExclusiveKey(); found {
if podAnnotations[constants.DisableExclusiveKeyAnnotationKey] == "" {
podAnnotations[constants.GroupExclusiveTopologyKey] = topologyKey
uniqueKey := rbg.GenGroupUniqueKey()
value, hasGroupUID := podLabels[constants.GroupUIDLabelKey]
if hasGroupUID && value != uniqueKey {
return nil, fmt.Errorf("exclusive topology for role %q: workload selector %s=%q conflicts with group identity %q",
role.Name, constants.GroupUIDLabelKey, value, uniqueKey)
}
if !hasGroupUID {
extraLabels = map[string]string{constants.GroupUIDLabelKey: uniqueKey}
}
podAnnotations[constants.GroupExclusiveTopologyKey] = topologyKey
err := setExclusiveAffinities(
&podTemplateSpec, uniqueKey, topologyKey, constants.GroupUniqueHashLabelKey,
&podTemplateSpec, uniqueKey, topologyKey, constants.GroupUIDLabelKey,
Comment thread
Copilot marked this conversation as resolved.
Resolved
Comment thread
cheyang marked this conversation as resolved.
Resolved
)
if err != nil {
return nil, err
Expand Down Expand Up @@ -152,7 +161,7 @@ func (r *PodReconciler) ConstructPodTemplateSpecApplyConfiguration(
r.gangScheduler.InjectPodSchedulingFields(rbg, role, gangStrategy, podTemplateApplyConfiguration)
}

podTemplateApplyConfiguration.WithLabels(podLabels).WithAnnotations(podAnnotations)
podTemplateApplyConfiguration.WithLabels(podLabels).WithLabels(extraLabels).WithAnnotations(podAnnotations)
return podTemplateApplyConfiguration, nil
}

Expand Down
Loading
Loading