Skip to content

feat: add WithSecretsFieldSelector to filter the core cache's Secrets - #596

Merged
soloio-bulldozer[bot] merged 2 commits into
mainfrom
chandler/solo-kit-store-every-secret
Nov 6, 2026
Merged

soloio-bulldozer[bot] merged 2 commits into
mainfrom
chandler/solo-kit-store-every-secret

Conversation

@chandler-solo

Copy link
Copy Markdown
Contributor

Summary

KubeCoreCache lists and watches every Secret in the namespaces it watches. Clusters often hold many Secrets the consumer never reads, such as Helm release storage (helm.sh/release.v1) and ServiceAccount tokens. The API server still sends all of them, and the cache keeps each one in memory for the life of the process.

This adds WithSecretsFieldSelector, an option for NewKubeCoreCacheWithOptions:

cache.NewKubeCoreCacheWithOptions(ctx, clientset, resync, namespaces,
    cache.WithSecretsFieldSelector(fields.AndSelectors(
        fields.OneTermNotEqualSelector("type", "helm.sh/release.v1"),
        fields.OneTermNotEqualSelector("type", "kubernetes.io/service-account-token"),
    )))

The selector is added to every LIST and WATCH request the cache makes for Secrets. The API server applies it, so excluded Secrets are never sent or stored.

Motivation

Gloo Edge uses KubeCoreCache. It downloads and holds every Secret.

Behavior

  • Compatibility: NewKubeCoreCacheWithOptions gains a variadic opts ...Option parameter, so existing calls compile unchanged.
  • Default: without the option, or with a nil or empty selector, the cache stores every Secret, as before.
  • Scope: Secrets only. Pods, Services, ConfigMaps and Namespaces are unaffected.
  • Effect on the kube secret client: the secret listers, and the kube secret client's List and Watch, skip excluded Secrets. Read fetches a Secret by name from the API server, so it still returns them.
  • Supported fields: Secrets can only be selected by metadata.name, metadata.namespace and type. Any other field is rejected by the API server, which leaves the cache unable to sync.

Testing

  • New unit tests: pkg/api/v1/clients/kube/cache/secrets_field_selector_test.go uses a fake clientset to record the namespace and field selector of every list and watch request. It covers:
    • no option, a nil selector and an empty selector
    • one term and two terms
    • the last of two options winning
    • a cache watching two namespaces, which sends the selector in both
    • no other resource getting a selector
  • Both sync paths: the fake can't serve watch-list streams, so the test rejects them and client-go falls back to LIST and WATCH. The selector is checked on all three kinds of request.
  • Race detector and feature gate: the tests pass with -race, with KUBE_FEATURE_WatchListClient both on and off.
  • Mutation check: with the filter removed, the selector cases fail.
  • Downstream: gloo's callers of NewKubeCoreCacheWithOptions compile against this branch.

## Summary

`KubeCoreCache` lists and watches every Secret in the namespaces it watches. Clusters often hold many Secrets the consumer never reads, such as Helm release storage (`helm.sh/release.v1`) and ServiceAccount tokens. The API server still sends all of them, and the cache keeps each one in memory for the life of the process.

This adds `WithSecretsFieldSelector`, an option for `NewKubeCoreCacheWithOptions`:

    cache.NewKubeCoreCacheWithOptions(ctx, clientset, resync, namespaces,
        cache.WithSecretsFieldSelector(fields.AndSelectors(
            fields.OneTermNotEqualSelector("type", "helm.sh/release.v1"),
            fields.OneTermNotEqualSelector("type", "kubernetes.io/service-account-token"),
        )))

The selector is added to every LIST and WATCH request the cache makes for Secrets. The API server applies it, so excluded Secrets are never sent or stored.

## Motivation

Gloo Edge uses `KubeCoreCache`. It downloads and holds every Secret.

## Behavior

- **Compatibility:** `NewKubeCoreCacheWithOptions` gains a variadic `opts ...Option` parameter, so existing calls compile unchanged.
- **Default:** without the option, or with a nil or empty selector, the cache stores every Secret, as before.
- **Scope:** Secrets only. Pods, Services, ConfigMaps and Namespaces are unaffected.
- **Effect on the kube secret client:** the secret listers, and the kube secret client's `List` and `Watch`, skip excluded Secrets. `Read` fetches a Secret by name from the API server, so it still returns them.
- **Supported fields:** Secrets can only be selected by `metadata.name`, `metadata.namespace` and `type`. Any other field is rejected by the API server, which leaves the cache unable to sync.

## Testing

- **New unit tests:** `pkg/api/v1/clients/kube/cache/secrets_field_selector_test.go` uses a fake clientset to record the namespace and field selector of every list and watch request. It covers:
  - no option, a nil selector and an empty selector
  - one term and two terms
  - the last of two options winning
  - a cache watching two namespaces, which sends the selector in both
  - no other resource getting a selector
- **Both sync paths:** the fake can't serve watch-list streams, so the test rejects them and client-go falls back to LIST and WATCH. The selector is checked on all three kinds of request.
- **Race detector and feature gate:** the tests pass with `-race`, with `KUBE_FEATURE_WatchListClient` both on and off.
- **Mutation check:** with the filter removed, the selector cases fail.
- **Downstream:** gloo's callers of `NewKubeCoreCacheWithOptions` compile against this branch.

## Changelog

`changelog/v0.41.3/secrets-field-selector.yaml` (NEW_FEATURE). The change is additive and opt-in, so it ships as a patch.

Signed-off-by: David L. Chandler <david.chandler@solo.io>
@solo-changelog-bot

Copy link
Copy Markdown

Issues linked to changelog:
https://github.com/solo-io/solo-kit/pull/TODO

Signed-off-by: David L. Chandler <david.chandler@solo.io>
@solo-changelog-bot

Copy link
Copy Markdown

Issues linked to changelog:
#184

@soloio-bulldozer
soloio-bulldozer Bot merged commit 4d85f66 into main Nov 6, 2026
4 checks passed
@soloio-bulldozer
soloio-bulldozer Bot deleted the chandler/solo-kit-store-every-secret branch November 6, 2026 16:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants