[andy-fong]

Description

fixes https://github.com/solo-io/solo-projects/issues/8668

The detail reproduce steps are in the ticket. Here is what happened:

• when an upstream is added pointing to an invalid service, we still set EDS on the cluster
• however because the cluster is invalid, we never send it over to envoy.
• So, the last xds CLA subscription from envoy still only subscribe to the 2 valid endpoints resources after that:

{"level":"debug","ts":"2026-01-30T20:53:17.480Z","caller":"cache/simple.go:281","msg":"open watch Priority Index 1 and Element Index 8 for type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment[working_gloo-system-10760264581693208509 rate-limit_gloo-system-15891676738363403541] from nodeID \"gloo-system~gateway-proxy\", version \"8495508594003352137-3112950219084914976\""}

• in the control-plane, the clusters is already updated with the upstream, so when we fix the service (now endpoints will get updated correctly but the cluster is effectly no changed, so no update is sent to envoy for clusters resource). Because envoy at this point is "out-of-sync" and have not asked for those new endpoints, control-plane just bails:

{"level":"debug","ts":"2026-01-30T20:53:36.477Z","caller":"cache/simple.go:162","msg":"respond open watch priority 8 and index 1 :[working_gloo-system-10760264581693208509 rate-limit_gloo-system-15891676738363403541] with new version \"8495508594003352137-13617073202604264992\""}
{"level":"debug","ts":"2026-01-30T20:53:36.477Z","caller":"cache/simple.go:320","msg":"ADS mode: not responding to request: \"test_gloo-system-5426613417106913200\" not listed"}

• in this particular case, the control plane also silently delete the watch even no xds response is sent to envoy. So, envoy is sitting there waiting for any endpoint updates but the watch has been deleted and no new endpoint changes will ever get sent to envoy. I wonder if we should fix this. See this PR for detail. This effectively hide any further debug logs like the above for any endpoints changes which gives the appearance that envoy is stuck.
• because of this, restarting the control plane does not fix it because it's envoy's subscription that is out of sync (asking for 2 endpoint resources but control plane has 3 endpoint resources), after the first control plane failed attempt to response to CLA subscription, the watch is deleted silently again.
• restarting envoy fixes it because now envoy is in sync again by asking for everything.

Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works

[github-actions]

Visit the preview URL for this PR (updated for commit 193db64):

https://gloo-edge--pr11118-andyfong-fix-xds-out-7pgn4whe.web.app

_{(expires Wed, 25 Feb 2026 19:26:12 GMT)}

_{🔥 via Firebase Hosting GitHub Action 🌎}

_{Sign: 77c2b86e287749579b7ff9cadb81e099042ef677}

[solo-changelog-bot]

Issues linked to changelog:
https://github.com/solo-io/solo-projects/issues/8668

[davidjumani]

the control plane also silently delete the watch even no xds response is sent to envoy

Should we worry about this causing issues in other scenarios ? I'm happy to get solo-io/solo-kit#586 in to prevent other similar issues from popping up

[andy-fong]

the control plane also silently delete the watch even no xds response is sent to envoy

Should we worry about this causing issues in other scenarios ? I'm happy to get solo-io/solo-kit#586 in to prevent other similar issues from popping up

yeah... definitely want more input from SME in this area. @yuval-k can you take a look on this PR and the one in solo-kit linked in description above as well?
I think we should at least print a warning (or maybe even error) that we deleted a watch without responding.