(discovery): auto-discover served models and gossip them to peers #179

Summary

A site should auto-discover the models it serves and gossip that set to peers, so routing knows which sites can serve a requested model without hand-maintained per-site model declarations.

Today

Model sets are declared statically (xtask def.models, enrollment --model) and the per-site capability is populated by hand (crdt::Capability::Model(...) via add_capability). Nothing discovers the served-model set from the backends, and nothing keeps the gossiped set in sync with reality: a model scaled to zero or removed still looks available. RHAIRFE-3356 scoped automatic model discovery out of the basic path, so this is the follow-on capability that lifts that.

Proposal

Discovery: each site learns its served-model set from the provider backends on a poll cadence (the same routing-input poll model the gateway is moving to), and a model that disappears drops from the set. The discovery source should stay pluggable, not pinned to one method. One mechanism is querying the provider's GET /v1/models (KServe LLMInferenceService and vLLM expose it); others are possible, for example reading the served models off the operator's own view of the deployments, so the design should treat the source as an interface even if /v1/models is the first implementation.

Represent: populate the existing per-site crdt::Capability::Model(...) set from discovery, last-writer-wins, single-writer-per-key so each site owns only its own model set.

Gossip: ride the existing SWIM StateBroadcast / CRDT channel that already carries site_cert_pem and the cross-site signals, bounded by a per-site model cap.

Consume: intelligent_route reads the gossiped per-site model set for its serves-the-model eligibility check (RHAIRFE-3356 AC eligibility), and the model-addressable endpoint resolves candidates from it.

Considerations

Freshness: the gossiped set carries freshness, and a stale set fails closed (do not route to a site whose model set is stale), consistent with routing-input freshness.

Logical model mapping: discovery yields raw served-model names. Mapping those to a logical model identity (RHAIRFE-3356) may stay admin-declared or be derived; open question, not required for the raw discovery+gossip.

Security: the gossiped set must be authenticated to the owning site. A peer must not advertise another site's models, which would poison routing. This depends on the signed-membership / mesh-identity work (#75, #90, #92).

Bounds: cap models per site and reject a malformed discovery response rather than clearing the set.

Acceptance criteria

• A site's served-model set is discovered from its backends, not from static config.
• The set gossips to peers over SWIM/CRDT as per-site Capability::Model, single-writer-per-key.
• intelligent_route eligibility reads the gossiped set; a model absent from a site is not routed there.
• A model removed or scaled to zero drops from the site's set within the freshness window; a stale set fails closed.
• A peer cannot advertise another site's models.