[nikolasmatt]

Description

After this, a gloo v1.19 release, and any PR build that pushes images, publishes only after a second person approves it. The Cloud Build config no longer carries committed credentials, glooctl uploads with the workflow's own token instead of a PAT, and every release is registered in solo-ci.

Refs #11482 · Refs solo-io/solo-tools#270

This is one of four matching PRs, one per release line that still ships: v1.22.x #11483, v1.21.x #11484, v1.20.x #11485, v1.19.x #11486. They can merge one at a time. A small PR against main (#11496) adds the /build-pr-images comment trigger. close-main-prs.yaml closed it because main is EOL, so a repo admin lands it after these four.

For customers

Nothing customer-facing changes. Images still go to quay.io/solo-io and gcr.io/gloo-edge, the chart to the solo-public-helm bucket, and glooctl to the GitHub release assets, under the same names and tags.

For maintainers:

• PR images (<version>-<PR number>) are no longer built for every PR. Comment /build-pr-images on the PR and a second person approves. Nothing in gloo or solo-projects pulls these images; they are for testing a PR by hand.
• Only a version with no suffix or -patchN is a regular release. Mark every other suffix (-beta1, -rc1, -fips) as a pre-release, or the Release workflow stops before publishing.

How a release works after this

1. A maintainer creates the release, as today. They create and publish the GitHub release on a v1.19.x commit in the UI, with notes and the pre-release flag (for example v1.19.27-beta1 as a pre-release). It must be a person: the workflow rejects a release published by an App.
2. The workflow checks the tag. The tag's commit must be on v1.19.x, and the pre-release flag must match the version. If the flag is wrong, edit the release and use "Re-run failed jobs".
3. A second person approves. The build job waits in environment release. A member of solo-io/gloo-release-owners other than the publisher approves it, after checking the tag's commit is on v1.19.x.
4. Cloud Build publishes. The job signs in to Google Cloud with a short-lived Workload Identity token and submits ci/cloudbuild/publish-artifacts.yaml as the gloo-release service account. Images go to quay.io/solo-io and gcr.io/gloo-edge, the chart to gs://solo-public-helm.
5. glooctl and solo-ci. glooctl is built and uploaded to the release, and the release is registered in solo-ci as product gloo.
6. The evidence stays with the run. The run records who approved, its summary links the Cloud Build id and logs, and solo-ci holds the release's image list for scanning.

To stop a mistaken release, run the gcloud builds cancel command the step log prints; cancelling the workflow run does not stop the build. Re-running a failed run attaches to the tag's existing build instead of publishing twice.

How PR images work after this

1. Someone asks. The PR author, or anyone with write, comments /build-pr-images. The relay on main starts "PR images" on the PR's base branch; it can also be run from the Actions tab.
2. A second person approves. The approver must not be the requester, the PR author or anyone who pushed to the PR, and approves only if the run title's commit is the one they reviewed. They check the PR's Makefile, ci/ and Dockerfile changes first, because the build runs that code with the quay.io robot logged in.
3. Cloud Build publishes as the gloo-pr-build service account, with the same tags and chart bucket build-bot used. Each new push needs a new request.

Known limit: once approved, a PR build could overwrite any tag in quay.io/solo-io and gcr.io/gloo-edge, release tags included. The approval is the control.

Learn more

• Releases and PR images in this repo: the runbook on this branch, including how to add a release line.
• Release guide for product owners on the solo-tools release platform. gloo stays on Cloud Build, so the approval model and solo-ci registration apply here, not the shared workflow.
• Pipeline API reference.
• A finished platform release from enterprise-kagent, to see the approval and run summary.

What changes

Area	Before	After
Release publish	build-bot publishes on any new tag	.github/workflows/release.yaml publishes after approval in environment release
PR images	built and pushed for every PR	.github/workflows/pr-images.yaml, on /build-pr-images plus approval
PR tests	build-bot	build-bot, unchanged; ci/cloudbuild/run-tests.yaml also cross-builds glooctl
cloudbuild.yaml	submits publish-artifacts.yaml	no longer publishes
Credentials	KMS ciphertext in publish-artifacts.yaml	quay.io password from Secret Manager; no GITHUB_TOKEN in Cloud Build
glooctl assets	uploaded from Cloud Build with a PAT	uploaded by the workflow with its own token; same names and .sha256 files
solo-ci	not registered	register job, solo-tools action pinned to release-v0.2.0
Waiting on builds	none	ci/cloudbuild/wait-for-build.sh, shared by both workflows
Docs	none	ci/cloudbuild/README.md Releases and PR images sections

Release notes, the destinations, image and chart names, and the builder images stay the same.

Before the first run

State on 2026-11-05.

• IT (solo-public): service accounts gloo-release and gloo-pr-build, and secret gloo-quay-io-password, readable only by them
• IT: Workload Identity provider gloo, accepting only solo-io/gloo, environment release, and release.yaml on a v* tag or pr-images.yaml dispatched (workflow_dispatch) on v1.19.x-v1.22.x, each with its own service account
• IT: write on gcr.io/gloo-edge and the Helm buckets for the new accounts; build-bot's account loses write on both Helm buckets and can't act as either account or read the secret
• environment release: required reviewers gloo-release-owners, prevent self-review, admin bypass off, deployments from v* tags and v1.19.x-v1.22.x
• variable GLOO_RELEASE_WIF_PROVIDER
• solo-ci product gloo (solo-io/solo-tools#509)
• Rav: rotate the quay.io robot password into gloo-quay-io-password (the secret has no version yet)
• revoke the token behind the removed GITHUB_TOKEN ciphertext, and make sure the org PAT and any shared machine-user token can't create releases on solo-io/gloo
• repo admin: set GLOO_RELEASE_SA and GLOO_PR_BUILD_SA once IT confirms the emails
• IT: enable the gloo provider and both service accounts at cutover (all disabled now)
• build-bot owner: turn off tag builds for gloo
• repo admin: ruleset making v* tags immutable, so a published tag can't be moved to another commit
• repo admin: land Relay /build-pr-images comments to the release-line PR images workflow #11496 on main after the four release-line PRs (disable "Close PRs to main", reopen and merge, enable it again)

Until the rotation and revocation are done, a PR can still decrypt the old credentials from git history through build-bot. After rotation, open PRs on an older base fail build-bot's publish step until they merge the new base.

Testing steps

• actionlint passes on both workflows, shellcheck on the wait script, and the Cloud Build configs parse.
• The solo-tools action is unchanged between the previous pin 2a6fe07c and release-v0.2.0; only the pin and its comment moved.
• A stubbed-gcloud test of submit-and-wait covers a first run, attaching to a running build, skipping a published tag, retrying after a failed build and a flaky status check.
• A stubbed gh runs the request, approver, post-approval and relay scripts through 35 cases, including a wrong base, a moved head, a spoofed requester and approval by the requester or author.
• The tag check maps v1.19.27 and -patch1 to a regular release and -beta1, -rc2, -fips to a pre-release, and fails on a wrong flag.
• Nothing has run against GitHub or Google Cloud yet. The first pre-release after merge is the end-to-end test: check that the approval view names environment release and that gcloud builds describe shows the gloo-release service account.

Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works