Repository navigation
Move job queues to lib-data-workqueue, fix duplicate job processing and drain on shutdown - #1201
Merged
Merged
Conversation
Move the job queues from the message stream API onto the work queue API of libseqera. `lib-data-workqueue-redis` is `lib-data-stream-redis` 1.5.0 on lease semantics: the same Redis stream layout (xadd/xreadGroup/xautoclaim, xack+xdel on settle, same `data` field), so the queue keys and the consumer group are left untouched and messages queued before the upgrade are still delivered afterwards. What the lease buys over the previous claim-timeout heuristic: an in-flight entry is renewed at visibility-timeout/4, so a slow consumer's message can no longer be claimed by another replica mid-processing; and close() waits cooperatively for the current cycle instead of interrupting the listener thread after 1s, letting a consumer finish its writes before teardown. `JobManager` keeps its own admission control - launchJob/processJob are unchanged and still return a boolean, now mapped onto ACK/RETRY at the addConsumer site - so the maxRunningJobs cap on concurrently running jobs is unaffected. Config keys are renamed `wave.message-stream.*` -> `wave.work-queue.*` (claim-timeout -> visibility-timeout). The consumer group keeps defaulting to `wave-message-stream` so existing pending entries stay visible. The prod/stage overrides in platform-deployment need the matching rename. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The `wave.work-queue.*` settings now fall back to the legacy `wave.message-stream.*` keys they replace, and default to the current production values (45s), so a deployment configured for the message-stream implementation keeps its behaviour after the work-queue upgrade with no config change. Update the configuration reference accordingly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Micronaut ends a placeholder at the first `}`, so the nested form
`${a:${b:default}}` resolved to `45s}` and failed Duration conversion,
taking down the eagerly-initialized context at boot and breaking 32
Redis-backed tests. The multi-key form `${a:b:default}` resolves the
legacy `wave.message-stream.*` fallback as intended.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…eam-with-workqueue
The work queue redelivers RETRY-settled jobs only after the visibility timeout (default 45s), which exceeds the 30s timeout of job timeout tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The processing queue consumer now submits each processJob call to the IO executor and returns DEFERRED; the task settles the lease (ack/retry) when it completes. The lease is bound to the task liveness, so a slow status check is renewed for as long as it runs instead of being redelivered to another replica, and status checks no longer run serially on the dispatcher thread. A ready() admission gate stops claiming while maxRunningJobs checks are in flight. The pending queue stays synchronous: parallel launches would make the maxRunningJobs check in launchJob0 racy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ClientBuilder leaves the OkHttp read timeout at 0 (infinite), so a stalled API server connection could block a job status check forever: with the DEFERRED model that task's lease would stay renewed and the job would never be recovered. Add `wave.build.k8s.read-timeout` (default 50s) to both the in-cluster and the kubeconfig clients. The default sits above the ~30s after which the API server fails a pod log request whose kubelet is unreachable (observed in prod as a 500), so that more specific error is reported rather than a client-side timeout racing it. The timeout bounds the gap between bytes, not the whole request, so a large pod log streamed steadily is not cut short. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
JobManager now implements GracefulShutdownCapable, so Micronaut runs the drain before destroying any bean and the jobs being processed complete while their collaborators (k8s client, Redis, database) are still usable. Previously each queue was closed in its own @PreDestroy, sequentially (up to 10s each) and during bean destruction. The drain, bounded by `wave.job-manager.drain-timeout` (15s): 1. stops both queues claiming, quiescing them in parallel within a quarter of the budget, so the next step always gets the remaining time 2. waits for the status checks running on the IO executor; a check not yet started once the drain begins is settled as retry instead 3. releases the queues with the remaining budget, making the @PreDestroy close a no-op Jobs not completed in time keep their entry and are redelivered to another replica once the lease expires. Graceful shutdown is enabled with a 20s grace period, which sits between the drain timeout and the pod default terminationGracePeriodSeconds (30s). This also makes the Netty server drain in-flight HTTP requests on shutdown. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace the DEFERRED status-check model with synchronous consumers that bind a constant liveness probe: the lease is renewed for as long as the consumer runs, so a slow job launch or status check is no longer redelivered to another replica. The graceful shutdown drain reduces to closing both queues within the drain timeout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pditommaso
changed the title
jordeu
approved these changes
Oct 20, 2026
Deployments not setting the legacy claim timeout went from 5s to 45s, delaying job re-checks and full-queue launch retries. 20s bounds that delay while leaving room for missed lease renewals. Document the changed defaults. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Moves the job queues from the message stream to the libseqera work queue and fixes the duplicate processing of slow job checks. Supersedes #1115 (migration included unchanged), closes #1129.
Problem
The message stream does not renew leases: a job check running longer than the claim timeout (45s) is claimed by another replica while still in progress. In prod (2026-10-06) a single build completion was processed by 3 replicas (~2.5 min each, stalled on a Tower credentials timeout), firing 3 completion events, duplicate scans and build record save errors. 8 checks took ≥60s in the last 30 days.
Changes
lib-data-workqueue[-redis]:2.0.0). Same Redis stream keys and consumer group, so no Redis migration. Eachwave.work-queue.*setting falls back to its legacywave.message-stream.*key, so no config change is needed.lease.bindLiveness(() -> true), so the lease is renewed for as long as the consumer runs. The constant probe is exact for a synchronous consumer: the lease exists only whileaccept()runs.wave.build.k8s.read-timeout, default50s). The client default is infinite, which, with the lease kept alive, would let a stalled API read hold a job forever. The value sits above the ~30s after which the API server fails an unreachable pod log request (3 times in prod in 30 days).JobManagerimplementsGracefulShutdownCapableand closes both queues withinwave.job-manager.drain-timeout(15s), before any bean is destroyed. A job not done in time is redelivered after the visibility timeout. Enablesmicronaut.lifecycle.graceful-shutdown(20sgrace period), which also drains in-flight HTTP requests. The deployment side is seqeralabs/platform-deployment#3563 (optional: 20s fits the default 30s termination budget).Rollout
No Redis migration and no required config change. Mixed-version replicas are safe, and delivery stays at-least-once.
Known gap
Launch is not idempotent: a job redelivered after its pod died between creating the k8s Job and acking the entry fails with 409 AlreadyExists. Follow-up.
🤖 Generated with Claude Code