Repository navigation
Conversation
main CI failed on the README badge commit because the headless Chrome dump timed out after 30s. The same test passed 35 minutes earlier on the parent commit, so it is a transient runner flake. Retry once with a fresh profile; a second failure is real and still fails the test.
* feat: reuse compatible GPU drivers during setup Validate the selected CUDA profile before replacing drivers. Install only missing runtime prerequisites, gate repairs on verified artifacts and node idleness, and expose resumable maintenance and reboot states to operators. Validation: 3007 tests passed, 1 skipped; 93.78% coverage. Ruff, mypy, shellcheck, and JavaScript syntax checks passed. Closes #127 * fix: copy driver reboot markers directly * fix: address GPU setup review findings Restore Python development headers for Triton's runtime compilation and preserve the selected GPU mask and architecture during CUDA proof. Rebuild compatible RPM and runfile/DKMS drivers, detect every driver package family, and enforce Volta-safe bootstrap limits before hardware measurement. Unload known NVIDIA leaf modules first and report persistent holders as maintenance rather than requesting repeated reboots. Limit DRM idleness checks to NVIDIA PCI devices and consolidate compatibility constants, compiler version parsing, prerequisite steps, and resume-state labels. Add setup regressions for fresh nodes, mixed GPU hosts, driver package variants, DKMS recovery, bootstrap limits, module holders, and DRM clients. Validation: 3021 tests passed, 1 skipped; 93.78% coverage. Ruff, mypy, ShellCheck 0.9/0.11, and JavaScript syntax checks passed.
Automatically generated by python-semantic-release
* Publish engine runtimes and setup bundles transactionally Verify immutable bundles and runtime generations before atomically activating them. Retain rollback state, reconcile interrupted activation, and record selected generation evidence in provisioning attempts. Add fault-injection setup and launch regressions plus recovery documentation. Fixes #128 * Fix transactional provisioning review regressions Separate service restart settings from fresh launches, preserve stop access with damaged runtimes, and tolerate system interpreter updates. Persist recorder locations and harden rollback, bundle repair, setup privileges, and generation marker parsing. Add recovery regressions for the feedback on PR #219 and document the resulting behavior.
Recover damaged runtime publications, skip malformed catalog entries, honor automatic job sizing, and bind compatibility to explicit engine/probe CUDA targets. Fix producer documentation and download diagnostics, and extend the coverage gate to the runtime and generation helpers.
Automatically generated by python-semantic-release
Signed-in users get /leaderboard: non-admin usage ranking (admins are never listed), their own tokens with create/delete (create keeps the one-token rule and replaces the previous token), optional model and agent-harness pickers with per-harness config download, and docs/user-guide.md covering the onboarding experience. fixes: #156
Automatically generated by python-semantic-release
- _probe_liveness now returns the /v1/models primary id from its fallback fetch; _refresh_self_setup_model reconciles the tracked model each cycle via etcd revision CAS, registry-first-on-CAS, endpoint-identity guard, last-known model on transient failures. - refresh runs before the half-open /v1/completions trial so breaker recovery uses the live model id. - run_health_checker/_probe_all_nodes/_probe_node thread etcd_client; main.py passes it. - regression tests in TestSelfSetupModelAutoRediscovery. fixes #222
Automatically generated by python-semantic-release
) * feat: power on new nodes via redfish and resolve bmc creds per lab Gate the placement SSH probe on Redfish power state: a host the BMC reports Off skips the probe and is powered on during provisioning, so powered-off new nodes stop being deferred as unreachable before _power_on_if_needed can run. Resolve BMC credentials per hostname with precedence full-FQDN override, lab identifier (dot-bounded substring), then the global default, so hosts across labs with different BMC passwords power on correctly. Fixes #225 * test: guard nested BmcCredential password secrecy in repr The repr assertion checked the default bmc_password, already a SecretStr, so a regression of BmcCredential.password to a plain str went undetected. Assert a distinctive lab-override value is absent from repr instead, so the test fails if the nested field stops being secret. * fix: address review findings on redfish power gate and bmc resolver Probe SSH first and read BMC power only when the probe fails, so reachable hosts never touch the BMC. Equal-length lab identifiers now resolve lexicographically regardless of config order, and bmc credential keys are normalized with invalid and duplicate keys rejected at startup. Adds the prod stuck-claim retry and fallback tests, and documents the power-on behavior, exclude_hosts, and the kwargs. * docs: describe bmc credential key normalization accurately Whitespace is stripped, a trailing dot is stripped on hosts keys and rejected only on labs keys; empty, leading-dot and duplicate keys are rejected.
Automatically generated by python-semantic-release
Add a local OpenID Connect sign-in option for dev testing and small installs, off by default so Google-only deployments are unchanged: - builtin auth.internal_oidc plugin (refuses to load unless plugins.config enables it and supplies server_metadata_url; reuses the INFERENCE_PROXY_OAUTH__* client triple) - AuthIdentity.issuer + plugin label; user rows keyed by (issuer, sub) via an additive users.issuer migration that rebuilds the table to replace the single-column UNIQUE(google_sub) (existing rows backfilled with the Google issuer) - provider selection: /auth/login?provider=<name>, session marker at flow start so the provider-less callback resolves the same plugin - sign-in page renders one button per configured provider (Local Auth only when the opt-in plugin is loaded); profile.js messaging is provider-neutral - vendored toy OIDC provider under oidc-provider/ (themed login page, OIDC_DOMAIN env for the email suffix, per-request users file, RSA key) - RPM: provider under /usr/share/qiip/oidc-provider, qiip-oidc.service installed but not enabled, /etc/qiip/oidc-users.txt + oidc.env (0600, %config noreplace), /var/lib/qiip/oidc key dir, nginx /oidc/ location; pyjwt/python-multipart joined the runtime deps - docs: README, user-guide, plugins.yml.example, UPGRADING
When the opt-in local OIDC provider is enabled, the sign-in page's local form is labelled "Local Login" and replaces both the Local Admin form and the separate Local Auth provider button: - the configured admin username/password is checked first and always wins (a same-named entry in the local user list is ignored); - any other username/password is checked against the provider's plaintext user list (new auth.internal_oidc users_file/domain config), and a match signs that local user in under the same (issuer, sub) identity the OIDC flow produces; - Google-only deployments keep the unchanged "Local Admin" + "Google Auth" buttons; a local-only deployment shows just "Local Login". Also: AuthPlugin gains a verify_local_credentials hook (Google returns None), the toy provider ignores an optionally configured OIDC_ADMIN_USERNAME in its user list, and docs/config examples are updated.
- store: atomic (issuer, sub) migration via users_new rebuild in one transaction; PRAGMA legacy_alter_table=ON keeps child-table FKs on users; leftover users_legacy_issuer/users_new recovery guards; cross- issuer email rebind refused (AccountConflictError, 409 on the form) - provider: HTML-escape every interpolated login form field; validate client_id/response_type/redirect_uri before any redirect (400 error page, no open redirect), revalidate on POST /login; /token requires the matching redirect_uri; UTF-8 byte compares everywhere - sign-in policy: shared check_signin_policy helper (email verification, allowed_domains, SSO whitelist) now used by the OIDC callback and the Local Login form; full-access grants are issuer-scoped (admin_only_tokens_full_access is Google-only, new admin_only_tokens_full_access_by_issuer map) - domain: single source via discovery user_email_domain; plugin consumes it and drops its own domain config key - plugin interface: AuthIdentity.issuer defaults to the Google issuer - docs: README/users_file paths, SSL_CERT_FILE trust, Local Login wording; UPGRADING classification and migration accuracy - rpm: systemd macros for qiip-oidc (post/preun/postun_with_restart), explicit python3-cryptography - tests: provider security suite, issuer-scoping + policy unit tests, interface default test, .env-independent test settings
The users_legacy_issuer repair selected an 'issuer' column the leftover pre-issuer table never had (rows stayed stranded), and failed on databases predating is_admin. Backfill the Google issuer with a bound parameter and select is_admin only when the column exists; run the insert + drop in one transaction. Add the review-requested regression tests: child-table DDL keeps REFERENCES users and PRAGMA foreign_key_check stays empty after the rebuild, and both crash shapes (rename and drop-before-rename) recover their rows on open.
self-signed trust via SSL_CERT_FILE works for local-only deployments but breaks Google's server-side discovery fetch; mixed deployments need the internal-CA route.
CI Quality runs mypy over inference_proxy and tests; the new tests broke no_implicit_reexport and json() typing: - import GOOGLE_ISSUER from its source (auth._constants) instead of the store module attribute - the fake discovery fixture now patches httpx.get directly and returns a properly typed dict payload
Automatically generated by python-semantic-release
When an assignment ends, QUADS lists the host as available at once but runs the move back to its default cloud minutes later, and that move powers the host off. Placement polled availability alone, so it started setup inside that window and lost the host mid-install, five times on 2026-10-11. Placement now reads /api/v3/moves on every pass and again at launch under the lifecycle lease. A host with a pending move is skipped before any probe, for fresh candidates and retries alike: no attempt is spent, the claim keeps its error and budget, and the host is reconsidered on the next pass. Manual managed setup answers 409 for such a host and 503 when the moves cannot be read. A pass also refuses to plan from a QUADS inventory snapshot older than the new quads.stale_after_seconds.
…rval A bound at or below the poll interval reads a healthy poller as stale whenever a pass lands late in the cycle, and the field only had a 60s floor. Validate it against poll_interval at startup, with the minimum for the configured interval in the message. Twice the interval is headroom over the poll cadence, not a tolerance for a fixed number of failed polls, since request time ages the snapshot too; the docs say so.
Record the latest attempt's engine when the reconcile gate refuses a new setup, and return actionable recovery guidance instead of 404 for legacy tasks without an engine.
Raise the per-test relaunch guard from 1s to 10s; the operations are mocked and normally instant, but Python 3.13 CI occasionally exceeded the old bound. This removes false failures unrelated to the navigation change.
Automatically generated by python-semantic-release
Use an epoch journalctl selector, keep benign journalctl stderr as a warning, and stop marking expected record splitting as a provisioning issue. fixes: #282
Cut records at a UTF-8 boundary and stop treating an expected engine split as an issue.
Drop the benign stderr exception and the nonzero-exit suppression; a journalctl parse failure means no journal was collected and must leave the attempt incomplete.
Clear a finished log stream when a new active task appears without a page action, so the node detail view opens a fresh EventSource. fixes: #284
Track the task identity used by the current EventSource and ignore late events from replaced sources, so reprovisioning a node with visible logs switches to the new attempt without a page refresh.
Pass the task attempt id when known and do not fall back to an older attempt while the new operation is still powering on.
Add an explicit model-cache deletion endpoint and a delete action on the Models page. Deletion removes every revision of one repository, rejects active downloads, concurrent deletions, and models referenced by registered nodes, and keeps the download fence held through request cancellation. Scope Hugging Face deletion to the target repository so a shared commit hash cannot delete a source or mirror repository.
Automatically generated by python-semantic-release
Render /profile for every signed-in user, expose whether the user may create tokens, and keep token creation admin-only while tokens and usage remain visible.
Pin normal and full-access can_create_tokens values on /auth/me.
Show model scope instead of full access for model-scoped tokens, pin the direct normal-user 403, update the user guide, drop the duplicate access test, and keep the caller capability off AdminUserDetail by using a ProfileUser response model.
Link the start page logo to the leaderboard, remove the duplicate header link, and show non-admin users an Onboard link instead of a Dashboard redirect.
Share the home-link rule in nav_home_link: normal users get Onboard, full-access users rely on Profile, and admins keep Dashboard.
Automatically generated by python-semantic-release
Replace the development rewrite with a linear third branch that preserves every unique commit, records duplicate patch skips, restores the stable version, and closes superseded sync PRs. Correct the next prerelease wording after a stable reset.
Require the old PR body and the full commit manifest to be carried into the replacement sync PR before the old PR is closed.
Contributor
Author
sadsfae
added a commit
that referenced
this pull request
Oct 28, 2026
Respond to review 5541395661 on PR #262: - Local OIDC issuer: prefer the discovery document's issuer, the value authlib pins the ID-token iss to, over the metadata-URL derivation, so the Local Login form and the callback key rows identically when a reverse proxy serves the document from a different path. Regression test where the two values differ. Rows created by earlier development builds under a divergent advertised issuer keep the old key and must be removed before that user signs in again. - Packaging: install /etc/qiip/oidc-users.txt comment-only so an unedited copy cannot enable a working account; the sample stays under %{_datadir} and the oidc.env hint no longer advertises change-me. The README says so and now names OIDC_BASE_URL as the public issuer. - Docs: the containerized nginx method cannot reach the loopback-bound local OIDC provider; README step 2 and the nginx.md troubleshooting table point at nginx on the provider host. - Frontend: JSON-encode USER_ID and make the jinja-in-script gate reject any script value that is not tojson or the poll expression (now rglob); fix the tokens and node-info skeleton cell counts. - Chat: bound the history that gets re-sent, newest turn always kept. - Leaderboard: set the board note before the empty-state return. - Release: wait for a stale build to finish and delete it only if it ended without publishing, and autolink artifact URLs in the notes.
sadsfae
added a commit
that referenced
this pull request
Oct 28, 2026
Respond to the re-review inline comments on PR #262: - store: wrap the same-issuer email refresh in the insert path's guard (rollback + AccountConflictError), so a provider moving a mutable email onto another row's address returns the conflict redirect/409 instead of a 500 with the SQLite writer lock held. Regression test asserts the conflict, the untouched row and no open transaction. The conflict copy (store message, 409 detail, profile page) now says "another account" because the colliding row can belong to the same provider. - local OIDC: require the ID token's aud to contain the configured client id, checked after iss. authlib's validate_azp only enforces azp == client_id when aud mismatches, so a token whose aud names another client passed. The explicit check is used rather than claims_options, which replaces authlib's issuer pin; a provider with a resource-style audience now needs an audience mapper (plugin docstring and example config say so).
sadsfae
added a commit
that referenced
this pull request
Oct 28, 2026
Respond to review 5541395661 on PR #262: - Local OIDC issuer: prefer the discovery document's issuer, the value authlib pins the ID-token iss to, over the metadata-URL derivation, so the Local Login form and the callback key rows identically when a reverse proxy serves the document from a different path. Regression test where the two values differ. Rows created by earlier development builds under a divergent advertised issuer keep the old key and must be removed before that user signs in again. - Packaging: install /etc/qiip/oidc-users.txt comment-only so an unedited copy cannot enable a working account; the sample stays under %{_datadir} and the oidc.env hint no longer advertises change-me. The README says so and now names OIDC_BASE_URL as the public issuer. - Docs: the containerized nginx method cannot reach the loopback-bound local OIDC provider; README step 2 and the nginx.md troubleshooting table point at nginx on the provider host. - Frontend: JSON-encode USER_ID and make the jinja-in-script gate reject any script value that is not tojson or the poll expression (now rglob); fix the tokens and node-info skeleton cell counts. - Chat: bound the history that gets re-sent, newest turn always kept. - Leaderboard: set the board note before the empty-state return. - Release: wait for a stale build to finish and delete it only if it ended without publishing, and autolink artifact URLs in the notes.
sadsfae
added a commit
that referenced
this pull request
Oct 28, 2026
Respond to the re-review inline comments on PR #262: - store: wrap the same-issuer email refresh in the insert path's guard (rollback + AccountConflictError), so a provider moving a mutable email onto another row's address returns the conflict redirect/409 instead of a 500 with the SQLite writer lock held. Regression test asserts the conflict, the untouched row and no open transaction. The conflict copy (store message, 409 detail, profile page) now says "another account" because the colliding row can belong to the same provider. - local OIDC: require the ID token's aud to contain the configured client id, checked after iss. authlib's validate_azp only enforces azp == client_id when aud mismatches, so a token whose aud names another client passed. The explicit check is used rather than claims_options, which replaces authlib's issuer pin; a provider with a resource-style audience now needs an audience mapper (plugin docstring and example config say so).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
QIIP POC Roll-up (release #2)
This pull request rolls the whole development train up to main: 74 merged development changes plus the reconciliation commit that makes this branch conflict free. It is deliberately left open for review by another maintainer; the author of the changes must not merge it.
What is in the roll-up
Why the reconcile commit is here
main and development both absorbed the same change with different SHAs: main through PR #216 (25c3f7d) and development through PR #215 (482fe78), and development later rewrote the same test file for the account cluster work (120ec7c). A merge of development into main therefore conflicted in one file, which is what GitHub reported as being unable to merge.
The reconcile commit merges main into development and resolves that file in development's favour: development's version already contains the retry that main added, in its later refactored form, so no behaviour is lost and the resulting tree is byte identical to development before the merge.
Verification of the reconcile
Commits in this roll-up
482fe78chore: absorb transient Chrome hangs in browser tests (chore: absorb transient Chrome hangs in browser tests #215), @sadsfae (Will Foster)71aada2feat: reuse compatible GPU drivers during setup (feat: reuse compatible GPU drivers during setup #218), @grafuls (Gonzalo Rafuls)0eabc710.2.0-dev.1, semantic-release (bot)c55c4cdchore: refresh uv.lock after version bump, @web-flow (qiip release)e35eae3Publish engine runtimes and setup bundles transactionally (Publish engine runtimes and setup bundles transactionally #219), @grafuls (Gonzalo Rafuls)2ebf719feat: reuse verified llama.cpp builds with bounded source fallback, @grafuls (Gonzalo Rafuls)03736a1fix: address llama.cpp artifact review findings, @grafuls (Gonzalo Rafuls)556680cfix: address native CPU and artifact runtime review, @grafuls (Gonzalo Rafuls)0402a240.2.0-dev.2, semantic-release (bot)d501b3fchore: refresh uv.lock after version bump, @web-flow (qiip release)b41d846feat: add user leaderboard and token dashboard, @sadsfae (Will Foster)ecd768b0.2.0-dev.3, semantic-release (bot)4f2d2efchore: refresh uv.lock after version bump, @web-flow (qiip release)6c66d25feat: auto-rediscover self-setup model drift in health cycle, @kambiz-aghaiepour (Kambiz Aghaiepour)5a85b380.2.0-dev.4, semantic-release (bot)69de6a4chore: refresh uv.lock after version bump, @web-flow (qiip release)9e5339cfeat: power on new nodes via redfish and resolve bmc creds per lab (feat: power on new nodes via redfish and resolve bmc creds per lab #228), @sadsfae (Will Foster)1c946c00.2.0-dev.5, semantic-release (bot)6d1eff9chore: refresh uv.lock after version bump, @web-flow (qiip release)435077afeat: opt-in local OIDC auth provider (RFE [RFE] Optional local OIDC auth provider for dev/testing and small installs #230), @kambiz-aghaiepour (Kambiz Aghaiepour)f41abf6style(tests): ruff format for local OIDC provider change, @kambiz-aghaiepour (Kambiz Aghaiepour)13b3906feat: Local Login merges the admin and local oauth sign-in paths, @kambiz-aghaiepour (Kambiz Aghaiepour)cbded89fix: address local OIDC provider review findings, @kambiz-aghaiepour (Kambiz Aghaiepour)02a133dfix: harden the issuer migration crash recovery, @kambiz-aghaiepour (Kambiz Aghaiepour)6894d5bdocs: warn that a global SSL_CERT_FILE replaces the trust store, @kambiz-aghaiepour (Kambiz Aghaiepour)cee2e76test: satisfy strict mypy for the new auth tests, @kambiz-aghaiepour (Kambiz Aghaiepour)db416100.2.0-dev.6, semantic-release (bot)6b04141chore: refresh uv.lock after version bump, @web-flow (qiip release)08842a6feat: wait for QUADS to finish releasing a host before placing on it, @sjug (Sebastian Jug)34fe664fix: require quads.stale_after_seconds to be at least twice poll_interval, @sjug (Sebastian Jug)e3c4ab80.2.0-dev.7, semantic-release (bot)cf8441achore: refresh uv.lock after version bump, @sadsfae (Will Foster)982deb9fix: enforce the stale bound only where placement runs, @sadsfae (Will Foster)f747f470.2.0-dev.8, semantic-release (bot)c5f43d2chore: refresh uv.lock after version bump, @web-flow (qiip release)a56b1f1feat: size automatic vLLM launches from available memory, @grafuls (Gonzalo Rafuls)2f3e0e2fix(vllm): address review on memory-aware sizing, @grafuls (Gonzalo Rafuls)66b65fefix(vllm): address the second review round on memory-aware sizing, @grafuls (Gonzalo Rafuls)e0c53b0test: widen timing-sensitive liveness timeouts, @grafuls (Gonzalo Rafuls)45faef2fix(vllm): keep the retried plan reason consistent with the effective context, @grafuls (Gonzalo Rafuls)a647d5e0.2.0-dev.9, semantic-release (bot)33ef768chore: refresh uv.lock after version bump, @web-flow (qiip release)9c16bdefix: price token estimates at Sonnet 5 rates, @sadsfae (Will Foster)09b38180.2.0-dev.10, semantic-release (bot)bf62aa7chore: refresh uv.lock after version bump, @web-flow (qiip release)e14cc11feat(provisioning): resume verified stages and retry transient failures, @grafuls (Gonzalo Rafuls)87d41b3fix: document new setup settings and satisfy strict typing, @grafuls (Gonzalo Rafuls)337837efix(provisioning): address setup resume review feedback, @grafuls (Gonzalo Rafuls)dc93cd4fix(provisioning): address second review round, @grafuls (Gonzalo Rafuls)196add1fix(provisioning): retry bare 5xx download failures, @grafuls (Gonzalo Rafuls)88c60910.2.0-dev.11, semantic-release (bot)abcf1f4chore: refresh uv.lock after version bump, @web-flow (qiip release)3b7fcc4chore: stop diagnostics tests flaking under load, @sadsfae (Will Foster)1473fdffeat: sort the token and usage tables by column, @sadsfae (Will Foster)93620500.2.0-dev.12, semantic-release (bot)564ff60chore: refresh uv.lock after version bump, @web-flow (qiip release)d1bcb53feat: copy the user's own token from the pages, @sadsfae (Will Foster)8ef4a190.2.0-dev.13, semantic-release (bot)a1328e8chore: refresh uv.lock after version bump, @web-flow (qiip release)120ec7cfix: unify the account cluster across the top bars, @sadsfae (Will Foster)28754ad0.2.0-dev.14, semantic-release (bot)554942cchore: refresh uv.lock after version bump, @web-flow (qiip release)56b8c40fix: run the chat playground on the user's token, @sadsfae (Will Foster)b511d280.2.0-dev.15, semantic-release (bot)2c9e354chore: refresh uv.lock after version bump, @web-flow (qiip release)a15b4c7fix: guard Copr publishes against duplicate NVRs, @sadsfae (Will Foster)0f9fb850.2.0-dev.16, semantic-release (bot)42d7896chore: refresh uv.lock after version bump, @web-flow (qiip release)814ac1dfeat: link the release RPM packages in the release notes, @sadsfae (Will Foster)37655ef0.2.0-dev.17, semantic-release (bot)5c5a405chore: refresh uv.lock after version bump, @web-flow (qiip release)0df4381fix: tell a Copr outage apart from missing dependency RPMs, @sadsfae (Will Foster)6f4060f0.2.0-dev.18, semantic-release (bot)14c1f87chore: refresh uv.lock after version bump, @web-flow (qiip release)For the reviewer
main requires the Quality and Python 3.13 checks, and admins cannot bypass them. This branch is expected to be green before it is merged; a rebase merge or a squash merge both work, a merge commit is not enabled on this repository.
Merging this roll-up is the maintainer's call and it finalizes the 0.2.0 stable release flow, which then resets development onto main.