Skip to content

Allow API clients to refresh Dag bundles - #72715

Open
fat-catTW wants to merge 9 commits into
apache:mainfrom
fat-catTW:add-dag-bundle-refresh-api
Open

fat-catTW wants to merge 9 commits into
apache:mainfrom
fat-catTW:add-dag-bundle-refresh-api

Conversation

@fat-catTW

@fat-catTW fat-catTW commented Sep 8, 2026 •

Copy link
Copy Markdown
Contributor

Why

Dag bundle updates currently depend on the configured refresh interval or a Dag processor restart. This can delay the discovery of newly deployed Dags and makes deployment automation less predictable.

In deployments with multiple Dag processors, a refresh request must reach every processor. An ephemeral or single-consumer signal could refresh only one processor while others continue using stale local bundle contents.

closes: #60616

Solution

Add an asynchronous REST endpoint for requesting a Dag bundle refresh:

POST /api/v2/dagBundles/{bundle_name}/refresh

Each request atomically increments a persistent refresh generation on the bundle. Every Dag processor tracks the latest generation it has processed and refreshes the bundle when its local value differs from the persisted generation. This allows every processor to observe the request independently.

The endpoint returns 202 Accepted once the request is persisted. It does not guarantee that bundle discovery or Dag parsing has completed. Multiple requests may be coalesced when a processor observes the latest generation.

Refresh requests require Dag edit authorization. Team-owned bundles use the owning team's scope, while global bundles require global Dag edit access. Audit events for team-owned bundle refreshes are attributed to that team.

The OpenAPI specification, generated clients, migration metadata, and API permission documentation are kept in sync. The generated TypeScript client changes do not add or alter any visible UI.

Testing

  • Added API response, authorization, and team audit attribution tests.
  • Added Dag processor generation tests, including two independent processors
    observing the same generation.
  • Added API permission extractor coverage.
  • Verified OpenAPI, generated clients, and database migration checks.
  • Ran Ruff, mypy, and the staged-file pre-commit suite.
  • Ran focused tests: 172 passed and 5 PostgreSQL-only tests skipped.
  • Ran permission extractor tests: 70 passed.
  • Ran follow-up audit and multi-processor tests in Breeze: 2 passed.
Was generative AI tooling used to co-author this PR?
  • [X] Yes (please specify the tool below)

Generated-by: [Codex] following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@fat-catTW

Copy link
Copy Markdown
Contributor Author

I have updated the implementation to reuse DagPriorityParsingRequest instead of adding refresh-generation state to DagBundleModel.

The updated flow is:

  • POST /api/v2/dagBundles/{bundle_name}/refresh persists a DagPriorityParsingRequest with relative_fileloc=None.
  • relative_fileloc=None represents a bundle-wide refresh, while a non-null value retains the existing file-level priority parsing behavior.
  • When a Dag processor claims the request, it marks the bundle for a forced refresh, bypasses the normal refresh interval, and consumes the request.
  • The endpoint returns 202 Accepted once the request is persisted. It does not indicate that the refresh or Dag parsing has completed.

With multiple Dag processors, only one processor claims the request. For versioned bundles, other processors detect the updated database version and refresh through the existing mechanism. For non-versioned bundles, they cannot detect the update through a version change and will refresh at their normal interval.

This PR preserves the existing non-versioned behavior. Please let me know if this API should instead trigger an immediate refresh on every Dag processor.

@fat-catTW
fat-catTW marked this pull request as ready for review October 1, 2026 13:28
@fat-catTW
fat-catTW force-pushed the add-dag-bundle-refresh-api branch from 5a69311 to eea30f8 Compare October 2, 2026 01:24
@fat-catTW
fat-catTW force-pushed the add-dag-bundle-refresh-api branch from 626d511 to 709ca41 Compare October 11, 2026 13:09
SQLite enforces foreign keys while Alembic rebuilds parent tables during column changes. Disable those checks around the Dag bundle batch operation so upgrades and downgrades complete safely.
This keeps bundle refresh signaling aligned with the existing Dag parsing request lifecycle and avoids a second coordination mechanism.
@fat-catTW
fat-catTW force-pushed the add-dag-bundle-refresh-api branch from 06154f8 to 05b81ee Compare October 16, 2026 06:41

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add REST API to Refresh / Reload DAG Bundle Without Restart

3 participants