Skip to content

deploy: fleet-ctl.sh applies a site's compose override - #77

Closed
eigmax wants to merge 1 commit into
mainfrom
deploy/compose-site-override
Closed

eigmax wants to merge 1 commit into
mainfrom
deploy/compose-site-override

Conversation

@eigmax

@eigmax eigmax commented Nov 4, 2026

Copy link
Copy Markdown
Member

Summary

deploy/fleet-ctl.sh applies a site's compose override. A site keeps its changes to the release's docker-compose.yml in deploy/docker-compose.override.yml, such as another gateway or an existing Postgres volume. A scheduler upgrade replaces only the release's file and leaves the override in place.

Until now every compose call passed only -f deploy/docker-compose.yml, so an upgrade or rollback recreated services from the release's file alone. The script now builds its compose command once and adds the override after the release's file when the override exists. Upgrades, rollbacks and status all use that command. Without an override, the command is the same as before.

The production control host needs this: it runs nginx with the public certificate and the existing Postgres volume through such an override.

Changes

  • deploy/fleet-ctl.sh: one COMPOSE command, with the override added when present, used by compose() and status.
  • deploy/tests/test_fleet_ctl.py: status with stand-in docker, curl and fleetctl passes only the release's file, or the release's file followed by the override.
  • docs/maintenance.md: where a site's compose changes belong.

Testing

  • python -m pytest -q bridge/tests deploy/tests passes (77 tests). The override test fails without the change.
  • fleet-ctl.sh status against the production control host lists all four services of the zkm-fleet compose project.

A site's changes to the release's docker-compose.yml (another gateway, an existing
Postgres volume) live in deploy/docker-compose.override.yml, which an upgrade does not
replace. Every compose call, status included, now applies it after the release's file;
before, an upgrade recreated services from the release's file alone.
eigmax added a commit that referenced this pull request Nov 4, 2026
#77 made the same change: one COMPOSE command, the site's override after
the release's file when present, used by every compose call and status.
Its tests check the exact command lines.
@eigmax

eigmax commented Nov 4, 2026

Copy link
Copy Markdown
Member Author

Folded into #78, which now carries this change (the same COMPOSE command and tests) together with the rest of the fixes. Closing in favor of #78.

@eigmax eigmax closed this Nov 4, 2026
@eigmax
eigmax deleted the deploy/compose-site-override branch November 4, 2026 10:40
eigmax added a commit that referenced this pull request Nov 4, 2026
…ay ports, fleet-ctl overrides (#78)

* scheduler: setting an app's disabled state it already has changes nothing

A starting prover's state is mirrored onto its proof app on every heartbeat and
scale pass. Each call wrote the store and logged "app setting changed", and on
Postgres moved disabled_ms to now, so the app's disabled-since time kept resetting.

* deploy: let a node move Ray's metrics and worker ports

ray-start.sh fixed the metrics port at 8080 and worker ports at 10002-19999. On
ant-5090-1 another service holds 8080 and a Bitcoin node 18332/18333.
RAY_METRICS_EXPORT_PORT and RAY_MIN/MAX_WORKER_PORT in watchdog.env move them;
the defaults are unchanged.

* deploy: fleet-ctl.sh loads a site's compose override

A site's own Compose settings (here: nginx with the public certificate instead of
Envoy, an external Postgres volume) live in deploy/docker-compose.override.yml.
fleet-ctl.sh loaded only the release's file, so an upgrade or status through it
would have run the site without them. An upgrade still replaces only the release's
file.

* scheduler: move an app that waits for room to a node that has it

An app the scheduler placed stayed on its first node for good: a start that
did not fit there waited, however much room other nodes had. Now the
scheduler sends a node with room the app's entry as the old node has it
(disabled, its machine set to the new node's); once that node reports it,
the old node removes its entry and the app starts on the new one.

A node refuses an entry whose host files it lacks (its worker program,
environment file, storage root and volume sources) and is not asked again;
the app keeps waiting where it is.

Also fixes enabling an app whose create a node refused: the scheduler held
its state lock while taking it again and stopped.

* scheduler: an app that moves takes care of its own storage

A node no longer checks an entry's host files before creating it: what an
app keeps on a node's disks is the app's own, and the scheduler copies
nothing of it. A node whose create fails, for any reason, is still not
asked again for that app.

* deploy: fleet-ctl.sh builds its compose command once (from #77)

#77 made the same change: one COMPOSE command, the site's override after
the release's file when present, used by every compose call and status.
Its tests check the exact command lines.

* bridge: removing an app keeps its data

A moved app's old node removes its entry: its configuration and what is
left of its containers and processes. The app's storage root, workdir, env
file and volume sources stay, since the app may keep its data on storage
the new node shares. A test now holds this.

* scheduler: name the move's steps for what they do

relocate becomes move_waiting_app and finish_move becomes hand_over; the
test's play_nodes becomes nodes_run_commands, beside node_writes_config.

* scheduler: a queued app start is placed where there is room

There is no moving: a start of an app the scheduler placed is placed when it
leaves the queue, as place_app places a new app. Its own node when that has
the room, else the node that fits it best now; with no room anywhere it
preempts on its own node and keeps the room that frees, even if another node
has room by then. move_waiting_app becomes place_start, hand_over start_on.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant