Skip to content

Add backend-local ephemeral session staging and cleanup - #1968

Draft
oznetmaster wants to merge 3 commits into
nunit:feature/remote-agentsfrom
oznetmaster:feature/session-deployment-20261115
Draft

oznetmaster wants to merge 3 commits into
nunit:feature/remote-agentsfrom
oznetmaster:feature/session-deployment-20261115

Conversation

@oznetmaster

@oznetmaster oznetmaster commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

For #167 / #1925, this draft adds backend-local ephemeral session deployment as an internal signed net10.0 library and coordinator. It stages a host, standard launcher bundles and a test package for each session; no resident server installation is required. The coordinator references the engine, and the production engine has no deployment dependency.

The base is feature/remote-agents at e667566, where standalone server #1962 is merged. This comparison contains the session implementation and its test-integration follow-ups; old stacked server commits are absent.

The existing modern nunit.engine.tests assembly contains 44 session cases: 34 file/staging unit cases, two Windows-only nonparallel Integration cases, and eight source-built lifecycle Acceptance cases. The bounded fault probe supplies readiness/EOF/exit signals without listening on a port or executing a test package. The normal solution builds the session library and probe with explicit Release configuration. The Framework test target excludes the session fixtures and references. No new test project is added, and the coordinator is not added to shipping packages. Test follow-ups change no production execution code, dependency pins, workflow files or release gates.

This slice expects backend-local host, launcher and test-package directories. SSH/SFTP transfer, remote bootstrap, runtime installation, hardware admission and authenticated WAN exposure remain outside this draft. Console execution selection and shipping package definitions are unchanged.

The coordinator supports one-shot execution, sequenced stdio events/cancellation and bounded hosting for an external frontend. Test outcomes remain separate from hosting/cleanup outcomes. Cleanup requires matching ownership, confirmed host shutdown, ordinary-tree validation and an independent native-process audit. Uncertain state retains payload and evidence; Dispose alone does not delete a deployment.

Validation:

  • Windows Release solution builds pass with zero warnings/errors under the repository's C# 13, signing and analyzer rules. Full suites pass 612 cases on net10.0 and 529 on net462, with zero failures and the same 20/12 existing skips. Pre-existing per-case outcomes are unchanged. The newest eight cases pass separately; a deliberate cleanup defect makes the corresponding regression fail.
  • Public-head 642edeb hosted CI 43817560324 passed. Hosted CI for this new source update is a separate result and must not be inferred from that earlier run.
  • Native macOS ARM64 and Linux aarch64 each passed the earlier 34 file-only cases using the same Windows-built managed closure. Those runs do not establish native source-build, Unix process-audit or real-agent coverage for the eight new cases.
  • Separate local Windows adapter acceptance passes 30 outer cases, including 14 real-agent sessions and 14 finite protocol controls. Two nested local controls pass through released NUnit3TestAdapter 6.3.1. The remote path uses an experimental workflow-adapter shim, not stock-adapter remote integration. All 14 real sessions confirm cleanup and 363 frozen inputs remain unchanged. This fixture remains local; it is not an upstream CI target.
  • The older 77-case private harness remains separate historical evidence. These source-built acceptance cases do not replace its entire scope. Earlier failed runs are retained honestly.

Integration and test ownership remain open for Neil's discussion with Charlie. Logical responsibilities do not require separate processes; coordinator placement and optional frontend-neutral lifecycle reuse are design questions, not cross-project policy decisions. This stays a draft. No merge or readiness transition is requested.

Prepared jointly by Neil Colvin and OpenAI Codex.

@CharliePoole

Copy link
Copy Markdown
Member

@oznetmaster Let's discuss the options for integration in our next chat.

Copy link
Copy Markdown
Collaborator Author

@CharliePoole Agreed. Keep draft for the chat. We can discuss the separate session library and coordinator, keeping deployment dependencies out of the engine, and which targets should own the tests. I also found the CI header failure; I’ll fix it before marking this ready, separately from the placement decision.

Prepared by Neil Colvin with assistance from OpenAI Codex.

Keep deployment in a separate signed library and coordinator solution on feature/remote-agents. Use repository file headers and describe the merged standalone server base. Integration and CI test ownership remain open.

Co-authored-by: OpenAI Codex <codex@openai.com>
@oznetmaster
oznetmaster force-pushed the feature/session-deployment-20261115 branch from 602fd99 to 4a827e1 Compare November 16, 2026 05:13

Copy link
Copy Markdown
Collaborator Author

@CharliePoole Corrected all seven session C# headers and refreshed this draft onto feature/remote-agents after #1962 merged. The PR now contains one session-only commit, 4a827e1, adding 12 files; the old stacked server commits and stale hold text are removed from the comparison/description. README.txt also names the merged server base.

The actual CheckHeaders target passes locally and fresh hosted CI 43808816609 is green. The signed session build has zero warnings/errors, and the separate 77-case private harness passes all cases against the corrected projects. Comparing the seven runtime files with the previous draft confirms that only their headers changed; integration and upstream test ownership are still open.

This stays a draft until Neil has talked to you. No merge or readiness transition has been performed. The process/lifecycle simplification refinement is also available on discussion #1960, including the distinction between embedded runtime and dynamic deployment.

Prepared by Neil Colvin with assistance from OpenAI Codex.

…ngine tests

Keep the Framework test target independent of the net10.0 session library.
Include the existing session library in the solution so Release configuration
is assigned during the normal solution build. Add no new test project or
production execution dependency. Keep this candidate local for integration review.

Co-authored-by: OpenAI Codex <codex@openai.com>

Copy link
Copy Markdown
Collaborator Author

@CharliePoole Validation follow-up for 642edeb3:

The current hosted CI run passed headers, build, tests and packaging. I also executed the 34 file-only SessionFileTests cases on both native platforms using the same frozen managed dependency closure:

Platform Architecture CLR Passed Failed Skipped
macOS 27.0.1 ARM64 10.0.13 34 0 0
Linux 6.8 / glibc 2.39 aarch64 10.0.13 34 0 0

These are native executions of Windows-built managed artifacts, not native source builds. The selected fixture exercises filesystem/staging behavior without launching NUnit agents, a backend server or processor tests. The two Windows-only native-process Integration cases and the broader private acceptance harness remain separate coverage; this adds no Unix process-audit or real-agent validation claim.

Both runs retained unchanged payload hashes and removed their test scratch directories. Existing gates were used and released; the Linux VM is stopped. The Linux run checked owned process references before shutdown, and a separate final readback confirmed the Mac gate absent, VM stopped and no owned Mac test processes. Result XML, payload hashes and cleanup receipts are retained locally. No runtime installation or source change was needed.

This remains a draft pending Neil's discussion with you. No merge or readiness transition was performed.

Prepared jointly by Neil Colvin and OpenAI Codex.

Build a bounded test-only host from source and use it in the existing modern
engine test assembly. Preserve missing-readiness, failed-exit, deadline,
retention, recovery and primary-error contracts without private fixture paths.
Keep the Framework host and all production execution source independent.

Co-authored-by: OpenAI Codex <codex@openai.com>

Copy link
Copy Markdown
Collaborator Author

@CharliePoole Further acceptance coverage is ready in this draft, without changing production execution code or resolving the integration-placement question.

Eight lifecycle Acceptance cases now use a small source-built bounded fault host in the existing engine test assembly. They cover missing readiness, unsuccessful shutdown, deadlines, disposal, evidence-write failure and idempotent cleanup. The probe has no listener or installed-agent dependency. All eight pass; deliberately allowing cleanup after a nonzero host exit makes the corresponding case fail. Complete Windows suites pass 612 cases on net10.0 and 529 on net462, with the same 20/12 existing skips and unchanged pre-existing case outcomes.

Separately, a local reproducible workflow-adapter fixture now builds the coordinator/server from this source and restores pinned agent packages rather than requiring previous private frozen inputs. All 30 outer cases pass on Windows: 14 real-agent sessions, 14 finite protocol controls, discovery and a released-adapter local control. That last control runs two ordinary local cases with unmodified NUnit3TestAdapter 6.3.1; it is not a claim of stock-adapter remote support. All 14 real sessions confirm cleanup, and all 363 frozen input files remain unchanged.

Earlier failures are retained. The old .NET 10 agent alpha.0.1 fails the cancellation/deadline controls; available alpha.0.3 passes them. Generated short work directories avoid a separately reproduced Windows process working-directory path limit. No production workaround was added.

These new results are Windows-only. The earlier native Mac/Linux file-only validation remains separate. The adapter fixture is still local; #1968 stays a draft while we discuss integration. No merge or readiness transition is requested.

Prepared jointly by Neil Colvin and OpenAI Codex.

Copy link
Copy Markdown
Collaborator Author

@CharliePoole Fresh hosted CI for d4a0cd0 passed, including 612 modern and 529 Framework engine cases, with zero failures and the same 20/12 existing skips.

The separate reproducible workflow-adapter fixture now also builds from source on native macOS ARM64 and Linux aarch64. Both cloned exact public engine commit d4a0cd0 and restored pinned packages into dedicated fresh caches. Each passes all 30 outer cases, with zero failures/skips, including VSTest, dotnet test, exact filtering, per-test results/events, cancellation, deadlines and finite protocol controls. Each released-adapter local control runs two additional passing ordinary local cases.

The first Mac run retained four inherited Windows-only skips. Removing that fixture restriction let all four dotnet-test controls pass on both Unix platforms. The final fixture also passes 30/30 on Windows. No production engine change was required; fixture portability changes remain local.

All 14 real sessions on each platform confirm cleanup and native process audit. The native input closures remain unchanged, and the three downloaded agent package hashes match Windows. Worker output confirms actual .NET 8/9/10 execution. Existing gates were used and released; the Linux VM is stopped, no owned test processes remain, and evidence is retained.

These are native source-build/local-backend results. They do not add cross-machine, WAN, processor or IDE-UI claims, and the released NUnit3TestAdapter control establishes ordinary local execution only. The separate eight new engine lifecycle Acceptance cases retain their Windows/hosted-CI evidence; they were not part of these native fixture runs.

#1968 remains a draft, integration placement is still open, and nothing has been merged.

Prepared jointly by Neil Colvin and OpenAI Codex.

Copy link
Copy Markdown
Collaborator Author

@CharliePoole The eight lifecycle Acceptance cases added at d4a0cd0 now also pass natively on macOS ARM64 and Linux aarch64: eight passed on each, zero failures/skips. Both built the exact public source and bounded fault host locally. This is separate from the earlier 30-case adapter fixture.

The cases preserve expected failed/retained states rather than claiming cleanup after uncertain shutdown. Existing gates were used and released; the Linux VM is stopped and independent native audits found no owned test processes. Test/evidence bytes are retained. No assertion, engine source or dependency pin needed changing.

An initial MyGet download failure succeeded on an unchanged retry. The Mac collector also encountered the intentional directory named session.json in the evidence-write failure case; its original collector error is preserved, and a separate read-only audit confirms the passing XML and all dispositions. The corrected collector handled Linux. These are recorded helper/restore issues, not hidden test failures or waivers.

#1968 remains a draft. No merge or integration-placement decision is requested.

Prepared jointly by Neil Colvin and OpenAI Codex.

Copy link
Copy Markdown
Collaborator Author

@CharliePoole Further validation is complete against public #1968 head d4a0cd0 using a separate SSH/SFTP prototype and pinned standard agent packages.

  • Windows client to Windows backend: all four expected outcomes passed—ordinary success, intentional assertion failure, cooperative cancellation and client deadline.
  • macOS client to Linux backend: those four outcomes plus explicit .NET 9 execution passed, using the existing SSH private key.
  • Two real Windows-to-Mac disconnect checks dropped only the validation session's SSH/SFTP connections after observed test progress. Each original run remained an infrastructure failure with cleanup unconfirmed. A separate explicit recovery verified durable shutdown, unchanged inputs and native process audit before removal and gate release. Original failure evidence remained unchanged.

The source-only prototype review bundle also reproduces from a fresh extraction and dedicated package cache: signed Release build and 75 offline cases pass, with no failures/skips. A new regression demonstrates and corrects a secondary stop-channel error masking the earlier operation error.

Earlier failures remain preserved. An older Linux host-key record was rejected before a test started; the current key was independently verified through the pinned Mac and established VM identity before a fresh attempt. A collection-helper issue was completed through a separate audited continuation without rerunning tests.

Independent final audits confirm owned test deployments/processes absent, gates released and the Linux VM stopped. These are local validation results, not new upstream CI coverage or a WAN/desktop-automation claim.

The attached bundle contains the proposed SSH source, patch, build/test scripts and evidence. These files have not been added to this PR, and the suggested experiment location remains for review. This does not settle integration placement; #1968 remains a draft.

SSH session review bundle

SHA-256: 36c4baa129832ae370f048382a5ef4250ae51922712abe96dce89f3efffa730c

Prepared jointly by Neil Colvin and OpenAI Codex.

Copy link
Copy Markdown
Collaborator Author

@CharliePoole A further Windows validation pass is complete using a separate local integration on main at b5de455, with NUnit.Engine.Api 4.0.0-beta.101.4 and the merged #1971 callback correction preserved. This combines the remote provider/server work with this PR's session implementation; it does not change the public PR branch.

  • The full Release solution builds with zero warnings/errors.
  • Required engine suites: net462 has 603 passed, zero failed and eight existing skips; net10.0 has 639 passed, zero failed and eight existing skips. All pre-existing per-case outcomes match the unmodified-main receipts.
  • The eight session Acceptance cases and 75 standalone SSH offline cases pass, with zero failures/skips.
  • Both ordinary Console entry points—the Framework console and modern .NET console—pass the exact local test selection using the unchanged published .NET 10 launcher package, without remote settings.

The initial Console controls lacked a discoverable extension installation. A loose root-level .addins file was also insufficient for the current assembly-based discovery path. Supplying the published NUnit.Extension.*/.../tools package layout resolved discovery; no engine correction was needed. Those failed setup results are retained.

I also exercised an actual coordinator-process crash, after backend-host readiness but before acquiring an agent or starting a test. Only the coordinator created by the validation harness was terminated. The backend host then exited with code zero, confirmed through a retained process handle with matching PID/start time. The durable session receipt remained Running, and no completed hosting.json existed.

The unchanged recovery implementation correctly refused cleanup, made zero destructive calls and retained the payload/owner. A separate manual reconciliation then verified host exit, both owner identities, all 32 payload files, an ordinary directory tree and an independent native-process audit before removal. The original interrupted-session evidence remains unchanged; no successful hosting receipt was synthesized.

That demonstrates conservative handling of this pre-agent crash, not automatic crash recovery or behavior during an active test. This new integration has only been validated on Windows; the earlier macOS/Linux and adapter/GUI results remain tied to their previously recorded source versions. No hardware or remote validation machine was contacted in this increment.

All 123 frozen SSH inputs and the previously uploaded review ZIP retain their hashes. The integration source and crash harness remain local. #1968 stays a draft at d4a0cd0; no source push or merge has been made, and no merge will be made without your approval.

Prepared jointly by Neil Colvin and OpenAI Codex.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants