Skip to content

Update to latest script version; verify dependency versions - #1944

Merged
CharliePoole merged 4 commits into
mainfrom
issue-1943a
Nov 4, 2026
Merged

CharliePoole merged 4 commits into
mainfrom
issue-1943a

Conversation

@CharliePoole

@CharliePoole CharliePoole commented Nov 1, 2026 •

Copy link
Copy Markdown
Member

Step 2 of issue #1943

Verification determines whether the correct expected version is installed. Consistency among various files is not checked. This will be done as a separate step.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Eager NuGet version resolution during package definition can block unrelated targets such as Clean during a feed outage, even when tools are cached.

1 open issue
What changed in this PR

This PR advances #1943 by updating the Cake recipe and adding latest-version dependency checks for NUnit packages and bundled agents.

Changes:

  • Upgrade NUnit.Cake.Recipe to 2.0.0-beta.5.2.
  • Check dependency versions and expand console package content validation.
  • Make bundled-agent references readonly and display recipe version badges.
File Description
README.md Add NuGet and MyGet version badges for NUnit.Cake.Recipe.
KnownExtensions.cake Make agent fields and the bundled-agent array readonly; document version defaults.
build.cake Upgrade the recipe and strengthen dependency and package-content checks.

Review effort: Balanced

Comment thread build.cake
HasDirectory("lib/net462").WithFile("nunit.common.dll"),
HasDirectory("lib/netstandard2.0").WithFile("nunit.common.dll"),
HasDependency("NUnit.Engine.Api")
HasDependency("NUnit.Engine.Api").LatestDevBuild

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot Show me a brief example of code to defer version resolution until DependencyCheck.ApplyTo runs

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For example, DependencyCheck can store a resolver instead of calling NuGet in the selector getter:

private Func<string> _versionResolver;

public DependencyCheck LatestDevBuild
{
    get
    {
        _versionResolver = () => new PackageReference(_packageId).LatestDevBuild.Version;
        return this;
    }
}

public override bool ApplyTo(DirectoryPath testDirPath)
{
    _packageVersion ??= _versionResolver?.Invoke();
    // existing check logic...
}

LatestStableRelease would set the same resolver using PackageReference(...).LatestStableRelease.Version. This keeps the existing call site, but performs feed access only when ApplyTo runs.

@oznetmaster oznetmaster left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@CharliePoole, I checked the published NUnit.Cake.Recipe 2.0.0-beta.5.3 payload, including the deferred DependencyCheck correction. That part now defers correctly, but I found three remaining points:

  1. The agent checks in build.cake still resolve during package definition. ExtensionSpecifier.LatestStableRelease calls PackageReference.LatestStableRelease immediately. Use the deferred DependencyCheck selector instead, for example:
HasDependency(KnownExtensions.Net462PluggableAgent.NuGetPackage).LatestStableRelease

The same change applies to the other three agent checks. The current four expressions make four feed calls before any verification runs, so the earlier concern about unrelated targets depending on feed availability is only partly resolved.

  1. In the recipe, PackageReference.LatestReleaseIncludingPreReleases uses DEVELOPMENT_SOURCE, making it equivalent to LatestDevBuild. Based on the policy described in #1943, it should query PRODUCTION_SOURCE with prereleases enabled. This selector is not used by the new console call sites, but it is incorrect in the recipe API supplied by this update.

  2. If a configured latest-version lookup finds no matching package, its version stays null. DependencyCheck.ApplyTo then treats the check as intentionally unversioned and can accept a single older installed version. An explicitly requested latest policy should fail with a resolution diagnostic in this case; an ordinary HasDependency with no version should retain its existing existence-check behavior.

Validation: an offline harness compiling the published recipe classes against fake feed/filesystem boundaries reproduced all three issues. Three cases failed before the isolated corrections; all six cases passed afterward, including controls for deferred verification and intentionally unversioned dependencies. This was not a full Cake invocation or a real feed-outage test. No maintained source or published package was modified.

Prepared by Neil Colvin with assistance from OpenAI Codex.

@CharliePoole

CharliePoole commented Nov 1, 2026 •

Copy link
Copy Markdown
Member Author

@oznetmaster
Good finds! I think that we'd do better making some of the changes through a PR in NUnit.Cake.Recipe, so we can comment on source line changes.

  1. I'll make that change in build.cake. The code in ExtensionSpecifier will be fixed in the recipe PR.

  2. Not used in build.cake as you point out. Deferring this to fix in the recipe PR.

  3. Deferring this to the recipe PR as well.

This PR will be marked as a draft until the recipe work is complete. Delaying this PR shouldn't cause any serious merge problems, since the remaining changes will be limited to the build.cake and package-tests.cake.

I like the idea of a validation harness for recipe changes. That would keep us from having to update myget in order to actually test. Let's discuss making that permanent.

Next step after I convert this to a draft: an issue or issues in NUnit.Cake.Recipe repo.

@oznetmaster

Copy link
Copy Markdown
Collaborator

@CharliePoole, I verified the updated agent call sites now use the deferred DependencyCheck selectors. That addresses the console-side portion of the first finding.

Moving the remaining fixes into recipe PRs makes sense. For the permanent harness, I suggest compiling the maintained recipe sources directly, with controlled feed/filesystem boundaries, so regressions can be tested without first publishing to MyGet. We should retain a separate Cake smoke test rather than treating the isolated harness as complete integration coverage.

Prepared by Neil Colvin with assistance from OpenAI Codex.

@CharliePoole

Copy link
Copy Markdown
Member Author

@oznetmaster If I expected future conflicts, I'd merge this now, but otherwise I see no need. We'll wait for the fixes in the recipe and then apply them here as needed.

Copy link
Copy Markdown
Collaborator

@CharliePoole, the recipe follow-up is now validated against the published NUnit.Cake.Recipe 2.0.0-beta.5.8 package following merged recipe PRs #135 and #136.

I checked the downloaded package against recipe main d546405: all 29 recipe source files match. The maintained harness, loading the public package instead of local recipe source, passes all 104 cases.

In an isolated checkout of #1944 at 9f4d385, I changed only the #load version from beta.5.3 to beta.5.8. The real Package target builds all four packages, installs them and passes all six package/symbol verification stages, including the deferred API/extensibility and stable-agent dependency checks. The original three review findings are addressed by your updated call sites and the recipe corrections.

For completeness, the broader execution results are qualified: all 52 Framework-console package scenarios pass. The modern console passes 37 of 52, with 15 failures. Repeating both console suites with the original beta.5.3 recipe and exactly the same built packages gives identical per-scenario outcomes. These failures therefore reproduce independently of adopting beta.5.8; I am not claiming a fully passing modern-console package suite. The failure evidence includes Framework-agent launch-path errors and agent compatibility rejection for the .NET 7 cases; those need separate investigation before assigning causes or proposing fixes.

The minimal follow-up here is updating the recipe #load to 2.0.0-beta.5.8. No engine, launcher, package checks or test expectations were changed for this validation. Publication was disabled and the local feed was confined to the isolated validation directory. Native Linux/macOS validation is not claimed.

Prepared by Neil Colvin with assistance from OpenAI Codex.

Copy link
Copy Markdown
Collaborator

@CharliePoole, the local-execution failures described in my earlier recipe validation now have separate, tested corrections: Framework-agent #15 fixes the device-prefixed executable paths, and console/engine #1947 makes launcher metadata selection use the resolved execution runtime. Both draft PRs have passing hosted CI and CLA checks.

The combined isolated #1944 validation passes all 52 Framework-console scenarios. Its modern-console suite passes 49/52 with this workstation's normal runtime catalogue; the three remaining comparisons expect older TargetFrameworkName values. All 52 pass with the expected legacy catalogue entries supplied privately, while the actual workers for those legacy-target cases remain .NET 8. No package assertion was weakened or global runtime installation changed. The existing comparison reads TargetFrameworkName, so I am keeping selected-target reporting separate from actual worker-CLR coverage.

This does not change the recipe conclusion: beta.5.8 passed the package and dependency checks, and those earlier execution failures reproduced with beta.5.3 as well. The recipe-pin follow-up can remain separate from the two execution corrections.

Prepared by Neil Colvin with assistance from OpenAI Codex.

@CharliePoole

Copy link
Copy Markdown
Member Author

@oznetmaster Are you still requesting changes in this PR as indicated by GitHub? I can override and merge it anyway if not.

@oznetmaster

Copy link
Copy Markdown
Collaborator

@CharliePoole, your call-site changes address the console-side finding. The remaining recipe corrections are published in NUnit.Cake.Recipe 2.0.0-beta.5.8, but this PR still loads beta.5.3.

Please update that pin to beta.5.8. We have already validated that combination in an isolated copy, including package and dependency verification. Once the updated head passes CI, I can replace the changes-requested review with approval; there should be no need to override it.

Prepared by Neil Colvin with assistance from OpenAI Codex.

@CharliePoole

Copy link
Copy Markdown
Member Author

OK... the status just was not clear.

@CharliePoole
CharliePoole marked this pull request as ready for review November 4, 2026 02:12
@CharliePoole
CharliePoole merged commit 2ed470b into main Nov 4, 2026
2 checks passed
@CharliePoole
CharliePoole deleted the issue-1943a branch November 4, 2026 02:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants