Skip to content

03_scripted_tests.md

Latest commit

 

History

History
177 lines (140 loc) · 10.1 KB

03_scripted_tests.md

File metadata and controls

177 lines (140 loc) · 10.1 KB
Raw

Scripted tests

End-to-end incremental-compilation scenarios. Each test drives a real compile, edits sources, and recompiles, asserting on what Zinc decided to recompile. The engine is a custom re-implementation in internal/zinc-scripted, not sbt's scripted plugin.

Layout

zinc/src/sbt-test/<group>/<name>/
  build.json               # optional; project list for multi-project tests
  *.scala, *.java          # the sources being compiled
  changes/                 # edited versions, copied over the originals mid-test
  incOptions.properties    # optional; see below
  test                     # the script of steps

Naming the script pending instead of test marks the scenario as known-failing: it still runs, but a failure is tolerated, and passing is what fails the build (a reminder to rename it back). A pending file takes precedence over a test file in the same directory.

Groups: source-dependencies, apiinfo, macros, pipelining, profiler, reporter, general.

Without a build.json the test is a single project named root rooted at the test directory. With one, each entry declares a name and optional dependsOn, in, and scalaVersion. A project's base directory is in when given, otherwise the subdirectory named after the project (IncHandler.scala:124).

scalaVersion takes a label rather than a patch version, so tests need no edits when the build moves to a new Scala patch release. The label picks both the compiler (versions from project/Dependencies.scala) and its bridge:

Label Compiler Bridge
2.10.x, 2.11.x, 2.12.x, 2.13.x scala210 ... scala213 the compiler-bridge sources in this repository
2.13.y scala213ForBridge scala2-sbt-bridge, which ships with Scala 2.13
3.x scala3ForBridge scala3-sbt-bridge, which ships with Scala 3

The default is 2.12.x. y only marks a second 2.13 variant: since Scala 2.13.12 and 3 ship their own bridges, sbt uses those for real builds, so 2.13.y and 3.x are how a test checks Zinc against them. A test can also give a full version, which selects the first label built for it.

The bridge is looked up by label rather than by Scala version, so 2.13.x and 2.13.y stay apart when both use the same Scala version; a full version is first mapped to a label. All projects in a test share one compiler, so a build.json whose projects use different labels fails, counting a project without scalaVersion as 2.12.x. > checkBridge scala2-sbt-bridge (or zinc, scala3-sbt-bridge) asserts which bridge that is, from the package of CompilerBridge in the bridge jar; the general/bridge-* tests do this for each label.

The test script is one step per line:

  • > compile runs a task on the root project, > use/compile on the use project.
  • -> compile expects the task to fail.
  • $ copy-file changes/A.scala A.scala runs a file command (copy-file, delete, exists, absent, newer, touch, sleep, pause).
  • Assertions are tasks too: checkRecompilations, checkIterations, checkProducts, checkDependencies, checkClasses, checkWarnings, checkErrors, checkSame, and others.

The full task list is the commands map in IncHandler.scala; file commands come from ZincFileCommands and sbt's FileCommands.

Negative tests (expected compilation errors)

-> compile passes on any failure, including an environment failure. To check that the failure is the expected one, follow it with the check commands. A failed compile stores no analysis, so they read the problems from its CompileFailed until the next compile starts.

-> compile
> checkErrors 1
> checkError 0 "type mismatch"

checkError <index> <substring> checks that the message contains the substring. Messages differ across Scala versions (Scala 3 writes Required:), and problems keep their order only within one source file. Examples: reporter/errors-reported, reporter/errors-reported-multi.

Running

sbt scripted                                                # all tests
sbt "scripted source-dependencies/abstract-class-to-trait"   # one test
sbt "scripted source-dependencies/*"                         # one group

Tests run at once, one per core; set localzinc.Scripted.scriptedParallelism := 1 runs them one at a time. Each test runs in its own temporary copy of its directory and writes a debug log of its own, in the directory named at the end of the run. Nothing waits between steps, so a test whose assertions compare modification times needs its own $ sleep.

incOptions.properties

Scripted tests have no build definition rich enough to configure Zinc, so this file is how a test sets a few IncOptions fields and the scalac options for a project. It is read by IncHandler.loadIncProperties and parsed by IncOptionsUtil.fromStringMap.

Place it in the project base directory: the test root for a single-project test, or each subproject directory (dep/, use/) for a multi-project one. Both incoptions.properties and incOptions.properties are accepted, lowercase taking precedence.

Standard java.util.Properties syntax (# comments, key = value). Omitting a key leaves the option at the effective scripted default shown below, which for some keys is not the IncOptions default.

# source-dependencies/same-source-transitive-invalidation
transitiveStep = 1

Supported keys

These take effect. "Default" is the value you get when the key is absent from a scripted test, which is what a test author needs; where that differs from the IncOptions default, the difference is noted.

Key Value Default
transitiveStep int: invalidation cycles before falling back to the transitive closure 3
recompileAllFraction double: fraction of sources invalidated that triggers a full recompile 1.0 (IncOptions uses 0.5)
relationsDebug boolean: verbose invalidation diagnostics, including the pruned relations and detected API changes false
apiDebug boolean: log API diffs true (IncOptions uses false)
apiDiffContextSize int: context lines in API diffs 5
recompileOnMacroDef boolean, or NOTHING to leave unset unset, which behaves as true
logRecompileOnMacro boolean true
useOptimizedSealed boolean: use the optimized sealed-children invalidation false
storeApis boolean: persist extracted APIs in the analysis true
pipelining boolean: pipelined compilation true (IncOptions uses false)
scalac.options space-separated options passed to the compiler; [basedir] expands to the project's absolute base directory. Handled by IncHandler, not IncOptionsUtil none
incOptions.storeApis boolean; same as storeApis, applied after parsing. Handled by IncHandler true

Two wrinkles in this table:

  • recompileAllFraction = 0.5 cannot be expressed. The engine detects "not set" by comparing the parsed value against IncOptions.defaultRecompileAllFraction() rather than checking whether the key is present (IncHandler.scala:837), so writing the default explicitly still yields 1.0. Any other value is honoured.
  • With pipelining on, -Ypickle-java -Ypickle-write <earlyOutput> is appended to scalac.options automatically. Java-heavy tests set pipelining = false to opt out.

Recognized but ineffective keys

IncOptionsUtil parses these, so they look supported, but nothing observes them in a scripted run.

Key Why
classfileManagerType Always overwritten with a transactional manager rooted at target/classes.bak (IncHandler.scala:363)
transactionalManagerBaseDirectory Only feeds the manager built from classfileManagerType, which is then discarded
allowMachinePath Not consulted; the engine hardcodes true when it builds the MappedFileConverter (IncHandler.scala:117)
apiDumpDirectory Unimplemented in Zinc itself, as incremental.contra notes
ignoredScalacOptions Only affects whether a change in scalac options forces a full recompile (MiniSetupUtil.scala:156), and scalac options cannot change during a scripted run, see caveats

Unsupported IncOptions fields

IncOptionsUtil recognizes only the keys listed above. Other IncOptions fields, including strictMode, enabled, useCustomizedFileManager, auxiliaryClassFiles, extra, and externalHooks, have no property key and cannot be set from this file.

Caveats

  1. The file is read once, at project initialization. incOptions and scalacOptions are vals on ProjectStructure (IncHandler.scala:350) and every compilation reuses them (IncHandler.scala:723), so copying changes/incOptions.properties over the live file partway through a test script has no effect. This is also why ignoredScalacOptions is inert: a test cannot vary its scalac options between compilations, so the comparison that key relaxes never sees a difference. source-dependencies/scalac-options is written as if it did work, and passes for other reasons.
  2. Unknown keys are silently ignored. There is no validation, so a typo, or one of the unsupported fields above, looks exactly like a working setting.