feat(thumbjar): exit thumbjar prepare with its own status when Gboard's Japanese dictionary never loads

.github/workflows/android-ime.yml

@@ -6,8 +6,9 @@ name: Android IME
 # Each job prepares a fresh emulator for its row's language and runs that
 # row's device test file. A job is red when a prep step failed or a test
 # failed, an editor assertion included, or as an infrastructure failure when
-# the emulator never booted or adb lost the device on the retry too. This
-# workflow is not a required check, so a red job blocks nothing.
+# the emulator never booted, adb lost the device or Gboard's Japanese
+# dictionary never loaded on the retry too. This workflow is not a required
+# check, so a red job blocks nothing.
 
 on:
   pull_request:
@@ -147,9 +148,10 @@ jobs:
           working-directory: packages/editor
           script: android/run-on-device.sh ${{ matrix.suite.file }}
 
-      # The emulator sometimes never comes online, or adb loses it after boot.
-      # Retry once on a fresh emulator then, never after a failure on a
-      # reachable device.
+      # The emulator sometimes never comes online, adb loses it after boot, or
+      # Gboard's Japanese dictionary never loads on that boot. Retry once on a
+      # fresh emulator then, never after any other failure on a reachable
+      # device.
       - name: Decide the retry
         id: retry
         if: steps.run1.outcome == 'failure'

packages/editor/android/README.md

@@ -25,13 +25,13 @@ The empty-block test in `tests/ja.test.ts` covers a bug the editor has had: Gboa
 
 ## Reading a run
 
-A job goes red when a device prep step failed, or a test failed, including a test whose editor assertion did not hold, or as an infrastructure failure when the emulator never booted or adb lost the device on the retry too. The workflow is not a required check, so a red job blocks no merge. The Result step's `run outcome` line, also in the job summary, says which:
+A job goes red when a device prep step failed, or a test failed, including a test whose editor assertion did not hold, or as an infrastructure failure when the emulator never booted, adb lost the device or Gboard's Japanese dictionary never loaded on the retry too. The workflow is not a required check, so a red job blocks no merge. The Result step's `run outcome` line, also in the job summary, says which:
 
-1. `run outcome: device lost twice, infrastructure failure, not a test result`, or a line naming an emulator that never booted, means the emulator failed on both runs. A line saying the run ended after boot without a completed device check means the last run stopped before its check could record a verdict. Say nothing about the editor from either.
-2. A `PREP FAILED: <step>` line means the device was never ready and no test ran. The `run outcome` line quotes it after `the prep failed:`, for example when Gboard's Japanese dictionary never loaded on that boot. `phases.log` and, for Japanese, `ja-setup.log` and `ja-dictionary.log` show where it stopped.
+1. `run outcome: device lost twice, infrastructure failure, not a test result`, or a line naming an emulator that never booted or Gboard's Japanese dictionary that never loaded, means the emulator failed on both runs. A line saying the run ended after boot without a completed device check means the last run stopped before its check could record a verdict. Say nothing about the editor from either.
+2. A `PREP FAILED: <step>` line means the device was never ready and no test ran. The `run outcome` line quotes it after `the prep failed:`, for example when Gboard did not convert with its Japanese dictionary after the load. `phases.log` and, for Japanese, `ja-setup.log` and `ja-dictionary.log` show where it stopped.
 3. Otherwise Vitest's output lists each test. A failed assertion shows the expected and actual values, and the `github-actions` reporter annotates the test file and writes the job summary. A driver fault the driver retried prints a `Driver fault in <test>` line.
 
-Each job retries once, on a fresh emulator, when the first emulator never booted (a `BOOT_RETRY` line) or when adb lost the device after boot, during prep or the tests (a `DEVICE_LOST_RETRY` line). Both lines also go to the job summary, and the `run outcome` line names the retry. A failure on a device adb still reaches is never retried, whether a test, a prep step or a driver fault failed. After a prep or test stage fails, `run-on-device.sh` reads `adb get-state` once, with its stdout apart from its stderr, where adb prints its daemon start-up lines, and writes its verdict to `timing.txt` as one line before anything else: `CHECKED device` when stdout is `device`, and `CHECKED lost` when stdout is `offline` or adb fails with `error: device offline`, the answer of a dead emulator, or reports no such device (`no devices/emulators found`, `device '<serial>' not found`). Anything else proves nothing and gets no verdict: a device that is `unauthorized`, in `bootloader` or `recovery` or in another state is still there, any other adb error says nothing about the device, and an `adb get-state` that does not answer within 5 seconds may only be a slow adb server. A retry on a reachable device could turn a real test failure into a passing retry. The read runs once the stage has ended, so it never changes a test's outcome. A run that needed no check writes `CHECKED skipped`. A lost verdict is retried whatever stops the run after it. A run that ended after boot without a verdict, because the step's `timeout-minutes` or the check's own timeout stopped it first or `adb get-state` proved nothing, is never retried. Its result reads `run outcome: the run ended after boot without a completed device check, not a test result`. A cancelled run reads `run outcome: cancelled`. `scripts/emulator-runs.mjs` decides the retry and the result from `timing.txt`, and the Boot time step prints each run's boot as `BOOT_SECONDS run <n>: <seconds>`.
+Each job retries once, on a fresh emulator, when the first emulator never booted (a `BOOT_RETRY` line), when adb lost the device after boot, during prep or the tests (a `DEVICE_LOST_RETRY` line), or when Gboard's Japanese dictionary never loaded on a device the check found reachable (a `DICTIONARY_RETRY` line). The lines also go to the job summary, and the `run outcome` line names the retry. `thumbjar prepare` exits 3 for that dictionary and 1 for every other prep failure, and `run-on-device.sh` writes `PREP DICTIONARY_NOT_LOADED` to `timing.txt` for status 3 only, so the retry never reads the prep's reason. Any other failure on a device adb still reaches is never retried, whether a test, another prep step, the dictionary check after the load or a driver fault failed. A dictionary that never loaded on the first run, then a run that could not retry, quotes the prep's reason like any other prep failure. After a prep or test stage fails, `run-on-device.sh` reads `adb get-state` once, with its stdout apart from its stderr, where adb prints its daemon start-up lines, and writes its verdict to `timing.txt` as one line before anything else: `CHECKED device` when stdout is `device`, and `CHECKED lost` when stdout is `offline` or adb fails with `error: device offline`, the answer of a dead emulator, or reports no such device (`no devices/emulators found`, `device '<serial>' not found`). Anything else proves nothing and gets no verdict: a device that is `unauthorized`, in `bootloader` or `recovery` or in another state is still there, any other adb error says nothing about the device, and an `adb get-state` that does not answer within 5 seconds may only be a slow adb server. A retry on a reachable device could turn a real test failure into a passing retry. The read runs once the stage has ended, so it never changes a test's outcome. A run that needed no check writes `CHECKED skipped`. A lost verdict is retried whatever stops the run after it. A run that ended after boot without a verdict, because the step's `timeout-minutes` or the check's own timeout stopped it first or `adb get-state` proved nothing, is never retried. Its result reads `run outcome: the run ended after boot without a completed device check, not a test result`. A cancelled run reads `run outcome: cancelled`. `scripts/emulator-runs.mjs` decides the retry and the result from `timing.txt`, and the Boot time step prints each run's boot as `BOOT_SECONDS run <n>: <seconds>`.
 
 Each job uploads an `android-ime-<file>-api36-<attempt>` artifact, `android-ime-ja-api36-1` for `tests/ja.test.ts`, with `vitest.log`, `phases.log`, `logcat.txt`, `diagnose.txt` and thumbjar's per-test artifacts: the page's state after each word as `<lang>-<test>.json`, screenshots and Gboard accessibility dumps.
 

packages/editor/android/run-on-device.sh

@@ -2,7 +2,9 @@
 # Runs inside android-emulator-runner's script block, emulator already booted.
 # Usage: android/run-on-device.sh tests/<file>.test.ts, with IME_LANG the keyboard to prepare.
 # Prepares the device with `thumbjar prepare`, then runs the one device test file it is given. A prep step
-# that breaks exits before any test runs, with a PREP FAILED line naming the step.
+# that breaks exits before any test runs, with a PREP FAILED line naming the step, which goes to timing.txt
+# for the job summary. A prep that exits 3, Gboard's Japanese dictionary never loaded on this boot, also
+# writes a `PREP DICTIONARY_NOT_LOADED` line there, which the workflow retries once on a reachable device.
 # Every stage runs under `timeout -k`, with the caps `scripts/budget.mjs` sums into the workflow's step
 # timeout: reading the test stage's cap for the file, prep, the tests under that cap, the device check
 # after a failed prep or test stage and the diagnostics after the tests.
@@ -33,11 +35,15 @@ server=$!
 trap 'kill $server 2>/dev/null' EXIT
 
 earlier_phases=$(cat "$out/phases.log" 2>/dev/null | wc -l)
-if ! pnpm exec thumbjar prepare --keyboard "$lang" --page-url http://localhost:8765/editor.html --out-dir "$out" --booted "$booted"; then
-  # The prep's own reason, for the job summary. Only the device check below decides a retry. A retry's
-  # `phases.log` still holds the lost boot's lines, so only the lines this prep appended count.
+pnpm exec thumbjar prepare --keyboard "$lang" --page-url http://localhost:8765/editor.html --out-dir "$out" --booted "$booted"
+prep_status=$?
+if [ "$prep_status" -ne 0 ]; then
+  # The prep's own reason, for the job summary. A retry's `phases.log` still holds the earlier boot's
+  # lines, so only the lines this prep appended count. The retry keys on the prep's status, never on the
+  # reason, and only together with the device check below.
   reason=$(tail -n +$((earlier_phases + 1)) "$out/phases.log" 2>/dev/null | grep -o 'PREP FAILED: .*' | tail -n 1)
   [ -n "$reason" ] && echo "$reason" >> "$out/timing.txt"
+  [ "$prep_status" -eq 3 ] && echo "PREP DICTIONARY_NOT_LOADED" >> "$out/timing.txt"
   device_check
   exit 1
 fi

packages/editor/android/scripts/emulator-runs.mjs

@@ -4,18 +4,19 @@ import {fileURLToPath} from 'node:url'
 
 // `timing.txt` holds one block of lines per emulator run, the retry's after a `RETRY` line. Each block
 // opens with `EMULATOR_START`, written just before its run step. A run prints `BOOTED` once its emulator
-// is up. A failed prep adds its `PREP FAILED: <reason>` line, which the job's result quotes. Once its last
-// stage has ended, it adds one `CHECKED <verdict>` line in a single write: `skipped`
-// when no check was needed, or the verdict of the device check after a failed stage, `device` or `lost`,
-// which the check writes itself before it does anything else. A booted run without that line ended before
-// a check reached a verdict, by its run step's timeout, the check's own, or an `adb get-state` that proved
-// nothing, and nothing proves what happened to its device: the action shuts the emulator down before any
-// later step could look.
+// is up. A failed prep adds its `PREP FAILED: <reason>` line, which the job's result quotes, and, when
+// Gboard's Japanese dictionary never loaded, a `PREP DICTIONARY_NOT_LOADED` line, which `thumbjar prepare`'s
+// exit status decides, never the reason. Once its last stage has ended, it adds one `CHECKED <verdict>`
+// line in a single write: `skipped` when no check was needed, or the verdict of the device check after a
+// failed stage, `device` or `lost`, which the check writes itself before it does anything else. A booted
+// run without that line ended before a check reached a verdict, by its run step's timeout, the check's own,
+// or an `adb get-state` that proved nothing, and nothing proves what happened to its device: the action
+// shuts the emulator down before any later step could look.
 
 /** The `adb get-state` read after a failed stage. Its process gets the rest of the script's cap to stop. */
 const DEVICE_STATE_MILLISECONDS = 5_000
 
-/** @typedef {'never booted' | 'unchecked' | 'lost' | 'reachable'} RunState */
+/** @typedef {'never booted' | 'unchecked' | 'lost' | 'dictionary not loaded' | 'reachable'} RunState */
 
 /**
  * What one `adb get-state` proves about the device, or `undefined` when it proves nothing. Only `device` on
@@ -51,8 +52,9 @@ export function deviceVerdict(adb) {
 }
 
 /**
- * Whether to run once more after a failed first run: only when its emulator never booted or adb lost the
- * device after boot, and never after a retry.
+ * Whether to run once more after a failed first run: only when its emulator never booted, adb lost the
+ * device after boot, or Gboard's Japanese dictionary never loaded on a device the check found reachable,
+ * and never after a retry.
  *
  * @param {string} timing
  * @returns {{retry: false} | {retry: true, line: string}}
@@ -76,12 +78,19 @@ export function retryAfter(timing) {
       line: 'DEVICE_LOST_RETRY: adb lost the device after boot, retrying once on a fresh emulator',
     }
   }
+  if (state === 'dictionary not loaded') {
+    return {
+      retry: true,
+      line: "DICTIONARY_RETRY: Gboard's Japanese dictionary never loaded, retrying once on a fresh emulator",
+    }
+  }
   return {retry: false}
 }
 
 /**
  * The job's result from the last run's step `outcome`. A last run whose emulator never booted or lost its
- * device fails as infrastructure, apart from any test result.
+ * device fails as infrastructure, apart from any test result, and so does a retry whose dictionary never
+ * loaded. A first run whose dictionary never loaded and that could not retry quotes the prep's reason.
  *
  * @param {string} timing
  * @param {string} outcome
@@ -106,7 +115,12 @@ export function runResult(timing, outcome) {
       line: `run outcome: the run ended after boot without a completed device check${retry}, not a test result`,
     }
   }
-  if (outcome !== 'success' && (last === 'lost' || last === 'never booted')) {
+  if (
+    outcome !== 'success' &&
+    (last === 'lost' ||
+      last === 'never booted' ||
+      (last === 'dictionary not loaded' && first !== undefined))
+  ) {
     return {
       passed: false,
       line: `run outcome: ${infrastructureFailure(last, first)}, infrastructure failure, not a test result`,
@@ -212,26 +226,39 @@ function runState(run) {
   if (verdict === undefined) {
     return 'unchecked'
   }
-  return verdict === 'lost' ? 'lost' : 'reachable'
+  if (verdict === 'lost') {
+    return 'lost'
+  }
+  return verdict === 'device' && /^PREP DICTIONARY_NOT_LOADED$/m.test(run)
+    ? 'dictionary not loaded'
+    : 'reachable'
 }
 
 /**
  * @param {RunState} state
  * @returns {string}
  */
 function retryReason(state) {
-  return state === 'never booted'
-    ? 'the emulator never booted'
+  if (state === 'never booted') {
+    return 'the emulator never booted'
+  }
+  return state === 'dictionary not loaded'
+    ? "Gboard's Japanese dictionary never loaded"
     : 'adb lost the device'
 }
 
 /**
- * @param {'lost' | 'never booted'} last
+ * @param {'lost' | 'never booted' | 'dictionary not loaded'} last
  * @param {RunState | undefined} first
  * @returns {string}
  */
 function infrastructureFailure(last, first) {
-  const failure = last === 'lost' ? 'device lost' : 'the emulator never booted'
+  const failure =
+    last === 'lost'
+      ? 'device lost'
+      : last === 'never booted'
+        ? 'the emulator never booted'
+        : "Gboard's Japanese dictionary never loaded"
   if (first === undefined) {
     return failure
   }