Repository navigation
test(editor): run the Android device tests from packages/editor with thumbjar - #3554
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Bundle Stats✅ No significant changes. All scenario measurements (37)🗺️
Significant means at least 1.0 KB and 1% gzip, or at least 5 ms and 10% import time. |
fe572d4 to
401d5cc
Compare
139b6fa to
3b33ef9
Compare
6aa32c8 to
5547682
Compare
3b33ef9 to
322fa4f
Compare
5547682 to
e3bfef4
Compare
e3bfef4 to
65770df
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit f8561d2. Configure here.
f8561d2 to
c355f1a
Compare
… `packages/editor/android`
A pure move, so `git log --follow` keeps each file's history. Every
file is renamed with identical bytes:
- the driver, `scripts/driver.ts`, and its test to `packages/thumbjar/src/`
- `scripts/lib/{typing,hierarchy}.ts` and their tests to `src/gboard/`
- the adb, CDP, uiautomator2, budget and util modules and their tests to
`src/device/`
- the prep scripts to `src/prepare/`, `run-on-device.sh` as `prepare.sh`
- `tests/global-setup.ts`, `tests/setup.ts` and `scripts/global-setup.test.ts`
to `src/vitest/`
- the recorded fixtures and fake-device suites to `src/fixtures/`, and
`scripts/fixtures/run-suite.ts` to `src/testing.ts`
- the app's manifest, `tsconfig.json` and `.gitignore` to
`packages/thumbjar/`
- the editor's device tests, their test pages, their two fake-typing
suites, `device-suites.test.ts`, `README.md`, `tsconfig.page.json`
and `vite.config.ts` to `packages/editor/android/`, and
`vitest.config.ts` to `packages/editor/vitest.android.config.ts`
- `scripts/lib/budget.test.ts` to `packages/editor/android/scripts/`: it
checks the editor's device test files and its `run-on-device.sh`
against the budget, not the driver
The lockfile moves the app's importer to `packages/thumbjar`.
`.prettierignore` ignores `src/gboard`, `src/device` and the moved
`budget.test.ts` for this commit only: they sat under the app's ignored
`lib/` directory and were never formatted, and the pre-commit oxfmt run
would otherwise rewrite them here. The next commit formats them and
drops the entry.
The moved tree does not run yet: imports, scripts and configs still
name the app's layout. The following commits fix that.
…`lib/` `apps/android-ime/scripts/lib/` was ignored by `.prettierignore`, so oxfmt never formatted its modules. Apply the oxfmt output to the twelve that now sit in `src/gboard/` and `src/device/` and to `budget.test.ts` in `packages/editor/android/scripts/`, and drop the temporary ignore entry.
`apps/android-ime` held both a general driver, which types on real
Gboard in an Android emulator and reports what a web page ended up
with, and the editor's own device tests. Turn the moved driver into a
private workspace package, `thumbjar`, that consumers depend on through
`workspace:*` and import from `dist`. It builds with
`pkg-utils build --strict --check` like `racejar`, so `turbo`'s `^build`
builds it before a consumer's build and tests, and a consumer's Vitest
config loads it on any Node the editor supports, where Node below 22.18
cannot import `.ts` source. The root `check:knip` runs Knip with the
`source` condition, because Knip loads the editor's Android Vitest
config, which imports `thumbjar/vitest`, straight after install and
before any build.
The public seam:
- `thumbjar` exports `typeOnDevice` and its `TypeOnDeviceOptions`,
`Word` and `Typed` types. `url` replaces the `page` file name: the
driver sets `prefix` as a search parameter on it, opens and matches
the CDP target by that one serialized URL, and reverses its
`localhost` port, the scheme's default when it names none, as
`prepare` does. `keyboard` replaces `lang`. Artifacts go to
`THUMBJAR_OUT_DIR`, `out` by default.
- `thumbjar/page` exports `recordInput({getText, field})`, which
records the same events and installs the same `window.__state` as
the app's two test pages, with the page supplying the text.
- `thumbjar/vitest` exports `deviceProject`, the device project the
app's Vitest config held, with the adb and page checks of its global
setup and the `afterAll` guards of its setup file. It hands Vitest the
two files as `thumbjar/vitest/global-setup` and `thumbjar/vitest/setup`,
resolved through the package's own exports, so the setup file shares
the one built driver module, and the `driverRuns` it checks, with
`thumbjar`. The page URL and the hint printed when it does not answer
are options, passed to the global setup with `provide`. It also
exports the budget's `DEVICE_TEST_FILE_SECONDS` and
`DEVICE_TESTS_PER_FILE`, so a consumer can run a device test file
under the bound the budget sums for it.
- `thumbjar/testing` exports `runSuite`, the app's helper that runs one
test file in a Vitest of its own and collects its result. The app
resolved the inner run's root from the helper's own location, which
only held while the helper sat next to its callers, so callers now
pass `root`.
- The `thumbjar` command runs `fetch-u2-jar`, `serve` and `prepare`.
`prepare` is the device prep of the app's `run-on-device.sh` with the
keyboard, page URL and output directory as flags. It never boots the
emulator. The build bundles the command and the Japanese setup, and
copies `prepare.sh` and `diagnose.sh` into `dist/prepare` next to them.
`bin` points at a checked-in `bin/thumbjar.js` that imports
`dist/bin.js`, because pnpm links a bin only when its file exists at
install time, which comes before the build. The uiautomator2 jar
cache is resolved from the package root instead of the module's own
directory, which in `dist` sits one level higher than in `src`.
The retry, the time budget and the `test.fails` accounting are the
app's. The editor's moved tests still use the app's layout until the
next commits move them onto the package. thumbjar stays out of the
stricter oxlint rules for published packages for now, which the app
never ran under.
thumbjar sat outside the stricter oxlint override for `packages/**`,
which the app it came from never ran under. Drop the exclusion and fix
the 227 errors it raised. Only the failure dump notes change behavior:
they now go to stderr.
- `curly`: braces on every single-statement `if` and loop body.
- `prefer-template`: template literals for string concatenation. Two
calls built a multi-line `map` inside the template, so their text
goes to a named local first.
- `require-await`: the fakes in the tests and fake-device suites return
`Promise.resolve` or `Promise.reject` instead of being `async`, so a
throwing fake still rejects rather than throws, and `deviceShell`
returns `tracked`'s promise without `async`. The fake page of
`device-types-digit-once.ts` answers from a synchronous
`pageReply`.
- `no-empty-pattern`: the unused options argument of a fake
`typeInChrome` is a named parameter.
- `no-control-regex`: `isAscii` matches `\p{ASCII}`, the same range.
- `no-console`: the `thumbjar` command's scripts write to
`process.stdout`, and the failure dump notes use `console.warn`. The
driver's `PHASE` lines keep `console.log` with a disable comment,
because they are progress output, not warnings.
…h `thumbjar` The editor's device tests, their test pages and their fake-typing suites, moved to `packages/editor/android`, run on `thumbjar`, and `run-on-device.sh` is rebuilt on its `prepare` and `serve` commands. The app's copy of `runSuite` gives way to `thumbjar/testing`. The tests run through their own Vitest config, `vitest.android.config.ts`, with `android/` as its root, anchored to the config file so the root does not depend on the working directory: a `device` project from `deviceProject` for `test:android`, and an `android-unit` project for `test:android:unit`, which runs `tests/ja.test.ts` against fake typing results. `vitest.config.ts` excludes `android/` from the `unit` project, which globs every `.test.ts` outside its exclusions, so `vitest --run` and the browser projects collect the same files as before. `check:types` also checks the tests and configs against `android/tsconfig.json`. The page bundles the built editor, so `build:android` checks its types and builds it after the editor's own build. The published package keeps its file list. `run-on-device.sh` changes to `packages/editor` before it runs anything, because `pnpm exec` finds no package in `android/`, which has no manifest. It serves `android/dist` and writes to `android/out`, and reads the test stage's timeout, `DEVICE_TEST_FILE_SECONDS`, from `thumbjar/vitest`. `scripts/budget.test.ts` checks the editor's own `run-on-device.sh` and `tests/*.test.ts` against that budget, so it reads them from `android/` and the budget from `thumbjar/vitest`, and runs in `android-unit`. The workflow keeps its jobs, caps and triggers. Its path filter, build and jar download point at `packages/editor`, its steps run there and start `android/run-on-device.sh`, and it uploads `android/out`. The fake-typing suite, which ran in root `pnpm test:unit` as the app's `test:unit`, still runs there: the editor's `test:unit` runs `test:android:unit` after its `unit` project. Neither runs device work.
The driver found its field with `document.querySelector('[contenteditable]')`
in two page expressions, the tap rect and the forced focus, and both focus
checks only accepted an active element that `isContentEditable`.
`recordInput` read every field's `textContent`, so a `textarea` recorded
an empty or stale initial text at each event, an `input` an empty text,
and the no-field `dom` reported the first `[contenteditable]` even when
the page typed into a `textarea`.
`recordInput` now installs `window.__field()` next to `window.__state`.
It returns the page's `field`, or without one the first match of
`TEXT_FIELD_SELECTOR`: a `textarea`, an `input` with no type or of type
`text`, `search`, `tel`, `url` or `password`, or a `[contenteditable]`.
Other input types take no text or reject `setSelectionRange`, so a
checkbox, hidden input or button before the field is skipped. The page
expressions move to `gboard/field.ts`. The forced focus puts the caret
after a `textarea` or `input` value with `setSelectionRange` and keeps
the range collapse for a contenteditable. The focus check accepts the
field itself or, for a contenteditable, an element inside it, so another
focused input no longer counts. A field's text is its `value` for a
`textarea` or `input` and its `textContent` otherwise, in each recorded
event and in `dom`.
The driver waited for `document.readyState` and `window.__state` only on
the first load. When Chrome was not in front it reopened the URL and
resolved the field at once, before the new document had run
`recordInput`, so a page that names a field had its first matching field
tapped instead. The readiness loop moves into `waitForPage`, which the
first load and the reopen both call, and it now also waits for
`window.__field()` to return an element, within the same 20 checks a
second apart. So a page that names no field may render it after
`recordInput`. Before, a page that had run `recordInput` without its
field yet passed the readiness loop, and the tap rect read `field.style`
on `null`, a page exception, which is never retried. Now the driver
waits, and a field that never renders fails with `page never rendered
its field`, a driver fault, so the attempt is retried. A page that never
loads or never runs `recordInput` still fails with `page never loaded
over CDP`.
That check is a separate evaluation, and `cdpEval` picks the first page
target with the URL afresh for each expression, with no document
identity. A navigation that commits after it, or an unnamed field that
re-renders, can still hand the tap rect and the forced focus a document
without a field, where `field.style` on `null` would be a page
exception, which is never retried. So `FIELD_RECT` and `FOCUS_FIELD` now
look the field up and return `null` without one in the same evaluation,
as `FIELD_FOCUSED` already returns `false`. The driver evaluates each
again a second later, up to 20 times, and then throws the same `page
never rendered its field`. An exception the page throws in either still
fails the attempt at once. Pages that render their field before
`recordInput` see one more evaluation per readiness wait and nothing
else. Each expression still calls `window.__field()` when it exists and
runs the same selector query otherwise. That query does not see a field
the page names. A page that names no field and has a contenteditable but
no `textarea` or text `input` resolves the field the old query did.
The JSDoc and the README say a page seeds the field and leaves it to the
keyboard: a `value` prop written back into a `textarea` on each input
nested 7 of 8 Japanese compositions. `jsdom` joins the dev dependencies
for the page and field tests.
`textarea.html` types into a plain `textarea` that `recordInput` names as its `field`, and `tests/en-textarea.test.ts` types `kyou` twice on English Gboard into it and expects `Kyou kyou `. It sits in a file of its own because `tests/en.test.ts` already holds the three tests a device test file may declare, and `run-on-device.sh` only runs `tests/$IME_LANG.test.ts`, so CI never runs it. It runs by hand against a prepared device.
…eps with `runOnDevice`
`typeOnDevice` only typed words and committed each by candidate or
Space, so a case could not press Enter while a word composed or
Backspace at a block start, and each new action would have needed an
entry of its own that opens its own page.
`runOnDevice(context, {url, keyboard, prefix, steps})` takes a case as a
list of steps: `compose` with `letters` or `text` and a `commit` of
`{candidate}`, `'space'` or `'none'`, `enter` and `backspace` with
`times`. `unsupportedNote` checks every step against what Gboard drives
on the keyboard before the device is touched: every character of
`letters` or `text` must be one of the `LETTERS` that `keyboardState`
requires of a whole layout, a candidate needs `ja`, and Japanese QWERTY
takes `letters` rather than `text`. `letters` names keys, not output
text, so a key is named by its lowercase letter on both layouts:
`tapKey` falls back to the capital label, so `'k'` taps the K key
whatever Gboard's Shift state, while `'K'` only resolves while Gboard
shows capitals, and is rejected with a reason that says so. A step that
is not an object, an empty slot of a sparse list, which every step loop
visits as `undefined`, a non-string `letters` or `text`, an unknown step
type and a commit that is none of the three, `null` included, are
rejected at runtime as well as by the `Step` union. A case with such a
step skips the test through `context.skip`, with a note that names each
such step by position, by type when it has a valid one, and by reason.
It never touches the device and writes no artifact, and Vitest reports
the test as skipped with the note. A first argument without the shape of
a test context, `undefined` or the options of a call written the old
way, throws before the steps are checked. Otherwise it returns one
`StepRecord` per step, of one shape for every type: `type`, `status`
(`done`, `failed` or `skipped`), `reason`, `textAfter`, `selectionAfter`
and `events`. `runSuite` from `thumbjar/testing` returns the note of a
skipped test.
`context` is the `TestContext` Vitest passes to the test function, and
everything that belongs to the test comes from it rather than from
`TestRunner.getCurrentTest()`, one module-level pointer that each test
sets as it starts: the skip, the signal, and the test name and `fails`
flag of the artifact names and the `driverRuns` record. The pointer can
name another test, a concurrent sibling or, when a test timed out while
its body was still pending, the next sequential test, and that test's
`skip` would mark it skipped and discard its failed assertion. In Vitest
5.0.5 `createTestContext` builds one context per task, its `skip` closes
over that task and its `signal` is that task's own controller, which
aborts only for that task, on its timeout or a hook timeout of its own,
or when the run is cancelled. So `runOnDevice` takes the context Vitest
passed to the calling test, and a concurrent test may call it. It checks
only the shape of `context`, the public `signal`, `skip` and
`task.name`, so the context of another test passes and makes that test
the owner of the call. `test.each` passes no context, so a test body
that needs one uses `test.for` or the context of a `beforeEach`, as the
README says. A call whose `context.signal` has already aborted, because
it resumed after its test timed out, records an `aborted` run against
that test, so the setup file fails the file of a `test.fails` test, and
rejects with the signal's reason before it checks the steps or touches
the device. `drive` takes the test's name, `fails` flag and signal as a
parameter, and `typeOnDevice` still reads them from `getCurrentTest()`,
so for a list of plain `Word` objects its records and artifacts of
sequential calls are unchanged. `drive` now reserves its turn, chaining
the call onto `previousRun`, before its first await, so simultaneous
calls of either entry run one after another in the order they were made.
Base replaced `previousRun` only after awaiting it, so `typeOnDevice`
calls that arrived together, before either had replaced it, overlapped
on the device. Sequential calls are ordered the same as before. The time
bounds are per call and start once the call drives the device, and a
queued call spends its own test's timeout while it waits.
Each step is bounded by two boundaries, each one `FIELD_BOUNDARY`
evaluation that returns `window.__state()` and the field's selection
together, so no event lands between the text, the selection and the
event count. The first is read once the field is focused, before the
first step, and one more after every step. A step's `events` are those
between its two boundaries, and `run.events` spans the first boundary to
the last, so an event recorded while the page loaded or the field was
focused is in no record and every other one is in exactly one. The
selection is read through `window.__field()` and counted in grapheme
clusters with `Intl.Segmenter`. A case with no steps returns the text
and selection of the first boundary.
Enter and Backspace tap the key found by resource id,
`key_pos_ime_action` and `key_pos_del`, and by label when the dump has
none, the way the language probe found them. `packageNodes` keeps each
node's resource id, and `packageKeys` is built on it.
`typeInChrome` now opens the page through `openKeyboard` and runs its
words through `runSteps`, the loop `runInChrome` uses, which converts
each word to a `compose` step only when it reaches that word. It reads
no boundaries, so the page sees the same `window.__state()` calls, one
after each word and one after the last.
Both entries go through `drive`, which owns the `driverRuns` record, the
retry, the attempt deadline and the Chrome cleanup, and both attempts
classify the unexpected digit with `throwOnUnexpectedDigit`. A
`runInChrome` step that throws is recorded `failed`, the steps after it
`skipped`, the records go to the attempt's artifact, and the error is
retried as a driver fault. Each attempt of `runOnDevice` holds its own
outcome, which the attempt's step failure sets with the thrown value and
the records. When a step of the last attempt threw the value the call
ends on, compared with `Object.is` so that `NaN` matches, `runOnDevice`
rejects with a new `Error` whose message is that value's message, or the
value as a string when it is not an `Error`, converted with `String`
inside the guard, so a Symbol shows by its description, or `<thrown
value that cannot be converted to text>` when reading or converting
either throws, followed by one line per step of that attempt: position,
type, status, reason and `textAfter`. Its `cause` is the thrown value,
which is left as it was. So the report of a failed test shows how far
the case got. `drive` takes how the call inspects a value it did not
create, a thrown value or an abort reason, as an `Inspection`, and so do
`timedAttempt` and, through an `AsyncLocalStorage` the way they read the
attempt's signal, the device helpers below it. `runOnDevice` passes the
guarded one. The step's record and every log line of the call, the
retry's, the failure dump's, the server shutdown's and the cleanup's
among them, describe the value through `guardedErrorRepr`, which
converts every part with `String` and falls back to the same text, so a
throwing `toString`, a throwing `message` getter, or a `message` that is
a Symbol or an object whose `toString` throws never replaces the step's
failure or the signal's reason. An `instanceof` check of the value,
which throws for a revoked Proxy, counts as no match, so such a value is
a driver fault that keeps its identity and records, and a real
`PageExceptionError` is still never retried. Each field of a step's line
is converted on its own and falls back to the same text. `typeOnDevice`
passes the plain one, which describes with `errorRepr` and lets every
check throw as before. Any other failure rejects with its value
unchanged: an attempt that threw before the first boundary, even after
an earlier attempt that had records, or a restart that threw before the
retry. Nothing outlives the call, so a value thrown again in a later
call carries only that call's records. No consumer checks the class of
the rejection, and `drive` records the run's state before the call wraps
the error. For a list of plain `Word` objects, every error of a
sequential `typeOnDevice` call is unchanged. A `runOnDevice` attempt
writes its failure artifact only after it passed the step failure on,
and serializing it runs the steps' getters and `toJSON`, so a failure to
write it is logged and never replaces the step's error or changes
whether it is retried.
For a list of plain `Word` objects in sequential calls, `typeOnDevice`
taps, records and writes the same as before, and its artifacts in the
unit suites are byte-identical. A `null` word or a hole still throws the
same error after typing the words before it. Outside that, two things
differ. A word's `candidate` is now read before its letters are typed
rather than after, which only shows when reading it throws. A word
without iterable `letters`, or a `words` that is not an array, throws
the same `TypeError` at the same point with different wording.
`DEVICE_TEST_FILE_SECONDS` was the cap of a file of `DEVICE_TESTS_PER_FILE`, three, tests, 1825 seconds, whatever the file held. A caller that ran a file of one test under it waited 1150 seconds longer than the file could take before cutting a hung run off, and a file of four tests needed the constant itself raised for every caller. `deviceTestFileSeconds(testCount)` from `thumbjar/vitest` returns the cap of a file of `testCount` tests: each test's 575-second timeout, the global setup's 40 and Vitest's own 60, 1825 for three tests as before. It replaces both constants, whose only user was the editor's device test script.
…t device once
The Android IME workflow ran one job per language, and
`run-on-device.sh` ran `tests/$IME_LANG.test.ts` under thumbjar's fixed
three-test cap, so a file could only run in CI by being named after a
language, and a fourth test in a file outran its cap. The job only
retried an emulator that never booted: when adb lost the device after
boot, the prep or the tests failed like a test result, and the job went
red on the infrastructure with nothing to tell it apart from an editor
bug.
The matrix now has one row per device test file, `{lang, file}`, for
`tests/ja.test.ts` and `tests/en.test.ts`. Each job is named after its
file and uploads `android-ime-<file>-api36-<attempt>`, and
`run-on-device.sh <file>` prepares the row's language and runs only
that file, under `deviceTestFileSeconds` of the number of tests
`scripts/budget.mjs` counts in it. `scripts/budget.test.ts` derives each
run step's `timeout-minutes` from the largest file in the matrix and
the job's from the sum of its step caps, and fails when the workflow
does not match, when the matrix does not list exactly those two files,
or when the count of a file differs from the tests Vitest collects.
`tests/en-textarea.test.ts` stays a control run by hand.
After a failed prep or test stage, once it has ended,
`run-on-device.sh` reads `adb get-state` once through
`scripts/emulator-runs.mjs device-state`, under its own 5-second timeout
with stdout and stderr apart, so adb's daemon start-up lines on stderr
never read as a lost device. The checker appends its verdict to
`timing.txt` as one line before it logs anything: `CHECKED device`
when stdout is `device`, and `CHECKED lost` when stdout is `offline`,
adb fails with `error: device offline`, the answer of a dead emulator,
or adb reports no such device. Any other answer proves nothing and
writes no verdict: an `unauthorized`, `bootloader` or `recovery`
device is still there, another adb error says nothing about the
device, and an `adb get-state` that hangs past its 5 seconds may only
be a slow adb server. A run that needed no check writes
`CHECKED skipped`. The read never runs inside
the test stage's cap or between inputs, so it cannot change a test's
outcome.
`emulator-runs.mjs retry` replaces the boot check: it retries once on a
fresh emulator when the first run never booted or lost its device, and
marks `RETRY` in `timing.txt`, so a retry is never retried, whatever
its first run's failure. `emulator-runs.mjs result` writes the
`run outcome` line to the log and the job summary, naming the retry,
and a last run that never booted or lost its device fails as
"device lost twice, infrastructure failure, not a test result" or an
equivalent line naming the emulator failures. A failure on a device
adb still reaches is never retried.
A lost verdict is retried whatever stops the run after it. A booted
run without a verdict ended before a check completed, by its run
step's `timeout-minutes`, the checker's own timeout or an
`adb get-state` that proved nothing, and the action
has shut its emulator down by the time a later step could look, so
nothing proves the device was lost. Such a run is never retried, and
its result reads "the run ended after boot without a completed device
check, not a test result", on the first run and on the retry alike. A
cancelled run reads "run outcome: cancelled" before any of this.
Each run's start is recorded, the retry's by the retry decision, and
`emulator-runs.mjs boot-seconds` prints `BOOT_SECONDS run <n>` for each,
where the old step subtracted every `BOOTED` from the first start. The
README describes `emulator-boot-timeout` as the runner's polling
allowance, which its unbounded `adb` polls can outlast, and the run
step's `timeout-minutes` as the bound.
`budget.mjs` and `emulator-runs.mjs` are plain JavaScript, typed with
JSDoc and checked by `android/tsconfig.json`, so `run-on-device.sh`,
the workflow and the unit tests run them with `node` on every Node the
editor supports, which includes versions that cannot strip types.
The device check adds 15 seconds to the script, 2540 for a file of
three tests, and each run step stays at 52 minutes. The job's cap was
153 minutes while its step caps summed to 151. It is now 152, the sum
with the step that names the suite.
…se dictionary in `thumbjar prepare` `thumbjar prepare` left Play Store enabled, and from about 150 seconds after boot it updated Play services, Chrome, WebView and Gboard on its own, and installing Play services killed Chrome mid-test. The Japanese prep added the layout and handed the device to the tests while Gboard might still run its minimal Mozc engine: a session created before the dictionary loads never converts, so `kyou` offered only きょう and a test that commits 今日 failed on the device, not on the page. The prep now disables Play Store with `pm disable-user` under its own 20-second cap, each adb call 8, and logs one `self-update block:` line saying whether the disable succeeded and whether `pm list packages -d` lists Play Store. It never fails the prep. Gboard downloads its dictionary itself, not through Play Store. For `--keyboard ja` the prep first captures the `main` log under a header naming the boot id and boot time, with a 16M buffer, logd's line counts and a background reader in a process group of its own. After the layout is added, `dist/prepare/dictionary.js wait`, bundled from `src/prepare/dictionary.ts`, reads the capture until Gboard's `New data is loaded` line for a `mozcdata-*.data` file, and counts only a line logged at most 360 seconds after `--booted`, the host time the emulator came online, or after the prep started without it. The wait moves that deadline onto the device's clock from one reading of `adb shell date +%s`, a whole second, between two host readings, and takes the latest cutoff that reading allows. It errs toward accepting: a load inside 360 seconds always counts, and a load at most the adb round trip plus 1 second late may count too. A load line logged later than that fails the wait even when the wait first reads it after the deadline, which is stricter than the reference harness, whose wait accepted it. A capture of an earlier boot or without its header fails at once. `dictionary.js check` then opens the page and raises Gboard with the driver's own `openKeyboard`, switches Gboard away from Japanese and back through `ensureJapanese` with the capture marked before the switch, requires a `RELOADED` line after that switch, types `kyou` in the page's field, requires a kanji candidate on the strip and deletes the composition. Reusing the driver's helpers gives the check its ANR dismissal, its Chrome foreground recovery, its Space swipe on a layout with no language key and its romaji QWERTY check. A load that never came fails the prep with a PREP FAILED line that says so first, then names Gboard's version, its fallback engine and cancelled dictionary downloads, and only then any gap in the capture. On every exit, a passing prep, a failed one, a prep past its deadline and one stopped by SIGTERM, the prep ends the capture's process group: SIGTERM, 2 seconds, then SIGKILL, and it reaps the reader before it returns. `prepWorstSeconds(keyboard)` from `thumbjar/vitest` bounds the prep per keyboard. English keeps its 600 seconds of stages and adds only the Play Store block, 20 with 2 to stop, under `timeout -k 10 622`, 632 in all. Japanese adds the capture's 15 and 2 before its deadline, and the wait's last read 45 and 5 and the check 180 and 10 inside it, under `timeout -k 10 862`, 889 in all. The 360-second load window falls inside the 600.
…oom, and quote a failed prep's reason in the result `run-on-device.sh` capped `thumbjar prepare` at a fixed 610 seconds, which no longer covers a Japanese prep that blocks Play Store updates and waits up to 360 seconds after boot for Gboard's dictionary. Each run step's `timeout-minutes` was the boot, the script and the runner allowance rounded up, with nothing for the action, shell and `pnpm` startup no cap counts, so a step could stop the script before it logged its own reason. A prep that failed on a reachable device left only `run outcome: failure` in the job summary, so a dictionary that never loaded read like any other failure. `scripts/budget.mjs` takes each keyboard's prep cap from thumbjar's `prepWorstSeconds`, 632 seconds for English and 889 for Japanese, and `runStepMinutes` adds `RUN_STEP_HEADROOM_SECONDS`, 60. Each matrix row carries its own caps: a run step of 57 minutes and a job of 163 for `tests/ja.test.ts`, whose script on three tests ends within 2819 seconds, and 53 and 155 for `tests/en.test.ts`, whose script ends within 2562. A job's cap is its step caps on the retry path and `JOB_HEADROOM_MINUTES`, 1, so the Result step still runs. The test stage's cap is unchanged. `run-on-device.sh` passes its `BOOTED` time to `thumbjar prepare` as `--booted`, so the dictionary wait counts its 360 seconds from when the emulator came online. After a failed prep it appends the prep's last `PREP FAILED:` line to `timing.txt`, and `emulator-runs.mjs result` quotes it as `, the prep failed: <reason>` in the log and the job summary. The retry still reads only `BOOTED` and `CHECKED`, so a prep that failed on a reachable device is never retried.
…ever allowed `clearChromeTabs` ran `adb shell rm -rf /data/data/com.android.chrome/app_tabs` after every test and after the Japanese prep's dictionary check. The CI emulator image runs adb as a non-root shell user, so the command failed with Permission denied on every job, and the catch logged `clearing chrome tabs failed` and went on. The step never cleared anything. Remove it. The cleanup now only stops Chrome, and so does the end of `dictionary.js check`. The tests in each device test file open distinct URLs, and `cdpEvalOnce` matches the page target by its exact URL, so a tab Chrome restores from another test in the run does not match. The cleanup bound shrinks to one adb call, 35 to 20 seconds, and the call, test and test file budgets derived from it shrink with it: a `typeOnDevice` call from 535 to 520 seconds, the device test timeout from 575 to 545, and `deviceTestFileSeconds(3)` from 1825 to 1735.
…leanup thumbjar's cleanup after each test no longer deletes Chrome's saved tabs, so its device test timeout drops from 575 to 545 seconds and a file of three tests runs under 1735 seconds instead of 1825. `run-on-device.sh` on a file of three tests now ends within 2472 seconds for English and 2729 for Japanese. The run steps' caps drop from 53 and 57 minutes to 52 and 56, and the job caps from 155 and 163 to 153 and 161. The README, the budget tests and the workflow matrix carry the new numbers.
c355f1a to
fbf00ea
Compare

The Android IME driver from #3511 lives in
apps/android-imetogether with the editor's device tests, so nothing outside that app can type on real Gboard. This PR moves the driver into a private workspace package,thumbjar, and makes the editor's device tests its first consumer.thumbjarexportstypeOnDevice,thumbjar/pageexportsrecordInputfor the test page, andthumbjar/vitestexportsdeviceProject. Thethumbjarcommand fetches the uiautomator2 jar, serves the page and prepares the device. The device tests, their page andrun-on-device.shmove topackages/editor/android, with their own Vitest config rooted there. The workflow runs frompackages/editor, andapps/android-imeis deleted. The fake-typing suite still runs in rootpnpm test:unitthrough the editor'stest:unit.thumbjarbuilds todistwithpkg-utilslikeracejar, withsourceanddefaultexport conditions, soturbo's^buildbuilds it before the editor's tests. Raw.tsexports made the editor'stest:unitfail at config load on Node below 22.18, which the editor still supports. Thethumbjarbin is a checked-in shim that importsdist/bin.js, because pnpm links a bin only when its file exists at install time. Rootcheck:knipruns Knip with thesourcecondition, since it loads the Android Vitest config straight after install.Two driver changes come with the move. The page URL is now built with
URLandsearchParamsinstead of string concatenation, andopenInChromeshell-quotes the remoteam startcommand, because adb joins its arguments into one device shell command and a&in the URL split it. A test runs that command through/bin/shand asserts the exact argumentsamreceives. It fails on the unquoted code.The move comes first as its own commit, with every file renamed unchanged, so
git log --followreaches the app's history. The formatting those files skipped under the app's ignoredlib/follows in the next commit.thumbjarnow runs under the stricter oxlint rules forpackages/, and the sharedrunSuitehelper is exported once fromthumbjar/testinginstead of copied into the editor. The editor's published file list is unchanged.Before the later commits, five measured runs of the device workflow on 401d5cc had all 10 jobs green on the first attempt. One of 30 cases was retried once after Chrome DevTools dropped its connection, and then passed. Job times were 4.1 to 5.7 minutes, against about 5 to 8 on the #3511 baseline.
This PR also carries the work that was stacked on it, in commit order. Each part below was reviewed as its own PR before it was folded in.
Type into
textareaand textinputfieldsthumbjar's driver could only type into a
[contenteditable]. It found its field withdocument.querySelector('[contenteditable]'), andrecordInputreadtextContent, so atextareaor textinputwas either missed or recorded with empty or stale text. That rules out a plaintextareaas a control page and any app whose field is not a contenteditable.A page now names its field through the existing
fieldoption ofrecordInput, and the driver asks the page for it throughwindow.__field(). Without a named field the driver takes the firsttextarea, textinput(no type,text,search,tel,urlorpassword) or[contenteditable]. Other input types are skipped because they hold no text or rejectsetSelectionRange. A field's text is itsvaluefor atextareaorinputand itstextContentotherwise. The forced focus puts the caret after the value withsetSelectionRangeand keeps the range collapse for a contenteditable. The focus check now accepts only the field or an element inside it, not any focused contenteditable. On the two shipped pages, which have one contenteditable and no other field, the driver taps, measures, focuses and reports exactly what it did before, including every answer of the focus polling.When Chrome has fallen out of the foreground and the driver reopens the page, it now runs the same readiness check as on the first load before it looks up the field. Without it, a page that names a field could get the first matching field instead, because the new document had not run
recordInputyet. When the page is already ready this adds the readiness reads and nothing else. A navigation that commits after that check passed on the old document can still outrun it, becausecdpEvalattaches to the first page target with the URL and has no notion of which document it is. The section on late fields below covers what the driver now does about that.The docs now say a page seeds its field and leaves it to the keyboard. A
textareawhosevaluewas written back from page state on each input nested 7 of 8 Japanese compositions on real Gboard.The second commit adds
textarea.html, a plaintextareacontrol page, andtests/en-textarea.test.ts, which typeskyoutwice on English Gboard and expectsKyou kyou. It lives in its own file becausetests/en.test.tsalready holds the three tests a device file may declare, andrun-on-device.shonly runstests/$IME_LANG.test.ts, so CI does not run it. It has not run on a device yet.The unit tests run the real
typeInChromeagainst a mocked CDP and a jsdom page and pin the named-field reopen, the absent-hook fallback, the 20-attempt readiness bound, the skipped input types and the narrower focus check. Each fails on the code before the change.Cases as step lists with
runOnDevicetypeOnDevicecan only type words and commit each one, so a case can't press Enter while a word is composing or Backspace at the start of a block.runOnDevicetakes a case as a list of steps instead:Every step is checked against what the driver can do on the keyboard before the device is touched. A step it can't drive skips the test through
context.skip, and the skip note names each such step by position and reason. Otherwise the call returns one record per step with its status, the text and selection after it, and the events between the step's two boundaries. A boundary reads the page state and the field's selection in one evaluation, so every event lands in exactly one record. A step that throws is a driver fault, so the whole case is retried once liketypeOnDevice, and a page exception is never retried. When the last attempt fails, the test fails with an error that lists that attempt's step records, with the thrown value ascause.Everything that belongs to the test (the skip, the timeout signal, the artifact names and the
driverRunsrecord) comes from the context Vitest passes to the test, not fromgetCurrentTest(). The current-test pointer can name a concurrent sibling, or the next test when a timed-out test's body resumes late, and skipping through it would hide that test's failure. Concurrent tests may callrunOnDevice. A late call after a timeout records anabortedrun against its own test and rejects without touching the device. The context check only looks at the shape, so passing another test's context makes that test the owner.There is one behaviour change in the shipped
typeOnDevice. Simultaneous calls of either entry now run one after another. Before, two calls that arrived together both awaited the same previous run and drove the device at the same time. A queued call spends its own test's timeout while it waits. Sequential calls behave as before. For plainWordlists,typeOnDevicetaps, records and writes byte-identical artifacts across the unit suites, compared case by case against the base branch. The commit message lists the two exotic-input differences (a throwingcandidategetter is read earlier, and a non-iterablelettersthrows with differentTypeErrorwording).Diagnostics on the
runOnDevicepath never replace the original failure. That includes a thrown value whosemessagethrows or is a Symbol, a revoked Proxy, a failure artifact that can't be serialized, and an abort reason that can't be printed.typeOnDevicekeeps its plain diagnostics. Two limits stay: a replacedconsolethat throws while logging the warning for an unwritable artifact, and a getter that never returns. In-process guards can't contain either.Each fix is pinned by a unit test that fails on the code before it. This has not run on a device yet.
One CI job per device test file
The Android IME workflow ran one job per language, so a device test file could only run in CI by being named after a language, and every file ran under thumbjar's fixed three-test cap. When an emulator died after boot, as hosted emulators sometimes do, the job went red with nothing to tell the infrastructure failure apart from an editor bug.
The matrix now has one row per device test file (
tests/ja.test.tsandtests/en.test.ts), and each file runs under a cap derived from how many tests it holds. thumbjar'sdeviceTestFileSeconds(testCount)replaces the fixed constants.scripts/budget.test.tsfails when the workflow'stimeout-minutesno longer match the caps, when the matrix doesn't list exactly those files, or when a file's counted tests differ from what Vitest collects.tests/en-textarea.test.tsstays a hand-run control.A failed run is retried once on a fresh emulator only when adb proves the device was lost. After a failed prep or test stage, the script reads
adb get-stateonce, under its own 5-second timeout, and records a single verdict line before logging anything.offline,error: device offlineand "no such device" count as lost. Those are the answers dead emulators gave in earlier runs, and the tests use them as fixtures. Anything else is not proof: anunauthorizedorbootloaderdevice is still there, and a hung adb may only be slow. A failure on a reachable device is never retried, because a retry could turn a real failure green. A booted run that ended without a completed check, for example by its step timeout, is reported as "the run ended after boot without a completed device check, not a test result" and is not retried either, because the emulator is already gone by the time a later step could look. The old never-booted retry stays, and a retry is never retried.The scripts are covered by unit tests with a fake adb, including runs of the real
run-on-device.sh. None of this has run on a hosted emulator yet.Device prep: Play Store block and the Japanese dictionary
Two things made a hosted-emulator device run fail for reasons that had nothing to do with the editor. From about 150 seconds after boot, Play Store updated Play services, Chrome, WebView and Gboard on its own, and installing Play services killed Chrome mid-test. And the Japanese prep handed the device to the tests while Gboard could still be running its minimal Mozc engine. A session created before the dictionary loads never converts, so
kyouoffered only きょう and a test that commits 今日 failed on the device, not on the page. Neither failure said what it was.thumbjar preparenow disables Play Store before the tests. The step has its own cap, logs oneself-update block:line and never fails the prep. Gboard downloads its dictionary itself, not through Play Store, so the block does not hold the dictionary back.For
--keyboard jathe prep captures the device log from the start, then waits for Gboard'sNew data is loadedline for itsmozcdata-*.datafile, at most 360 seconds after the run script saw the emulator come online (--booted). Once it has loaded, the prep switches Gboard away from Japanese and back, requires aRELOADEDline after that switch, typeskyouand requires a kanji candidate. The check reuses the driver's ownopenKeyboard,ensureJapanese,dismissAnrandgboardKeys, so it gets the same ANR dismissal, Chrome foreground recovery, Space swipe and romaji QWERTY check as the tests. The only change intyping.tsis theexporton those four functions. A dictionary that never loaded fails the prep with a line that says so first, then names Gboard's version, its fallback engine and cancelled downloads.The 360-second deadline is on the host clock and the log lines are on the device clock. The wait converts once, from one whole-second device reading taken between two host readings, and takes the latest cutoff that reading allows. So a load inside 360 seconds always counts, and a load at most the adb round trip plus 1 second late can count too. A load later than that fails even when the wait first reads it after the deadline, which is stricter than the harness this was ported from. The log reader runs in its own process group, and every exit path (pass, fail, deadline, SIGTERM) stops it and reaps it, pinned by tests with a fake reader that ignores SIGTERM.
On the CI side, each row now carries its own keyboard's caps: 632 seconds of prep for English (the old 600 plus the Play Store block, nothing else changes) and 889 for Japanese. Every run step gets 60 seconds of named headroom for the action, shell and
pnpmstartup that no internal cap counts, so the step cannot end before the script logs its own reason. The run steps come to 53 and 57 minutes and the jobs to 155 and 163. A failed prep's reason goes into the job summary, and a prep failure on a reachable device is still never retried.Everything is covered by unit tests with a fake adb, including runs of the real
prepare.shandrun-on-device.sh. The dictionary check has not run on a hosted emulator yet. The first CI run is its first real test.Wait for a field that renders late
recordInputdocuments that a page that names no field may render the field after it runs, but the driver only waited forwindow.__state. A page that had runrecordInputwithout its field yet passed the readiness check, and the tap rect then readfield.styleonnull. That is a page exception, which is never retried, so a slow render failed the test as if the page were broken. The readiness check inwaitForPagenow also waits for the field, within the same 20 checks a second apart. A field that never renders fails withpage never rendered its field, a driver fault, so the attempt is retried. A page that never loads or never runsrecordInputstill fails withpage never loaded over CDP.That check is its own evaluation, and
cdpEvalpicks the page target by URL afresh for each expression, so a navigation or a re-render after it could still hand the tap rect or the forced focus a document without a field.FIELD_RECTandFOCUS_FIELDnow look the field up and returnnullwithout one in the same evaluation, and the driver evaluates each again a second later, up to 20 times, before it throws the same driver fault. An exception the page throws in either still fails the attempt at once. A document that replaces the page and has not runrecordInputby the time the driver readswindow.__state()still raises a page exception, whether it commits before or after the focus. The README says so. The tests run the realtypeInChromeagainst a mocked CDP and a jsdom page, and each fails on the code before the change.Stop deleting Chrome's saved tabs
The cleanup after each test, and after the Japanese prep's dictionary check, ran
rm -rfon Chrome'sapp_tabsdirectory. The CI emulator image runs adb as a non-root shell user, so the command failed with Permission denied on every job and the driver logged it and went on. The step never cleared anything, so it is removed, and the cleanup only stops Chrome. The tests in each device test file open distinct URLs, and the driver matches the page target by its exact URL, so a tab Chrome restores from another test does not match.The cleanup bound drops from 35 to 20 seconds, and the budgets derived from it drop with it: a
typeOnDevicecall from 535 to 520 seconds, the device test timeout from 575 to 545, and a file of three tests from 1825 to 1735. The workflow follows, with run steps of 52 and 56 minutes and jobs of 153 and 161 for English and Japanese.One case this does not cover: the Japanese prep's dictionary check opens
editor.html?prefix=, the same URL as the second Japanese test, so a tab restored from the prep could match that test's page. This is not new, because the removed step never succeeded on CI, and it is left for a follow-up.Note
Medium Risk
Changes Android CI orchestration and the shared device-typing harness used to guard editor input/composition behavior; production npm
filesstay editor-only, but mis-tuned retries or budgets could hide or flake real regressions.Overview
Removes
apps/android-imeand colocates Android Gboard emulator device tests underpackages/editor/android, with the typing driver, device prep, and Vitest hooks extracted into a new workspace packagethumbjar(typeOnDevice,recordInput,prepare/serveCLI,deviceProject).Test pages and specs now call
thumbjar(keyboard+ fullurlinstead oflang/page), and CI builds via@portabletext/editor build:androidandthumbjar fetch-u2-jar. Path filters and docs point atpackages/editor/androidandpackages/thumbjar.The Android IME workflow is reworked: the matrix is one job per device test file (
ja.test.ts/en.test.ts) with per-rowrun_minutes/job_minutesfrombudget.mjs(aligned bybudget.test.ts). Retries useemulator-runs.mjsfor boot failures and adb “device lost” after boot, withadb get-stateverdicts intiming.txtso infra failures are separated from test/prep failures.run-on-device.shtakes a single test file, runsthumbjar prepare, and records richer result lines for the job summary.Supporting updates:
vitest.android.config.ts, knip entries for editor android + thumbjar, rootcheck:knipwith thesourcecondition, and a hand-runen-textarea.test.tscontrol (not in the CI matrix).Reviewed by Cursor Bugbot for commit fbf00ea. Bugbot is set up for automated code reviews on this repo. Configure here.