Skip to content

fix(integration): re-centre every sample's EIC when Apply changes the tR - #302

Merged
Jack-Coutts merged 5 commits into
mainfrom
fix/264-recentre-eics-on-tr-change
Nov 10, 2026
Merged

Jack-Coutts merged 5 commits into
mainfrom
fix/264-recentre-eics-on-tr-change

Conversation

@Jack-Coutts

@Jack-Coutts Jack-Coutts commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #264. Fixes #188.
Re-applies #266 onto main, with the session-import half of #267.

The bug

After a tR change, plots of one compound could be centred on different tRs, even when every sample had the same tR. Apply re-extracted a sample only when the new integration window fell outside its stored EIC, with 0.001 min of slack. A tR change inside the stored data left the plot where it was. Sub-second differences in each file's first scan decided which samples moved. Session import used the same test, so an imported session showed the same split.

The test read each sample's data range from a cache in the widget. That cache paired sample names with EICs by position, so one sample with no EIC shifted every later sample's bounds onto the wrong name (#188).

The fix

Each eic row now stores the tR it was extracted at, in a new extraction_rt column. Import, Apply, the tR Window update and the mass tolerance reload all write it.

One check in eic_importer.py, eic_needs_reextraction, is used by Apply and by reextract_session_eics. A sample is re-extracted unless its stored extraction_rt equals its new tR and its stored tR Window covers both offsets. A sample with no EIC is always re-extracted.

  • A tR change re-extracts every sample it applies to, so every plot is centred on its tR.
  • An offset change inside the tR Window re-extracts nothing, as before.
  • An offset wider than the tR Window re-extracts, as before.
  • Reset restores the compound tR without re-extracting. The next Apply sees the stored tR differs and re-centres it.
  • When the tR Window field shows a range, each re-extracted sample keeps its own window, raised only to cover its own offsets. regenerate_compound_eics takes a window per sample, as it already took a tR per sample. Session import does the same. With one value in the field, nothing changes.

Why store the tR instead of reading it from the stored scans: an earlier version of this branch judged "centred" from the first and last stored scan and the median scan interval. With irregular scans that skipped a needed re-extraction, and a trace clipped by the start or end of the run failed the check on every Apply. Comparing the stored tR is exact and needs no tolerance. Both values come from the same parsed field or the same database value, so they compare equal when the tR has not changed.

The tR Window itself is not compared. Apply never changes the window unless an offset outgrows it, and the tR Window update re-extracts every sample on its own.

Older databases get the column through _run_migrations. Their EICs have no stored tR, so the first Apply on each compound re-extracts them once.

The per-sample bounds cache, its refresh after regeneration, calculate_integration_boundaries and check_boundaries_within_window are deleted. This closes #188.

Unlike #266, the old "Regeneration Complete" box is not touched: main already shows a toast.

Cost

One Apply that moves Lactate's tR by 0.02 min on the bench's 30 labelled samples: 2.0 ms on main (re-centres 0 of 30) and 70 ms here (re-centres 30 of 30). Median of 5 alternating runs per side, range 67 to 112 ms. read_cdf_file takes about two thirds of it. The machine was heavily loaded during the runs, so treat the range as wide. These were measured before the check stopped decompressing stored traces, which only makes the check cheaper.

Results

bench/reference.py check passes. The reference never changes a tR, so it cannot see this change.

Exported areas after a tR change do move. With deconvolution on, the fit uses the whole stored trace, so the area depends on where the trace was cut. I applied tR moves to Pyruvate, Lactate, Glutamate and Valine on the bench data and exported from main and from this branch. Pyruvate moved by about 0.01%. Lactate moved by a median of 0.14% and up to 5%. Two Glutamate samples moved by up to 17% in raw values. Valine did not move. I then set those four tRs in the compound list, extracted fresh, and exported. That export matches this branch in every area. Only the header row that prints the compound-list tR differs. So after this change, a tR set with Apply gives the same areas as the same tR in the compound list. On main it depended on edit history.

Checks

tests/test_apply_recentres_eics.py drives the real IntegrationWindow on real CDF files.

  • Two files whose scans sit half a scan apart. Without the fix: assert [(4.8, 5.2), (4.8, 5.2)] == [(4.9, 5.3), (4.9, 5.3)], so the plots stay on the old tR. With it, both re-centre, a narrower offset re-extracts nothing, and an offset past the window re-extracts.
  • S1 holds a 0.2 min window and S2 a 0.4 min one. Apply moves the tR from 5.0 to 5.02. Each keeps its window. On the previous head: {'S1': 0.4} != {'S1': 0.2}.
  • S1's window is narrower than the new right offset, S2's run starts after its window opens, and S3 has no EIC. Apply re-extracts only S1 and S3. On the previous head: [['S1', 'S2', 'S3']] == [['S1', 'S3']].
  • Irregular scans (4.799, then 4.81 to 5.19 every 0.02) extracted at tR 5.0. A move to 4.995 must re-extract, since a fresh extraction takes the 4.799 scan. On the previous head: [] == ['S1'].

The session import test has a sample nudged inside its stored data. Without the fix only Moved is re-extracted. With it, Moved and Nudged are, each at its own window, and Kept is not.

  • uv run python -m pytest -q: 652 passed.
  • On the bench labelled data, an Apply that keeps every compound's tR and offsets as the fields show them re-extracts 0 of 2,850 EICs.
  • An eic table from main's schema gains extraction_rt on start, and its rows are marked for re-extraction.

Left out

  • _populate_range_fields still reads every EIC only to check that one exists. Left alone to stay in scope.

Apply and session import now re-extract a sample unless extracting its
tR Window around its tR would return the stored trace unchanged and that
window covers both offsets. This replaces the coverage test with 0.001 min
of slack and the per-sample bounds cache, which stored bounds under the
wrong sample when one had no EIC.
Conflicts:
- docs/01_user_guide.md: main's session-import step and backward-compatibility
  text, with this branch's sentence on when an import re-extracts.
- integration_window_widget.py: keep main's _stored_tr_windows (#298); drop
  check_boundaries_within_window, which this branch replaces with
  eic_needs_reextraction.
- test_rt_window_centering.py: regeneration now refreshes only the tR Window
  field, plots and charts (no bounds refresh here, no fields refresh after #289).

test_session_apply_refresh.py (#289) edited tR on fake CDF paths and relied on
Apply not re-extracting. With this branch a tR change re-extracts, the read
fails, and the error box blocks the test. Its two tests now edit the left
offset inside the tR Window on EICs stored centred on the tR, which keeps
Apply on the no-re-extraction path #289 guards.
…EIC was cut around

Apply with a tR Window range re-extracted every sample at the widest
stored window. Each re-extracted sample now keeps its own window, raised
only to cover its own offsets. Session import does the same.

The re-extraction check no longer guesses from the stored scans. Each EIC
row stores the tR it was extracted at (eic.extraction_rt, added by a
migration; NULL in older databases means re-extract). A sample is
re-extracted unless that tR equals its new tR and its stored window
covers both offsets. This fixes irregular scans that skipped a needed
re-extraction and runs clipped at the start or end that re-extracted on
every Apply.
@Jack-Coutts

Copy link
Copy Markdown
Collaborator Author

Review verdict for head a64ae13: CLEAN after one fix round. Reviewed by GPT-6.1 Sol. Before the review, main (42e3562) was merged into this branch in e5bd61f. The merge kept eic_needs_reextraction, the removal of the bounds cache, #298's tR Window range handling and #289's timer-free refresh. It also updated two #289 tests, which put a tR change on fake CDF paths and would otherwise hang on a modal error box.

Round 1, at e5bd61f:

  • The main claim holds. Sol applied tR 5.0, then 5.02, then 5.04 on two samples with deconvolution on, and separately imported fresh at 5.04. The two exports matched in every result cell. On main the same two exports differ by up to 0.00733% on this synthetic data.
  • Blocker: with the tR Window field showing a range (S1 at 0.2, S2 at 0.4), a small tR edit re-extracted both at 0.4. main keeps 0.2 and 0.4 for that edit.
  • Warning: the "already centred" check, which judged the stored trace by median scan spacing, skipped a needed re-extraction on irregular scans. It also re-extracted on every Apply for traces clipped by the start or end of the run.
  • Test gaps from deleted tests: selective reload, and a sample with no stored EIC.
  • Suite: 649 passed. Results check: no changed cells.

Fix, e5bd61f..a64ae13:

  • A tR Window range now sends each sample its own stored window, raised only to cover its own offsets. Session import does the same.
  • A new eic.extraction_rt column, with a migration where NULL means re-extract, records the tR each EIC was cut around. The check is now exact: re-extract unless extraction_rt equals the tR and the stored window covers both offsets.
  • Four new tests.

Round 2, the fix diff only:

  • S1 and S2 keep 0.2 and 0.4 for small and large moves. Only the sample that left its EIC reloads, and a missing EIC is queued without reloading the others. Irregular-scan and run-clipped traces behave correctly.
  • The new tests fail against e5bd61f's source.
  • The 5.0, 5.02, 5.04 export still matches a fresh extraction at 5.04 in every result cell.
  • The exact tR comparison holds on every path that supplies a tR. A tR imported with more than three decimals re-extracts once, on the first unchanged Apply, then stays put. An unchanged Apply re-extracts 0 of 2,850 bench EICs.
  • A live MainWindow to EicRegenerationWorker probe carried the per-sample window dict through the thread. The scalar path for Apply window still works.
  • Suite: 652 passed. Results check: both datasets matched.

Merge note: #304 and #308 edit the same EIC inserts. When resolving, keep the sixth batch value (the extraction tR) and the six-placeholder INSERT, so the mass tolerance reload records extraction_rt for every EIC it writes.

@Jack-Coutts
Jack-Coutts merged commit 0e81338 into main Nov 10, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Plots of one compound are centred on different tRs after a tR change Integration window stores data bounds under the wrong sample

1 participant