Skip to content

Add optional extract function to variables - #3

Merged
ronisbr merged 2 commits into
JuliaSpace:mainfrom
RianKoja:feat/extract-callback
Oct 22, 2026
Merged

ronisbr merged 2 commits into
JuliaSpace:mainfrom
RianKoja:feat/extract-callback

Conversation

@RianKoja

@RianKoja RianKoja commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Adds an optional extract field to TelemetryVariableDescription (and keyword to add_variable!) for variables whose location in the frame depends on the packet content.

Motivation

In many telemetry formats, the same parameter can sit at different offsets depending on a selector in the packet, such as the APID, a frame format code, or a subcommutation counter. A fixed position and size can describe only one of those placements, and btf only receives that slice, so it cannot check the selector.

Behavior

add_variable!(db, :param, 0, 0, tf, btf, rtf;
              extract = frame -> frame[1] == 0x01 ? view(frame, 3:4) : nothing)
  • extract(frame) receives the whole unpacked frame and returns the variable frame passed to btf, already in the variable byte order (endianness is not applied).
  • Returning nothing means the variable is not in this packet: every view of it is missing, its transfer functions are not called, and dependents see (raw = missing, processed = missing) in the context.
  • Extract-backed variables must use position = 0 and size = 0.

Backward compatibility

The change is intended as a minor release (3.1.0):

  • The new field defaults to nothing, so the keyword constructor is unchanged, and the previous 11-argument positional constructor is kept as an outer constructor (the test suite itself uses it).
  • add_variable! gains one keyword with a default. Existing calls are unchanged.
  • ExecutionNode is unchanged. Variables with extract are wrapped in a new internal ExtractExecutionNode, so variables without it follow the same path as before.
  • Persistence is unaffected: save_telemetry serializes packets, not variable descriptions.

Checked:

  • All 263 existing tests pass unmodified; 12 new tests cover extract (present and absent, every view, dependents, validation, the 11-argument constructor).
  • Tests pass on Julia 1.10 and 1.12.
  • A downstream database with about 2500 variables decoded the same set of about 1100 real packets into identical DataFrames (isequal per column) with 3.0.1 and with this branch.
  • benchmark/benchmarks.jl (best of two runs each, 1 thread): every scenario is within 4% of main except packets-128, which is 15% faster, and allocation counts are equal or lower. Plan building for variables without extract costs the same as on main.
  • With extract, decoding 2000 packets x 32 variables (half of them absent) takes about 6% longer than the same variables read by position.

The version in Project.toml is left for the release commit.

A variable can set `extract`, a function that receives the whole unpacked
frame and returns the variable frame, or `nothing` when the variable is not
in that packet. Absent variables are `missing` in every view and their
transfer functions are not called.

This supports databases where a parameter's location depends on the packet
content (APID, frame format, subcommutation counter), which a fixed
`position` and `size` cannot express.

The field defaults to `nothing`, the 11-argument positional constructor is
kept, and variables without `extract` follow the same code path as before.
Adding a type parameter to ExecutionNode made plan building about 0.4 us
slower per variable, even for variables without an extract callback. Those
variables now use the unchanged ExecutionNode, and only extract-backed ones
are wrapped in an ExtractExecutionNode. Nodes are pushed from each branch so
plan building does not test which node type it holds.
@RianKoja
RianKoja marked this pull request as ready for review October 21, 2026 01:40
@ronisbr

ronisbr commented Oct 22, 2026

Copy link
Copy Markdown
Member

Thanks @RianKoja !

@ronisbr
ronisbr merged commit 82ab66f into JuliaSpace:main Oct 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants