Skip to content

embed: QueryModel moves into mcp-kit (0.7.0) - #22

Merged
shtse8 merged 2 commits into
mainfrom
feat/shared-splitter
Oct 2, 2026
Merged

shtse8 merged 2 commits into
mainfrom
feat/shared-splitter

Conversation

@shtse8

@shtse8 shtse8 commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Moves lockdocs' query-side embedder into mcp_kit::embed so there is one owner of the WordPiece and identifier splitting.

  • New embed::QueryModel (open(dir), embed(text)): reads only the vocabulary and the rows of the query's own tokens with positioned reads; same vector as Model::embed under Tokenization::Identifiers.
  • Model and QueryModel now share one wordpiece routine, one identifier_words, one Fx hasher.
  • Test: query_model_matches_the_full_model (moved from lockdocs, plus a truncated-file case).
  • Version 0.7.0, CHANGELOG entry. Additive.

Checks: cargo test --workspace --locked --all-features, clippy -D warnings. Needs review; not armed.

@shtse8
shtse8 enabled auto-merge (squash) October 2, 2026 22:54
@shtse8
shtse8 merged commit c415811 into main Oct 2, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant