Repository navigation
feat(qmd): tree-sitter AST break points for code files - #100
Merged
Merged
Conversation
Adds the `ast` feature (default on) with `chunk::ast`: language detection for upstream's 11 extensions, LANGUAGE_QUERIES copied verbatim, upstream scores and strictly-highest dedup per position. Grammars are pinned to the exact versions upstream installs (typescript 0.23.2 — also used for JavaScript — python/go 0.25.0, rust 0.24.0) on the tree-sitter 0.26 runtime. Positions are converted from tree-sitter byte offsets to UTF-16 indices, which is what web-tree-sitter's startIndex reports, so non-ASCII files match too. A new upstream-recorded golden (18 files: every extension and capture, invalid syntax, empty, CJK, emoji, markdown, unknown extension) replays with zero mismatches. qmd-cli/qmd-mcp now depend on qmd with explicit features so `--no-default-features` really drops tree-sitter. Also folds in the remaining P5 test gaps: named-index daemon data, named + default daemons side by side, --daemon over stale and foreign pidfiles, single EOF breadcrumb, EPIPE on stderr, encodeQmdPath. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
fjh2333
added
enhancement
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
astfeature onqmd(default on; forwarded byqmd-cli) withchunk::ast:detect_language(11 extensions → 6 languages, case-insensitive,qmd://paths) andast_break_points.LANGUAGE_QUERIEScopied verbatim (ast.ts:94-151); upstream scores; highest score per position; sorted. Unsupported, empty and invalid input returns no break points.tree-sitter-typescript=0.23.2 (also used for JavaScript, as upstream does),tree-sitter-python/-go=0.25.0,tree-sitter-rust=0.24.0, ontree-sitter0.26.13 (upstreamweb-tree-sitter0.26.12).startIndexreports, so non-ASCII files match upstream exactly.qmd-cli/qmd-mcpdepend onqmdwith explicit features (llama), so--no-default-featuresdrops tree-sitter.extractSymbols(upstream stub with no callers) and the grammar-load repair message (grammars are statically linked, so that failure cannot happen).--daemonover stale and foreign-PID pidfiles, single EOF message, EPIPE on stderr,encodeQmdPath.Related issue
Closes #99
How it was tested
scripts/gen_ast_golden.tsrecorded upstreamgetASTBreakPointson 18 files (every extension and capture kind, invalid syntax, empty, CJK, emoji, markdown, unknown extension); in-crate replay has zero mismatches.ast.test.tscases + UTF-16 conversion tests; 7 new P5 gap tests.just allgreen;cargo denyclean;qmdandqmd-clibuild with--no-default-features. No newunsafein our code.chunk::asthas no callers yet; P6.2 will report the measured delta.