Skip to content

fix(rag): keep oversized AST chunk IDs unique - #1546

Open
embrace897 wants to merge 1 commit into
VoltAgent:mainfrom
embrace897:fix/rag-ast-chunk-ids
Open

embrace897 wants to merge 1 commit into
VoltAgent:mainfrom
embrace897:fix/rag-ast-chunk-ids

Conversation

@embrace897

@embrace897 embrace897 commented Nov 15, 2026 •

Copy link
Copy Markdown

PR Checklist

  • The proposed commit message follows Conventional Commits.

Bugs / Features

  • Related issue(s) linked. Reproduced directly on main.
  • Tests for the changes have been added.
  • Docs have been added or updated.
  • A patch changeset for @voltagent/rag has been added.

What is the current behavior?

When an oversized AST block is split by the token chunker, its output IDs add both the callback index and the current output length. Later AST blocks can reuse an ID already emitted by the split.

What is the new behavior?

Derive each split chunk ID from the number of chunks already emitted. The regression covers a large AST block and two subsequent small blocks.

Notes for reviewers

  • The upstream fixture produces four unique IDs for five chunks. The patched regression also checks the exact sequence from code-0 through code-4.
  • vitest run --typecheck passed 48 RAG tests across 20 files.
  • Scoped TypeScript, changed-file Biome, changeset Prettier and whitespace checks passed.
  • CommonJS, ESM and type declaration builds passed.

OpenAI Codex assisted with reproduction, implementation and local validation.


Summary by cubic

Fixes duplicate chunk IDs when token splitting expands an oversized AST block before other blocks. Split chunks now derive their ID from the number of chunks already emitted instead of combining the callback index with the output length, so later blocks can no longer reuse an ID from the split.

Adds a regression test for a large AST block followed by two smaller blocks, plus a patch changeset for @voltagent/rag.

Written for commit ca72ade. Summary will update on new commits.

View guided diff

Summary by CodeRabbit

  • Bug Fixes
    • Fixed duplicate code chunk IDs when a large syntax block is split into multiple chunks before other blocks. Chunk IDs now remain unique and sequential.

@changeset-bot

changeset-bot Bot commented Nov 15, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: ca72ade

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
@voltagent/rag Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@coderabbitai

coderabbitai Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The AST code chunker now assigns token-based chunk IDs from the current output length. A regression test checks that IDs remain sequential and unique when an oversized AST block is split before two smaller blocks.

Changes: AST chunk ID assignment

Layer / File(s) Summary
Chunk ID assignment and regression coverage
packages/rag/src/chunkers/code-chunker.ts, packages/rag/src/chunkers/code-chunker-ast-ids.spec.ts, .changeset/unique-code-ast-chunk-ids.md
The chunker assigns each token chunk an ID based on the current output length. The regression test checks five sequential, unique IDs. A changeset declares a patch release for @voltagent/rag.

Estimated code review effort: 2 (Simple) | ~8 minutes

Change: Bug fix

Suggested reviewers: omeraplak

Merge Risk: Minimal · up to ca72a

This fixes duplicate chunk IDs when an oversized code block is split. The change is small and covered by a regression test, so merge risk is minimal.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title names the main change: a fix in the rag package so that IDs for oversized AST chunks stay unique. The changeset, the regression test, and the one-line change in code-chunker.ts all match thi…
Description check Passed The description follows the repository template. It includes the PR checklist, the current and new behavior, and reviewer notes. The author checked the test and changeset items. Two items stay uncheck…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title names the main change: a fix in the rag package so that IDs for oversized AST chunks stay unique. The changeset, the regression test, and the one-line change in code-chunker.ts all match this. The title follows the conventional commit style and is 45 characters long.

Full details: Description check

Explanation

The description follows the repository template. It includes the PR checklist, the current and new behavior, and reviewer notes. The author checked the test and changeset items. Two items stay unchecked: related issue linking (the author notes the bug was reproduced on main but gives no issue) and docs. The template also expects a 'fixes (issue)' line, which is missing. The description is mostly complete, so the check passes.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: edd2418f-a082-4691-9373-8bb469d61c2d
Commits

Reviewing files that changed from the base of the PR and between e9fc599 and ca72ade.

Files selected for processing: 3

  • .changeset/unique-code-ast-chunk-ids.md
  • packages/rag/src/chunkers/code-chunker-ast-ids.spec.ts
  • packages/rag/src/chunkers/code-chunker.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 8 remain after this review.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 3 files

View guided diff | Re-trigger cubic

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant