Repository navigation
fix(rag): stop duplicating kept separators when merging chunks - #9488
Vidit Ostwal (Vidit-Ostwal) merged 1 commit into
Conversation
Strix Security ReviewWarning This pull request has 29 commits after the last Strix review ( ✅ No open security findings. PR summary: Fixes the RAG recursive text splitter so that when `keep_separator=True` it no longer inserts the separator a second time while merging chunks, and adds tests covering text, JSONL and chunk-boundary cases. Review summaryThis PR only changes how the recursive chunker rejoins split pieces: Review 1 on this PR · Reviewed by Strix |
SIDDARTHA REDDY (SIDDARTHAREDDY8)
left a comment
There was a problem hiding this comment.
I ran the new splitter code locally (it is stdlib-only, so I could execute it directly) and the three reported cases now round-trip exactly: "a\nb", "Para one.\n\nPara two." and the JSONL example each come back as a single chunk instead of doubling the separator. The fix is the right shape: with keep_separator=True the separator already lives inside each split, so joining with "" is correct, and the size accounting in _merge_splits is now consistent too. The old code had a special case that joined " "-separated splits with "" but still added len(" ") to total, so chunks were flushed early; I ran the old logic side by side and "a b c d e" at chunk_size=5 used to split as ["a b", " b c", " c d", " d e"] and now packs as ["a b c", " c d", " d e"].
One behavior worth a second look: continuation chunks start with a bare separator. With chunk_size=5 and overlap=0, "a\nb\nc\nd" splits into ["a\nb\nc", "\nc\nd"], and with overlap the same pattern repeats. That follows from _split_text_with_separator attaching the separator to the following piece, but a chunk that starts with "\n" (or " " in the space case) will be embedded and retrieved with that leading separator. Is that intended, or should _merge_splits strip a leading separator when a new chunk starts?
|
SIDDARTHA REDDY (@SIDDARTHAREDDY8) The leading separator on continuation chunks was already there before this change: it comes from |
RecursiveCharacterTextSplitter already attaches the separator to each piece when keep_separator is True, but _merge_splits joined the pieces with the separator again (only ' ' was special-cased). So 'a\nb' became 'a\n\nb', paragraph breaks doubled and JSONL records got blank lines. Merge with an empty string whenever separators are kept. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cd56da4 to
13c2a7b
Compare
Vidit Ostwal (Vidit-Ostwal)
left a comment
There was a problem hiding this comment.
Thanks, this is a clean, minimal fix: one join string in _merge_splits replaces both special-case branches, and the length accounting follows it.
Verified on base vs head: the issue's three examples and your new tests fail on base (['a\n\nb']) and pass on head. The keep_separator=False cases pass on both, so that path is unchanged.
Optional follow-up: a test with chunk_overlap > 0 and keep_separator=True would pin the overlap accounting. Existing stored chunks will differ on re-ingest, which is expected.
Thanks for your first contribution to CrewAI, Nurlan Mammadli (@nurlanmnn)!We really appreciate the time and care you put into this PR. Your work is now part of CrewAI — welcome to the contributor community. Want to share your contribution? Totally optional: Share on X · Share the PR on LinkedIn If the links don't work, feel free to copy and personalize this: If you share, we'd love to see it — tag crewAI (@crewAIInc) on X and CrewAI on LinkedIn. |
Related issue
Fixes #9487
Summary
When
keep_separator=True,RecursiveCharacterTextSplitter._split_text_with_separatoralready puts the separator back on each piece, but_merge_splitsjoined the pieces with the separator a second time (only" "was special-cased). That turned"a\nb"into"a\n\nb", doubled paragraph breaks and put blank lines between JSONL records._merge_splitsnow uses""as the join string whenever separators are kept, for both flush paths, and the length accounting follows the same join string. Withkeep_separator=Falsethe separator is still put back between pieces as before.Verification
tests/rag/test_data_types.py: TEXT and JSONL chunks match the source text, a chunk-boundary case, andkeep_separator=Falsestill rejoins with"\n","\n\n",", "and" ")pytest lib/crewai-tools/tests/rag: 357 passed;ruff check/ruff format --checkclean)Additional context
None.
🤖 Generated with Claude Code