Skip to content

fix(rag): stop duplicating kept separators when merging chunks - #9488

Merged
Vidit Ostwal (Vidit-Ostwal) merged 1 commit into
crewAIInc:mainfrom
nurlanmnn:fix-chunker-kept-separator
Nov 19, 2026
Merged

Vidit Ostwal (Vidit-Ostwal) merged 1 commit into
crewAIInc:mainfrom
nurlanmnn:fix-chunker-kept-separator

Conversation

@nurlanmnn

Copy link
Copy Markdown
Contributor

Related issue

Fixes #9487

Summary

When keep_separator=True, RecursiveCharacterTextSplitter._split_text_with_separator already puts the separator back on each piece, but _merge_splits joined the pieces with the separator a second time (only " " was special-cased). That turned "a\nb" into "a\n\nb", doubled paragraph breaks and put blank lines between JSONL records.

_merge_splits now uses "" as the join string whenever separators are kept, for both flush paths, and the length accounting follows the same join string. With keep_separator=False the separator is still put back between pieces as before.

Verification

  • Tests added or updated for the changed behavior (tests/rag/test_data_types.py: TEXT and JSONL chunks match the source text, a chunk-boundary case, and keep_separator=False still rejoins with "\n", "\n\n", ", " and " ")
  • Relevant tests and quality checks pass locally (pytest lib/crewai-tools/tests/rag: 357 passed; ruff check / ruff format --check clean)

Additional context

None.

🤖 Generated with Claude Code

@strix-security

strix-security Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Strix Security Review

Warning

This pull request has 29 commits after the last Strix review (cd56da4). Strix has not reviewed these changes.
Automatic review on push is off for this repository. To review the latest changes, tag @strix-security in a comment, or turn on re-review on push.

✅ No open security findings.

PR summary: Fixes the RAG recursive text splitter so that when `keep_separator=True` it no longer inserts the separator a second time while merging chunks, and adds tests covering text, JSONL and chunk-boundary cases.

Review summary

This PR only changes how the recursive chunker rejoins split pieces: _merge_splits now uses an empty join string whenever separators are already kept on the pieces, instead of special-casing only a single space. I read the full splitter module and traced every separator branch (keep_separator on/off, empty separator, recursive sub-splits, character splitting, and the overlap accounting). The method only performs string joins and length arithmetic, handles no attacker-controlled input at any dangerous sink, terminates on every path, and its output stays bounded by the input size, so the change introduces no injection, resource-exhaustion, data-loss or fail-open behaviour. The rest of the diff is test-only assertions about chunk content. No security issues found in the changed code.

Review 1 on this PR · cd56da4 · requested Nov 16 16:52 UTC when the PR was opened


Reviewed by Strix
Re-run review · Configure security review settings · Try a deeper review with @strix-security /deep

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran the new splitter code locally (it is stdlib-only, so I could execute it directly) and the three reported cases now round-trip exactly: "a\nb", "Para one.\n\nPara two." and the JSONL example each come back as a single chunk instead of doubling the separator. The fix is the right shape: with keep_separator=True the separator already lives inside each split, so joining with "" is correct, and the size accounting in _merge_splits is now consistent too. The old code had a special case that joined " "-separated splits with "" but still added len(" ") to total, so chunks were flushed early; I ran the old logic side by side and "a b c d e" at chunk_size=5 used to split as ["a b", " b c", " c d", " d e"] and now packs as ["a b c", " c d", " d e"].

One behavior worth a second look: continuation chunks start with a bare separator. With chunk_size=5 and overlap=0, "a\nb\nc\nd" splits into ["a\nb\nc", "\nc\nd"], and with overlap the same pattern repeats. That follows from _split_text_with_separator attaching the separator to the following piece, but a chunk that starts with "\n" (or " " in the space case) will be embedded and retrieved with that leading separator. Is that intended, or should _merge_splits strip a leading separator when a new chunk starts?

@nurlanmnn

Copy link
Copy Markdown
Contributor Author

SIDDARTHA REDDY (@SIDDARTHAREDDY8) The leading separator on continuation chunks was already there before this change: it comes from _split_text_with_separator attaching the separator to the following piece. Running the old code on "a\nb\nc\nd" with chunk_size=5 gives ['a\n\nb', '\nb\n\nc', '\nc\n\nd'], so only the doubled separator goes away here. I kept this PR to the duplication fix; stripping the leading separator would change chunk contents more broadly, so I'd leave that call to the maintainers.

RecursiveCharacterTextSplitter already attaches the separator to each
piece when keep_separator is True, but _merge_splits joined the pieces
with the separator again (only ' ' was special-cased). So 'a\nb' became
'a\n\nb', paragraph breaks doubled and JSONL records got blank lines.
Merge with an empty string whenever separators are kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this is a clean, minimal fix: one join string in _merge_splits replaces both special-case branches, and the length accounting follows it.

Verified on base vs head: the issue's three examples and your new tests fail on base (['a\n\nb']) and pass on head. The keep_separator=False cases pass on both, so that path is unchanged.

Optional follow-up: a test with chunk_overlap > 0 and keep_separator=True would pin the overlap accounting. Existing stored chunks will differ on re-ingest, which is expected.

@Vidit-Ostwal
Vidit Ostwal (Vidit-Ostwal) merged commit 1096c2a into crewAIInc:main Nov 19, 2026
55 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

Thanks for your first contribution to CrewAI, Nurlan Mammadli (@nurlanmnn)!

We really appreciate the time and care you put into this PR. Your work is now part of CrewAI — welcome to the contributor community.

Want to share your contribution? Totally optional:

Share on X · Share the PR on LinkedIn

If the links don't work, feel free to copy and personalize this:

I just made my first contribution to @crewAIInc!

Excited to help build the future of AI agents with CrewAI.

https://github.com/crewAIInc/crewAI/pull/9488

#OpenSource #AI #CrewAI

If you share, we'd love to see it — tag crewAI (@crewAIInc) on X and CrewAI on LinkedIn.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] RAG text chunker doubles every newline it keeps (keep_separator=True)

3 participants