Skip to content

fix(rag): stop duplicating separators when keep_separator is enabled - #9500

Closed
电大老baby (HhH666-ux) wants to merge 2 commits into
crewAIInc:mainfrom
HhH666-ux:fix/chunker-keep-separator-duplication
Closed

电大老baby (HhH666-ux) wants to merge 2 commits into
crewAIInc:mainfrom
HhH666-ux:fix/chunker-keep-separator-duplication

Conversation

@HhH666-ux

Copy link
Copy Markdown
Contributor

Fixes #9487

With keep_separator=True (the default), _split_text_with_separator already attaches the separator to every split, but _merge_splits joined the splits with the separator again. Only " " was special-cased, so newlines were doubled.

_merge_splits now joins with an empty string whenever keep_separator is on (or the separator is ""), and the length accounting uses the same joiner. keep_separator=False is unchanged.

input before after
"a\nb" ['a\n\nb'] ['a\nb']
"Para one.\n\nPara two." ['Para one.\n\n\n\nPara two.'] ['Para one.\n\nPara two.']
JSONL '{"a": 1}\n{"b": 2}' ['{"a": 1}\n\n{"b": 2}'] ['{"a": 1}\n{"b": 2}']

Tests

  • New tests/rag/test_chunkers.py covers the three cases above plus the keep_separator=False path.

@strix-security

strix-security Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Strix Security Review

Warning

This pull request has 4 commits after the last Strix review (4e103d8). Strix has not reviewed these changes.
Automatic review on push is off for this repository. To review the latest changes, tag @strix-security in a comment, or turn on re-review on push.

✅ No open security findings.

PR summary: Fixes the recursive character splitter so that, when `keep_separator` is enabled, merged chunks no longer duplicate the separator (which was doubling newlines), and adds tests covering the fixed behaviour plus the unchanged `keep_separator=False` path.

Review summary

The change is confined to RecursiveCharacterTextSplitter._merge_splits in the RAG chunker: it hoists the join string into join_with and uses an empty join whenever keep_separator is on, so the separator attached during splitting is not inserted a second time (and the running length accounting now matches the actual join). I traced the full ingest path — loader output into BaseChunker.chunk, through _split_text, _split_text_with_separator and _merge_splits, then into embedding and vector-store writes — and confirmed the change only shapes chunk text: no reference, path, URL, metadata, credential or parser handling is touched, and no new sink is reached. The path/URL validation and metadata assignment that surround ingestion live in the untouched callers and behave identically. Chunk size bounds are unchanged (bounded by chunk_size plus the longest single split, as before), so the altered join does not create an unbounded chunk or a new denial-of-service path. No security issues found.

Ruled out (3)
  • Can the changed merge logic let RAG ingestion accept a source path or URL that the caller's validation would otherwise reject?
    The path and URL checks in `RagTool.add` run before the loader is invoked and resolve the reference itself, while `_merge_splits` only joins text pieces the loader has already returned, so the input the loader sees is unchanged by this PR.
  • Does joining split pieces with an empty string change what content is embedded or how chunk metadata is written?
    When `keep_separator` is on the separator is already present in each split, so the merged chunk holds the same text minus the duplicate, and the `base_metadata`/`chunk_index` assignment in the adapter is untouched.
  • Can the new `join_with` accounting let a chunk grow past `chunk_size` without bound, letting a crafted document exhaust memory or embedding quota?
    Chunk length remains bounded by `chunk_size` plus the longest single split — measured at 100 characters for `chunk_size=50` on line-oriented text — which is the same bound the previous code produced.

Review 1 on this PR · 4e103d8 · requested Nov 17 03:11 UTC when the PR was opened


Reviewed by Strix
Re-run review · Configure security review settings · Try a deeper review with @strix-security /deep

@surlymochan Nick (surlymochan) left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed this as a non-maintainer (verification only, no approval). I read the full diff and the new tests against the current HEAD.

Verified:

  • The fix is correct as written: with keep_separator the split step already attaches the separator to each piece, so joining with it again in _merge_splits duplicated every break. Using join_with = "" in that case, and keeping the separator otherwise, fixes the duplication, and the overlap accounting (total -= len(join_with)) now matches the length actually joined.
  • The new tests cover the reported cases ("a\nb", "Para one.\n\nPara two.", JSONL lines) and the keep_separator=False rejoin case.

One thing the maintainers should be aware of:

  • #9488 (opened 2026-11-16, same file and function) implements the same behavior change — it normalizes separator = "" if self._keep_separator else separator at the top of _merge_splits and reuses it through the same three join sites. The two PRs overlap and cannot both land as-is. #9500's tests live in a dedicated new test_chunkers.py, while #9488's are parametrized by data type in test_data_types.py. It would help to decide which one to take forward (or rebase one onto the other) before either merges.

@github-actions github-actions Bot closed this Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] RAG text chunker doubles every newline it keeps (keep_separator=True)

2 participants