Repository navigation
fix(rag): stop duplicating separators when keep_separator is enabled - #9500
电大老baby (HhH666-ux) wants to merge 2 commits into
Conversation
Strix Security ReviewWarning This pull request has 4 commits after the last Strix review ( ✅ No open security findings. PR summary: Fixes the recursive character splitter so that, when `keep_separator` is enabled, merged chunks no longer duplicate the separator (which was doubling newlines), and adds tests covering the fixed behaviour plus the unchanged `keep_separator=False` path. Review summaryThe change is confined to Ruled out (3)
Review 1 on this PR · Reviewed by Strix |
Nick (surlymochan)
left a comment
There was a problem hiding this comment.
I reviewed this as a non-maintainer (verification only, no approval). I read the full diff and the new tests against the current HEAD.
Verified:
- The fix is correct as written: with
keep_separatorthe split step already attaches the separator to each piece, so joining with it again in_merge_splitsduplicated every break. Usingjoin_with = ""in that case, and keeping the separator otherwise, fixes the duplication, and the overlap accounting (total -= len(join_with)) now matches the length actually joined. - The new tests cover the reported cases (
"a\nb","Para one.\n\nPara two.", JSONL lines) and thekeep_separator=Falserejoin case.
One thing the maintainers should be aware of:
- #9488 (opened 2026-11-16, same file and function) implements the same behavior change — it normalizes
separator = "" if self._keep_separator else separatorat the top of_merge_splitsand reuses it through the same three join sites. The two PRs overlap and cannot both land as-is. #9500's tests live in a dedicated newtest_chunkers.py, while #9488's are parametrized by data type intest_data_types.py. It would help to decide which one to take forward (or rebase one onto the other) before either merges.
Fixes #9487
With
keep_separator=True(the default),_split_text_with_separatoralready attaches the separator to every split, but_merge_splitsjoined the splits with the separator again. Only" "was special-cased, so newlines were doubled._merge_splitsnow joins with an empty string wheneverkeep_separatoris on (or the separator is""), and the length accounting uses the same joiner.keep_separator=Falseis unchanged."a\nb"['a\n\nb']['a\nb']"Para one.\n\nPara two."['Para one.\n\n\n\nPara two.']['Para one.\n\nPara two.']'{"a": 1}\n{"b": 2}'['{"a": 1}\n\n{"b": 2}']['{"a": 1}\n{"b": 2}']Tests
tests/rag/test_chunkers.pycovers the three cases above plus thekeep_separator=Falsepath.