Repository navigation
fix: flush byte fallback runs at end of input - #414
Merged
pcuenca merged 5 commits intoOct 28, 2026
Merged
Conversation
pcuenca
reviewed
Oct 26, 2026
pcuenca
reviewed
Oct 26, 2026
pcuenca
approved these changes
Oct 28, 2026
pcuenca
added a commit
that referenced
this pull request
Oct 28, 2026
Confirmed locally against the Python reference.
pcuenca
added a commit
that referenced
this pull request
Oct 28, 2026
* [CI] Test offline tokenizers tests on Linux * Fix expectation after #414 Confirmed locally against the Python reference.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ByteFallbackDecoder flushes buffered bytes only when an ordinary token follows. A sequence ending in byte-fallback pieces therefore loses its last character: decoding the four UTF-8 pieces for an emoji returns an empty string.
Flush the remaining byte buffer at end of input using the same UTF-8 conversion as an interior run. This keeps decoding within the configured pipeline, so subsequent Fuse/Strip stages still apply normally.
Adds five in-memory regressions covering terminal emoji, incomplete UTF-8, mixed interior/terminal runs, downstream cleanup, and PreTrainedTokenizer decoding with special-token filtering.
Validation on macOS / Swift 6.4, based on c21fdcd:
swift test --jobs 2 --no-parallel --filter ByteFallbackEndOfInputTestsThis was a focused CPU-only package run using the actual decoder factory and tokenizer implementation. No pretrained-model downloads, GPU workloads, benchmarks, or full upstream test matrix were run.