[jeethu]

While profiling model load in an app that uses swift-transformers, I found that AutoTokenizer.from(modelFolder:) is most of the wall time for large BPE vocabularies. Gemma 4's tokenizer.json (31 MB, 262,144 vocab entries, 514,906 merges) takes about 1.4 s on an M-series Mac in a Release build, and about 2.9 s in Debug. This PR cuts that by roughly 2.3x without changing the loaded data. It is a small change, and I'm happy to rework it however you prefer.

What the profile showed (Time Profiler, Release): about 64% of the time is in BPETokenizer.init, mostly building bpeRanks.

• BinaryDistinctString.string was String(nsString) where nsString is String(utf16CodeUnits:count:) as NSString. Every vocab key and merge half therefore reached bpeRanks and the token dictionaries as a bridged (foreign) String, so each hash and == went through the UTF-16-via-CoreFoundation path, and non-ASCII keys also paid for normalization in the hash.
• BinaryDistinctString.init(_: NSString) was Array(str as String).flatMap { $0.utf16 }, which splits the string into grapheme clusters only to concatenate their UTF-16 again.
• bpeRanks grew from empty to about 515k entries with no reserveCapacity.

Changes:

• BinaryDistinctString.string is now String(decoding: value, as: UTF16.self) (a native String).
• init(_: NSString) and init(_: String) take .utf16 directly.
• nsString returns string as NSString when the UTF-16 is well formed, and falls back to the previous construction otherwise. This keeps lone surrogates exactly as before: String(decoding:) would turn one into U+FFFD, which would change which NSString a vocabulary key is. (A lone \ud800 escape is rejected by yyjson, so a tokenizer.json cannot carry one, but a Config built in code can.)
• BPETokenizer.init calls bpeRanks.reserveCapacity(merges.count).

Measurements: AutoTokenizer.from(modelFolder:) on an Apple Silicon Mac, median of 5 timed runs after one warm-up, base = 1.3.5, interleaved runs:

tokenizer	build	1.3.5	with this PR
Gemma 4 (262k vocab, 515k merges)	Release	1.34 - 1.44 s	0.59 - 0.61 s
Gemma 4	Debug	3.1 - 4.2 s	1.5 s
Qwen3.5 (248k vocab, 248k merges)	Release	0.62 - 0.64 s	0.33 - 0.34 s
Qwen3.5	Debug	1.7 - 1.9 s	1.0 s

Some Debug baseline runs were noisy because other jobs were running on the machine; the PR numbers were stable.

Behaviour: I encoded a multilingual probe (accents, composed and decomposed forms, ZWJ emoji, Hangul, CJK, Arabic, a ligature) with both builds for both tokenizers and the ids are identical (same FNV digest of the id sequence). There is a new test, BinaryDistinctStringTests, in TokenizersTests. It keeps the old implementations as a reference and checks that the UTF-16 values, string view and nsString view match for well-formed strings (composed and decomposed forms stay distinct) and for lone surrogates. It also builds a BPETokenizer from JSON and from a Config containing a lone-surrogate vocabulary key, and checks the vocabulary and merge ranks are unchanged. The test passes on both the old and the new code. The package tests pass locally (I skipped the Benchmarks target; one test was already skipped).

Something I did not do: reading model.vocab and model.merges straight from the yyjson document into native dictionaries, without going through Config, would likely save more (the yyjson-to-Config conversion and mergesFromConfig were about 40% of the profile before this change). That is a bigger change, so I kept this PR to the part that doesn't touch the parsing structure. If that direction interests you I can try it separately.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	⚠️ Failed 2026-10-28T07:56:39.504638Z We couldn’t complete this review. Comment @codex review to try again.	3a0299b	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[pcuenca]

Thanks, this looks good. However, the lone surrogate behaviour differs on Linux. I verified locally using this test, which for some reason I can't submit to your PR.

Could you please accept that commit into your branch so we can run CI here to see how the lone surrogate tests fail? Then we can maybe just run them on Darwin, or fix them so they pass on Linux too.

[jeethu]

Thanks! I've merged your commit. The failures are in the test's expectations, and not a behaviour change.

On Linux, swift-corelibs-foundation stores an NSString as a Swift String, and String(utf16CodeUnits:count:) decodes with repair, so a lone surrogate always became U+FFFD there, before this PR too. The pre-PR code fails the same 7 assertions on Linux. I've made those two assertions platform-specific (Darwin keeps the lone surrogate, Linux gets U+FFFD), so the test now passes on Linux against both the old and the new code.

PTAL.