[shubhransh-gupta]

Fixes #383.

Summary

In PreTrainedTokenizer, addedTokensRegex was constructed by wrapping every added token in a capturing group (\(token)) even when neither lstrip nor rstrip was requested.

For tokenizers with large added token vocabularies (e.g. Google Gemma-3 with 6,415 added tokens), this caused an $O(N^2)$ quadratic slowdown in Foundation's ICU regular expression engine due to capturing group table allocations, slowing down single newline \n encoding from ~0.02 ms to ~29 ms (~1.1s for a 40-line prompt on device).

Changes

1. Non-capturing groups by default: When $0.prefix and $0.suffix are both false, compile the token using a non-capturing group (?:\(token)) instead of (\(token)).
2. Preserve whitespace stripping: Added tokens with lstrip or rstrip still retain their capturing group so split(by: NSRegularExpression) continues to isolate and strip surrounding whitespace as expected.
3. Empty check guard: Avoid compiling an empty regex pattern when unwrappedAddedTokens is empty.
4. Regression test: Added addedTokensRegexPerformanceAndCorrectness in Tests/TokenizersTests/TokenizerTests.swift.

Verification

• Executed swift test across all targets (50/50 tests passing).
• Validated code style with swift-format lint --strict --recursive Sources Tests (clean, 0 warnings).

[pcuenca]

Hi @shubhransh-gupta, thanks a lot for your contribution, and sorry for the delay 🙏

Your optimization seems plausible, but the tests you propose are not checking for correctness or performance. I'm happy to merge without them and rely on the existing test corpus, as long as you can provide some indication about performance improvements you measured. Would you happen to have something handy?

[pcuenca]

This doesn't really verify the quadratic explosion claim.

[pcuenca]

Actually, #383 provides sufficient evidence, I'll add @xocialize as a co-author to this PR.