fix(Tokenizers): avoid quadratic regex matching for added tokens without whitespace padding - #386

fix(Tokenizers): avoid quadratic regex matching for added tokens without whitespace padding#386

pcuenca merged 4 commits into

huggingface:mainhuggingface/swift-transformers:mainfrom

shubhransh-gupta:fix/added-tokens-regex-perfshubhransh-gupta/swift-transformers:fix/added-tokens-regex-perfCopy head branch name to clipboard