BPETokenizer: build vocab and merge strings natively to speed up load - #416

BPETokenizer: build vocab and merge strings natively to speed up load#416

pcuenca merged 4 commits into

huggingface:mainhuggingface/swift-transformers:mainfrom

numen-tech:bpe-init-utf16numen-tech/swift-transformers:bpe-init-utf16Copy head branch name to clipboard