UnicodeTokenizer
UnicodeTokenizer: tokenize all Unicode text
切词规则 Tokenize Rules
- break line
- Punctuation
- UnicodeScripts
- Split(" ?[^(\\s|[.,!?…。，、।۔،])]+"
- break word
use
pip install UnicodeTokenizer
from UnicodeTokenizer import UnicodeTokenizer
tokenizer=UnicodeTokenizer()
line = """
首先8.88设置 st。art_new_word=True 和 output=[açaí]，output 就是最终� no such name"
的输出คุณจะจัดพิธีแต่งงานเมื่อไรคะ탑승 수속해야pneumonoultramicroscopicsilicovolcanoconiosis"
하는데 카운터가 어디에 있어요ꆃꎭꆈꌠꊨꏦꏲꅉꆅꉚꅉꋍꂷꂶꌠلأحياء تمارين تتطلب من [MASK] [PAD] [CLS][SEP]
est 𗴂𗹭𘜶𗴲𗂧, ou "phiow-bjij-lhjij-lhjij", ce que l'on peut traduire par « pays-grand-blanc-élevé » (白高大夏國).
""".strip()
print(tokenizer.tokenize(line))
print(tokenizer.split_lines(line))or
git clone https://github.com/laohur/UnicodeTokenizer
cd UnicodeTokenizer # modify
pip install -e .reference
- PyICU https://gitlab.pyicu.org/main/pyicu
- tokenizers https://github.com/huggingface/tokenizers
- ICU-tokenizer https://github.com/mingruimingrui/ICU-tokenizer/tree/master
