stokenizer

Multilingual Partial Syllable Tokenization - A rule-based tokenization method designed to align with linguistic nuances while minimizing False Positive errors.


License
MIT
Install
pip install stokenizer==0.1.0