wordpieces
This crate provides a subword tokenizer. A subword tokenizer splits a token into several pieces, so-called word pieces. Word pieces were popularized by and used in the BERT natural language encoder.
Split tokens into word pieces
Homepage Cargo Rust Documentation Download
This crate provides a subword tokenizer. A subword tokenizer splits a token into several pieces, so-called word pieces. Word pieces were popularized by and used in the BERT natural language encoder.
The Tidelift Subscription provides access to a continuously curated stream of human-researched and maintainer-verified data on open source packages and their licenses, releases, vulnerabilities, and development practices.
Learn more →Something wrong with this page? Make a suggestion
Export .ABOUT file for this package
Last synced: 2022-10-10 15:00:17 UTC
Login to resync this project