A high-performance, Rust-native natural language processing toolkit: tokenization, stemming, phonetic matching, string distance, n-grams, normalization, inflection, a trie, transliteration, WordNet, TF-IDF, sentiment analysis, and Bayes/logistic-regression/MaxEnt classifiers — each designed for Rust from the start.

Status: pre-1.0; every production crate is implemented, documented, and test-pinned.

Read the documentation →

The site is not a summary of the API — it is where the library explains when to use each of its variants and what each one costs. Its centrepiece is Choosing the Right API: whenever more than one function solves the same conceptual problem, the docs must say why each exists, what it allocates, whether it is lazy, whether it reuses memory, and which one you should call.

The complete documentation layout — public site, Rust API contracts and internal engineering records — is defined in DOCUMENTATION.md. Contribution expectations are in CONTRIBUTING.md.

Source in site/, built with VitePress and published from main. Building it locally:

cd site
npm install
npm run dev                            # http://localhost:5173

python3 check-snippets.py              # every published example compiles and passes
npm run build && python3 check-links.py  # no dangling internal link or anchor

Documentation is treated as part of the code: a change to public behaviour that does not update the site is an incomplete change. The rules are in AGENTS.md.

Every public behaviour is written down before it is implemented, and every documented behaviour is pinned by a test that fails when it changes. Nothing is left to "obvious from the code": defaults, error cases, Unicode handling, ordering and floating-point behaviour are all specified explicitly.

Where a subsystem's output is large or structured — tagger output, WordNet lookups, classifier scores — it is pinned with committed golden files reviewed by hand once and diffed on every run thereafter. A diff in a golden file is a behaviour change and has to be justified.

crates/
  verbora-core/            traits, Token, stop-word state, string semantics
  verbora-tokenizers/      25 tokenizers, the Tokenize trait
  verbora-distance/        Levenshtein, Damerau, Jaro–Winkler, Dice, Hamming
  verbora-phonetics/       SoundEx, Metaphone, Double Metaphone, D-M Soundex
  verbora-ngrams/          n-gram windows, frequency stats, Chinese n-grams
  verbora-normalizers/     diacritics, contractions, Japanese width/kana
  verbora-inflectors/      pluralise/singularise, ordinals (en/fr/ja)
  verbora-trie/            prefix tree
  verbora-transliterators/ Japanese kana → romaji
  verbora-wordnet/         lexical database, synsets, relation traversal
  verbora-tfidf/           term interning, incremental idf cache
  verbora-sentiment/       fourteen lexicons across ten languages
  verbora-classifiers/     Bayes, logistic regression, MaxEnt + GIS
  verbora-analyzers/       sentence analysis
  verbora-spellcheck/      Norvig-style correction
  verbora-stemmers/        Porter × 13, Lancaster, Japanese, Indonesian
  verbora-tagger/          Brill POS tagger, trainer, tester
  verbora-language/        script and language detection, phonetic strategy
  verbora-util/            stop words, abbreviations, graph utilities
  verbora-examples/        the code the documentation site publishes (dev-only)
site/                 the documentation site (VitePress) — see "Documentation" above
docs/                 internal engineering/research archive, not user docs — see docs/README.md
tools/
  bench-data/           shared benchmark inputs
benches/data/         inputs shared across benchmarks

cargo test                             # unit + integration + doctests
cargo bench -p verbora-distance        # benchmarks

Correctness before speed, and speed proven by measurement. The priority order is correctness → specified behaviour → performance-aware architecture → memory → API quality → maintainability → hot-path tuning. No optimisation lands without the test suite re-run, and none is claimed without a benchmark. A measured regression is documented in the benchmarks page precisely because "it's Rust, so it's fast" would have shipped it.

Several API levels, sharing one primitive. A high-level, ergonomic API; a lazy iterator; a low-level one that writes into caller-supplied buffers so hot loops can amortise allocation; and, for thirteen crates where a real benchmark justified it, an opt-in parallel batch API — each built on top of the one before it, never a second implementation:

tokenizer.tokens(text)                // lazy iterator — the primitive
tokenizer.tokenize(text)              // owned, ergonomic — collects the iterator
tokenizer.tokenize_into(text, &mut v) // reuses the caller's buffer
tokenizer.tokenize_borrowed(text)     // &str slices, zero copies

Fast paths that are exact, not approximate. String distances are defined over UTF-16 code units, and the difference is observable — LevenshteinDistance("a😀b", "ab") is 2 under that definition and 1 under a char-based one. verbora-distance runs ASCII operands over &[u8] and promotes only genuinely non-ASCII input to Vec<u16>. For ASCII, one byte is one code unit, so the fast path is not a shortcut.

Explicit, not flattering. DiceCoefficient("", "") returns NaN; HammingDistance returns -1 for length mismatch; search offsets can be negative. Each of these is a deliberate, documented choice with a test behind it, not an accident smoothed over in prose.

verbora-distance across its own API levels and input sizes — full table and method in the benchmarks page.

Thirteen crates additionally ship an optional parallel Cargo feature for batch workloads where a real crossover benchmark justified it — see Parallelism for which crates, and the measured numbers.

Verbora is measured against the wider Rust ecosystem: 290 real, version-pinned benchmarks across 13 modules — strsim, rapidfuzz, tantivy, rust-stemmers, rphonetic, symspell, harper-core, smartcore and more — plus a language-detection accuracy report. Every loss is published alongside every win, with its investigated cause. See Competitive benchmarks for the full results and docs/PERFORMANCE_GAPS.md for every gap this audit found.

crates/verbora-distance is the worked reference for every step: specify the behaviour in rustdoc first, implement it, pin it with tests, benchmark it, and publish the documentation page in the same change. The rules are in AGENTS.md.

MIT. Linguistic data assets (WordNet, sentiment lexicons) carry their original licences and attribution — see each feature's own documentation page before redistributing.