English | 中文

Nexaloid is a Chinese tokenizer runtime. The core is implemented in Zig, exposes a stable C ABI, and ships bindings for Python, Node.js, C++, Zig, Go, and Rust.

It is designed for workloads that need correctness, throughput, and cross-language consistency, such as search engines, RAG pipelines, e-commerce catalogs, and text analytics.

• A dictionary-based Chinese segmentation engine using a double-array trie and Viterbi decoding
• A conservative Search mode: it expands only the Accurate/Viterbi best path; RecallSearch is available when you need more aggressive candidate recall over all lattice edges
• An explainable tokenizer: every token carries source, offsets, and score
• An optional CandidateProvider plugin ABI; the core runs independently without plugins
• C ABI and language bindings preserve both UTF-8 byte offsets and Unicode character offsets

from nexaloid import Tokenizer

tokenizer = Tokenizer()
for token in tokenizer.tokenize("武汉市长江大桥"):
    print(token.text, token.start_byte, token.end_byte, token.source, token.score)

# jieba-compatible API
import nexaloid.compat_jieba as jieba
print(jieba.lcut("研究生命起源"))
# ['研究', '生命', '起源']

$env:PYTHONPATH = "$PWD\bindings\python\src"
python -c "import nexaloid.compat_jieba as jieba; print(jieba.lcut('武汉市长江大桥'))"

Projects that already use jieba can keep common function names by changing the import:

import nexaloid.compat_jieba as jieba

The currently compatible module-level APIs include cut, lcut, cut_for_search, lcut_for_search, load_userdict, add_word, del_word, and suggest_freq.

cut returns an iterator of strings, and lcut returns a list. cut_for_search uses the conservative Search mode, which avoids cross-boundary candidates such as 市长 in 武汉市长江大桥; use the native Mode.RECALL_SEARCH when you need more aggressive recall. load_userdict accepts jieba-style word freq tag text dictionaries. Passing tag to add_word emits a warning because POS tagging is unavailable, while use_paddle=True raises NotImplementedError; HMM=True loads the bundled BMES HMM plugin to recover unknown words. Use nexaloid.Tokenizer directly when you need offsets, source, score, or batch tokenization.

const { Tokenizer } = require("@nexaloid/nexaloid");

const tokenizer = new Tokenizer();
console.log(tokenizer.lcut("武汉市长江大桥"));
// ['武汉市', '长江大桥']
tokenizer.close();

cd bindings/node
npm run build
npm run test:binding

#include <nexaloid.hpp>

NxConfig cfg{};
cfg.dict_path = "data/dict/nexaloid.tsv";   // .nxdict is also supported
nexaloid::Tokenizer tokenizer(cfg);
for (const auto& word : tokenizer.cut("武汉市长江大桥"))
    std::cout << word << "\n";

tokenizer, _ := nexaloid.New("data/dict/nexaloid.tsv")
defer tokenizer.Close()
tokens, _ := tokenizer.Tokenize("武汉市长江大桥", nexaloid.Accurate)

use nexaloid::{runtime_version, Mode, Tokenizer, VERSION};

assert_eq!(runtime_version()?, VERSION);
let tokenizer = Tokenizer::new_default()?;
let tokens = tokenizer.tokenize("武汉市长江大桥", Mode::Accurate)?;

Every Rust engine initialization compares the nexaloid crate, nexaloid-sys, and native runtime versions. Missing or mismatched native versions fail before an engine is created; plugin ABI and model-format checks run when a plugin is loaded.

Raw Search / RecallSearch token output is positional and is not deduplicated by text. Convenience cut_for_search APIs keep their search-term behavior by filtering one-character terms and deduplicating text.

Nexaloid filters pure whitespace tokens by default, which is usually better for search and RAG pipelines. Enable whitespace preservation when you need to reconstruct the input shape; spaces, tabs, newlines, full-width spaces, and similar spans are emitted as standalone tokens:

tokenizer = Tokenizer(preserve_whitespace=True)
print(tokenizer.lcut("中文 English\t混排\n第二行"))
# ['中文', ' ', 'English', '\t', '混排', '\n', '第二行']

The same switch is exposed through NxConfig.preserve_whitespace in the C ABI and through constructor options in each language binding.

The core engine supports both the text format (.tsv) and the binary packed-trie format (.nxdict). .nxdict is compact, about 38 MiB, loads quickly, and can be memory-mapped on Windows. .tsv must be parsed and inserted line by line, so startup is slower; it is mainly useful for development and debugging. Use .nxdict in production.

Default constructor behavior by binding:

The rule matcher adds candidates for structured tokens such as URLs, email addresses, ISO timestamps, Windows paths, IPv6 addresses, number-unit spans, market-day expressions such as T+3日, and mixed ASCII terms such as GPT-5.5. Built-in rules are enabled by default.

Scores are internal decoder weights, not strict probabilities. During Viterbi decoding, each candidate edge's score is added directly to the path base score. A high score such as 300.0 strongly protects structured tokens from being split, while a low score such as 3.0 for ascii_term can still lose to higher-scoring dictionary entries.

from nexaloid import Tokenizer

tokenizer = Tokenizer(rule_config={
    "ascii_term": False,        # Disable the ascii_term rule
    "scores": {
        "url": 120.0,
        "email": 120.0,
    },
})

Rule configuration only controls candidates generated by rules. Entries from the base or user dictionary can still cover the same span and keep their own source. For example, if GPT-5.5 already exists in the dictionary, the final token may have source=base_dict even though the ascii_term rule can also propose the same span.

Custom rules are defined as JSON, parsed by the Zig core, and behave consistently across all language bindings.

Common fields are name, kind, score, enabled, and boundary. boundary accepts none, ascii, or ascii_or_han.

{
  "version": 1,
  "rules": [
    {
      "name": "stock",
      "kind": "prefixed_number",
      "prefixes": ["SH", "SZ", "HK"],
      "digits": {"min": 4, "max": 6},
      "score": 80
    },
    {
      "name": "sku",
      "kind": "charset_span",
      "charset": "A-Z0-9-_",
      "min_len": 4,
      "max_len": 32,
      "score": 60
    },
    {
      "name": "model_name",
      "kind": "ascii_chain",
      "charset": "A-Za-z0-9.-",
      "must_contain": ["-", "."],
      "min_len": 4,
      "max_len": 32,
      "score": 80
    },
    {
      "name": "dose",
      "kind": "number_unit",
      "units": ["mg", "%", "mmol/L"],
      "score": 90
    }
  ]
}

All bindings (C, Node.js, C++, Zig, Go, Rust) expose equivalent functions and pass JSON through to the core unchanged, keeping matching behavior consistent.

For tokens produced by custom rules, the flags field stores the 1-based index in the JSON rules array. The first rule has index 1. This index applies only to custom rules and does not include built-in rules. Disabled rules still occupy their array position, so indexes remain stable for auditing and debugging.

python tools/rule_audit.py data/rules/v4_sample_rules.json data/badcases/rules_v4.json

UTF-8 text
  -> Scanner (codepoints + byte/char offsets + class)
  -> Dictionary Matcher (double-array trie traversal + overlay trie)
  -> Rule Matcher (built-in/custom rules)
  -> CandidateProvider Plugins (optional HMM/NER/domain candidates)
  -> Lattice (all candidate edges from dict/rule/plugin/unknown sources)
  -> Viterbi Decoder (globally best path)
  -> Token Stream (filtering, offset preservation)

The core is a single Zig library, libnexaloid. All language bindings call the C ABI and do not reimplement tokenization logic. Runtime plugins currently support only CandidateProvider; the other hook kinds are reserved ABI surface.

All bindings call the same core engine. With the same input and dictionary, token text and offsets should match exactly across languages.

core/include/nexaloid_plugin.h defines eight hook interfaces. Only CandidateProvider is currently available; the others are reserved.

Plugins are loaded through nx_load_plugin. ABI v2 gives CandidateProviders the scanner's codepoints, offsets, character classes, and flags, avoiding a second UTF-8 scan; the core loader remains compatible with ABI v1 providers. Plugins write candidates into the lattice using character offsets, and the core maps those offsets back to byte offsets before Viterbi decoding. A plugin edge that overlaps a loaded user-dictionary span is rejected unless both edges have the same span, so plugins cannot remove an explicit user word from the main path.

The optional BMES HMM lattice artifact is located at:

data/hmm/bmes_hmm_wordhub_lattice.nxhmm
data/hmm/bmes_hmm_wordhub_lattice.manifest.json

This artifact is produced by the NexaloidHMM project and consumed by the HMM CandidateProvider plugin. It is packaged with Python, Node.js, Rust, and native SDK releases, and remains opt-in at the binding layer.

Bindings expose the bundled artifact path:

from nexaloid import hmm_artifact_path, hmm_manifest
print(hmm_artifact_path())
print(hmm_manifest()["quality"]["lattice_heldout"]["token_f1"])

const { hmmArtifactPath, hmmManifest } = require("@nexaloid/nexaloid");
console.log(hmmArtifactPath);
console.log(hmmManifest().quality.lattice_heldout.token_f1);

println!("{}", nexaloid::bundled_hmm_artifact_path().display());

HMM plugin configuration example, accepting either an artifact path or JSON:

{"artifact":"data/hmm/bmes_hmm_wordhub_lattice.nxhmm","hmm_score":-14.0}

hmm_score is an empirical weight used directly as the candidate edge score during Viterbi decoding. It is added with rule scores and dictionary log-probabilities, and controls the merge strength for unknown words. The default value, -14.0, is audited with manually selected risk cases, WordHub cases, and structured-token probes (tools/hmm_score_audit.py) to balance recall and segmentation precision.

tools/entity_bmes_plugin.zig is a model-backed CandidateProvider for entity nouns. It memory-maps a .nxbmes artifact produced by the separate NexaloidBMES project and runs an O/B/M/E/S averaged-perceptron decoder with hashed character, character-class, and gazetteer features. It remains opt-in and can propose entities that are absent from the base dictionary.

Source checkouts can build and stage the native plugin after building the core library:

python tools/stage_assets.py

Load the plugin with either an artifact path or JSON configuration:

{"artifact":"data/entity/entity_bmes_perceptron.nxbmes","score_per_char":3.0,"edge_penalty":10.0,"min_margin":35.0,"min_chars":2,"max_chars":64,"flags":4}

Python packages expose both bundled paths:

import json
from nexaloid import Tokenizer, entity_artifact_path, entity_plugin_path

tokenizer = Tokenizer()
tokenizer.load_plugin(
    entity_plugin_path(),
    json.dumps({"artifact": str(entity_artifact_path())}),
)

For each decoded entity, min_margin filters the average per-character emission margin. Terms already present in the model's general lexicon are not emitted, keeping this CandidateProvider focused on unknown entities. Accepted scores are min(400, score_per_char * margin - edge_penalty), so length alone cannot dominate dictionary paths. Model features are indexed at load time, and a high-precision gate skips full Viterbi inference on text without a strong entity signal. Unicode whitespace and punctuation are hard boundaries; ·, -, ‐, ‑, &, and / remain internal only when both neighbors are letters or digits. ASCII entities still require ASCII boundaries. Emitted tokens use source=plugin, and flags defaults to 4 so it does not overlap the HMM plugin's 1/2 values. The bundled release-safe model is trained from THUOCL (MIT), JD comments (Apache-2.0), and deterministic synthetic examples; its manifest reports dev F1 0.793487 and test F1 0.864987. As with every loaded plugin, batch tokenization is currently serialized.

Measure plugin overhead on a gzipped news JSONL corpus with interleaved rounds and a median threshold:

python tools/entity_plugin_benchmark.py news.jsonl.gz --dict finance.tsv --rounds 7 --max-overhead 5

All exported functions use the nx_ prefix. The main header is core/include/nexaloid.h.

Detailed declarations:

NxStatus nx_engine_new(const NxConfig *config, NxEngine **out_engine);
void      nx_engine_free(NxEngine *engine);
NxStatus  nx_tokenize(NxEngine *engine, const char *text, size_t text_len,
                      NxMode mode, NxTokenCallback callback, void *user_data);
NxStatus  nx_tokenize_batch(NxEngine *engine, const char *const *texts,
                            const size_t *text_lens, size_t text_count,
                            NxMode mode, uint32_t thread_count,
                            NxBatchTokenCallback callback, void *user_data);
NxStatus  nx_add_word(NxEngine *engine, const char *word, size_t word_len,
                      uint32_t word_id, float score, uint16_t pos_id);
NxStatus  nx_reload_user_dict(NxEngine *engine, const char *user_dict_path);
NxStatus  nx_set_rule_config(NxEngine *engine, uint32_t enabled_mask,
                             const float *scores, size_t score_count);
NxStatus  nx_load_rules_json(NxEngine *engine, const char *json,
                             size_t json_len);
NxStatus  nx_clear_rules(NxEngine *engine);
NxStatus  nx_load_plugin(NxEngine *engine, const char *plugin_path,
                         const char *config_json);

enabled_mask uses 1u << NxRuleId; pass NX_RULE_ALL_MASK for defaults. scores may be NULL, or an array ordered by NxRuleId.

Headers: core/include/nexaloid.h, core/include/nexaloid_plugin.h

Every token contains these fields:

python tools/dict_builder.py --out data/dict/nexaloid.tsv

The command auto-detects an installed jieba package; otherwise pass --jieba-dict with a dict.txt path. data/dict/demote.tsv lowers noisy base-word scores before overlay.tsv applies manual weighting.

python tools/nxdict_builder.py data/dict/nexaloid.tsv data/dict/nexaloid.nxdict

python tools/validate_overlay.py data/dict/overlay.tsv

Normal users do not need this step. It is used to import reviewed candidate entries from the external NexaloidData repository.

python tools/import_nexaloid_data.py --data-root F:\Code\03_OpenCode\NexaloidData
python tools/validate_overlay.py data/dict/generated/overlay.generated.tsv

cd core
zig build          # Build the shared library
zig build test     # Run core tests
cd ..

# Python regression tests
$env:PYTHONPATH = "$PWD\bindings\python\src"
python tools\badcase_runner.py
python tools\validate_overlay.py
python tools\benchmark.py

# Windows: Go tests need the native DLL on PATH
$env:PATH = "$PWD\core\zig-out\bin;$env:PATH"
cd bindings\go
go test ./nexaloid

$env:PYTHONPATH = "$PWD\bindings\python\src"
python tools\benchmark.py -n 1000

1. Check out the matching release branch: release/c, release/cpp, or release/zig
2. Download the zip for the same version and platform from GitHub Releases
3. Extract it, then copy the zip's lib/ directory into the root of your cloned checkout
4. Build and run the examples

Each SDK contains headers, the platform native library, examples, and data/dict/nexaloid.nxdict.

Python and npm packages cover only the platforms that their pipelines actually build and test. Rust uses more granular target-specific native crates, such as nexaloid-sys-linux-x64, to stay within crates.io package size limits.