name: bpetokenizer
description: A Byte Pair Encoding (BPE) tokenizer, which algorithmically follows along the GPT tokenizer(tiktoken), allows you to train your own tokenizer. The tokenizer is capable of handling special tokens and uses a customizable regex pattern for tokenization(includes the gpt4 regex pattern). supports `save` and `load` tokenizers in the `json` and `file` format. The `bpetokenizer` also supports [pretrained](bpetokenizer/pretrained/) tokenizers.
license_spdx: MIT
version: 1.0.0
spec_version: 1.0
home_url: https://github.com/Hk669/bpetokenizer
news_url: https://libraries.io/pypi/bpetokenizer/versions.atom
date: 2024-05-27
about_resource: NONE
libraries_io_url: https://libraries.io/pypi/bpetokenizer?number=1.0.0
vcs_tool: git
vcs_repository: https://github.com/Hk669/bpetokenizer
vcs_branch: main
