Tixent: A Text Splitting Tool
PyPI
CI/CD
Build System
Code
Docstrings
Installation
pip install tixentExample
Suppose we have a function template that generates a string from a list of texts. Additionally, suppose we have a large list of texts. When you apply that list of texts to the function, it generates a long string.
Tixent can split the string generated by the template function so that the return value of counter for each element is less than a certain number.
Here, counter is a function that maps a string to an integer.
Examples of such functions are len, which measures the length of a string, or tiktoken_counter("text-davinci-003"), which measures the number of tokens in a string
from typing import List
from tixent import split, tiktoken_counter
def summarization_template(texts: List[str]) -> str:
text = " ".join(texts)
t = "Summarize the following text.\n"
t += f'Text: """{text}"""'
return t
texts = [
"Lorem ipsum dolor sit amet",
"consectetur adipiscing elit",
"sed do eiusmod tempor incididunt ut labore et dolore magna aliqua",
"Ut enim ad minim veniam",
"quis nostrud exercitation ullamco laboris nisi",
"ut aliquip ex ea commodo consequat",
"Duis aute irure dolor in reprehenderit in voluptate velit",
"esse cillum dolore eu fugiat nulla pariatur",
"Excepteur sint occaecat cupidatat non proident",
"sunt in culpa qui officia deserunt mollit anim id est laborum",
]
counter = tiktoken_counter("text-davinci-003")
max_count = 60
split_texts = split(texts, summarization_template, counter, max_count)
for text in split_texts:
count = counter(text)
assert count <= max_count
print(f"count: {count}")
print(text)
print()count: 60
Summarize the following text.
Text: """Lorem ipsum dolor sit amet consectetur adipiscing elit sed do eiusmod tempor incididunt ut labore et dolore magna aliqua Ut enim ad minim veniam"""
count: 58
Summarize the following text.
Text: """quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat Duis aute irure dolor in reprehenderit in voluptate velit"""
count: 43
Summarize the following text.
Text: """esse cillum dolore eu fugiat nulla pariatur Excepteur sint occaecat cupidatat non proident"""
count: 31
Summarize the following text.
Text: """sunt in culpa qui officia deserunt mollit anim id est laborum"""
