Duplicate Text Finder
duptextfinder is a python library to detect duplicated zones in text. Primarily meant to detect
copy/paste across medical documents. Should be faster than python's built-in
difflib algorithm and more robust to whitespace, newlines and other irrelevant
characters.
Installation
duptextfinder can be installed through pip:
pip install duptextfinder
Usage
from pathlib import Path
from duptextfinder import CharFingerprintBuilder, DuplicateFinder
# load some text files
texts = [p.read_text() for p in Path("some/dir").glob("*.txt")]
# init fingerprint and duplicate finder
fingerprintBuilder = CharFingerprintBuilder(fingerprintLength=15)
duplicateFinder = DuplicateFinder(fingerprintBuilder, minDuplicateLength=15)
# call findDuplicates() on each file
for i, text in enumerate(texts):
id = f"D{i}"
duplicates = duplicateFinder.findDuplicates(id, text)
for duplicate in duplicates:
print(
f"sourceDoc={duplicate.sourceDocId}, "
f"sourceStart={duplicate.sourceSpan.start}, "
f"sourceEnd={duplicate.sourceSpan.end}, "
f"targetStart={duplicate.targetSpan.start}, "
f"targetEnd={duplicate.targetSpan.end}"
)
duplicated_text = text[duplicate.targetSpan.start : duplicate.targetSpan.end]
print(duplicated_text)WordFingerprintBuilder can be used instead of CharFingerprintBuilder. For
more details, refer to the docstrings of DuplicateFinder,
CharFingerprintBuilder and WordFingerprintBuilder.
How to run tests
- Install package in editable mode with test and extra dependencies by running
pip install -e ".[tests, ncls, intervaltree]"in the repo directory - Launch
pytest tests/
About ncls and intervaltree
This tool can be used without any additional dependencies, but performance can be improved when using interval trees. To benefit from this you well need to install either the ncls package or the intervaltree package.
References
- Evaluating the Impact of Text Duplications on a Corpus of More than 600,000 Clinical Narratives in a French Hospital. https://www.hal.inserm.fr/hal-02265124/
