medspellchecker

Fast and effective spellchecker for Russian medical texts


Keywords
spellchecker, nlp, medical, text, correction, python, russian, spelling-correction
License
Apache-2.0
Install
pip install medspellchecker==0.0.1

Documentation

Project Status: Active – The project has reached a stable, usable state and is being actively developed. Test Python 3.9+ License

MedSpellChecker

Fast and effective tool for correcting spelling errors in Russian medical texts. The tool takes the raw medical text and returns the corrected text in lemmatized form.

This project is in its final stages. If you have found a bug or would like to improve something, please create an issue or pull request.

How to install

The project is written in Python 3.9 and works on Python version 3.9 or higher.

  • From the pip repository

    pip install medspellchecker
  • Or from the sources

    • Install requirements
      pip install -r requirements.txt
    • Install the package
      pip install .

Note! MedSpellChecker has the editdistpy package as a dependency. To install it you will need Microsoft Visual C++ 14 (for Windows) or build-essential package (for Linux).

How to use

An example of using the `medspellchecker' is shown below.

from medspellchecker.tool.medspellchecker import MedSpellchecker
from medspellchecker.tool.distilbert_candidate_ranker import RuDistilBertCandidateRanker

candidate_ranker = RuDistilBertCandidateRanker()
spellchecker = MedSpellchecker(candidate_ranker)
fixed_text = spellchecker.fix_text(
    "Π£ Π±ΠΎΠ»ΡŒΠ½ΠΎΠ³ΠΎΠ΄ΠΈΠ°Π³Π½ΠΎΡΡ‚ΠΈΡ€ΠΎΠ²Π°Π½ ΠΈΠ½Ρ„Ρ€ΠΊΡ‚ ΠΈ Ρ‚ΡƒΠ±Ρ€ΠΊΡƒΠ»Π΅Π·"
)

print(fixed_text)
# -> "Π£ больной Π΄ΠΈΠ°Π³Π½ΠΎΡΡ‚ΠΈΡ€ΠΎΠ²Π°Ρ‚ΡŒ ΠΈΠ½Ρ„Π°Ρ€ΠΊΡ‚ ΠΈ Ρ‚ΡƒΠ±Ρ€ΠΊΡƒΠ»Π΅Π·"
  1. The first two lines import the main class of the package.
  2. Lines 3 and 4 then import one of the three available classes to rank the edit candidates.
  3. Line 6 creates an instance of the candidate ranking class based on the fine-tuned DistilBert model.
  4. Line 7 creates an instance of the class to correct spelling errors.
  5. On line 8, the fix_text method is called, which takes the raw text and returns the corrected text.
  6. Finally line 10 prints the corrected result, which looks like "Π£ больной Π΄ΠΈΠ°Π³Π½ΠΎΡΡ‚ΠΈΡ€ΠΎΠ²Π°Ρ‚ΡŒ ΠΈΠ½Ρ„Π°Ρ€ΠΊΡ‚ ΠΈ Ρ‚ΡƒΠ±Ρ€ΠΊΡƒΠ»Π΅Π·".
  7. In this way, the package can be used to correct spelling errors in Russian medical texts in a few lines.

Tips for using the tool

  1. Initiate and create MedSpellchecker class once. And then use it many times to correct Russian medical texts.
  2. The tool must be used before all the preprocessing steps. This way the various preprocessing steps cannot affect the result of the tool correction.
  3. Nevertheless, the input raw text must be cleared of all sorts of tags, technical symbols and other trash. The input text should be plain text without any structure.
  4. Before using the tool on the whole dataset, test how it works on a small chunk to choose the best available candidate ranker.
  5. If the available candidate rankers do not perform well enough, consider fine-tuning the BERT-based model on your dataset and using the candidate ranker on the fine-tuned model. The quality of the corrections should improve.

Demo

Here is an example of how to correct a spelling mistake with MedSpellChecker.

Demo

Steps for reproducing the demo:

  1. Clone the project
  2. Install all requirements
  3. Go to demo folder
  4. Run demo Flask server
  5. Open demo website and enjoy!

Supported errors

MedSpellChecker supports fixing the following types of errors.

Supported errors

Internals

MedSpellChecker uses the SymDel algorithm to speed up the generation of correction candidates, and a fine-tuned BERT-based machine learning model to rank candidates and select the best fit.

The architecture of the MedSpellChecker tool is shown below.

Arch

This architecture allows each component to be developed almost independently and the correction process to be implemented flexibly.

  • Spellchecker Manager - responsible for coordinating other components and implementing high-level logic.
  • Preprocessor and PostProcessor - responsible for splitting the incoming text and assembling the result.
  • Dictionary - contains a dictionary of correct words, which allows to check the correct word or not.
  • Edit Distance Index - allows to optimize and speed up the calculation of the editing distance required to generate candicates for fixing an incorrect word.
  • Error Model - responsible for generating candidates for fixing incorrect words.
  • Language Model - based on the fine-tuned BERT-based model, ranks candidates for fixing and selects the most suitable word for correction.

Supported Candidate Rankers

Candidate rankers rank the candidates to replace the incorrect word and choose the most appropriate one.

The tool contains several rankers based on different BERT-based models. However, the list of supported rankers can easily be extended with rankers based on any other approach and models. It is sufficient to implement the AbstractCandidateRanker interface.

Available rankers out of the box:

More information

This project is part of master's thesis. The current state is the result of the first year of work. More details about MedSpellCHecker you can find in the text of the term report .