A Python package (using a Docker image under the hood) to lemmatize German texts.
- IWNLP uses the crowd-generated token tables on de.wikitionary.
- GermaLemma: Looks up lemmas in the TIGER Corpus and uses Pattern as a fallback for some rule-based lemmatizations.
It works as follows. First spaCy tags the token with POS. Then
German Lemmatizer looks up lemmas on IWNLP and GermanLemma. If they disagree, choose the one from IWNLP. If they agree or only one tool finds it, take it. Try to preserve the casing of the original token.
You may want to use underlying Docker image: german-lemmatizer-docker
- Install Docker.
pip install german-lemmatizer
- Read and accept the license terms of the TIGER Corpus (free to use for non-commercial purposes).
- Make sure the Docker daemons runs.
- Write some Python code
from german_lemmatizer import lemmatize lemmatize( ['Johannes war ein guter Schüler', 'Sabiene sang zahlreiche Lieder'], working_dir='*', chunk_size=10000, n_jobs=1, escape=False, remove_stop=False)
The list of texts is split into chunks (
chunk_size) and processed in parallel (
escape parameter if your text contains newslines.
remove_stop removes stopwords as defined by spaCy.