Tokenizer POS-tagger and Dependency-parser for Classical Chinese


Keywords
udpipe, mecab, nlp, ancient-chinese, classical-chinese, literary-chinese
License
MIT
Install
pip install udkanbun==3.4.3

Documentation

Current PyPI packages

UD-Kanbun

Tokenizer, POS-Tagger, and Dependency-Parser for Classical Chinese Texts (漢文/文言文), working on Universal Dependencies.

Basic usage

>>> import udkanbun
>>> lzh=udkanbun.load()
>>> s=lzh("不入虎穴不得虎子")
>>> print(s)
# text = 不入虎穴不得虎子
1			ADV	v,副詞,否定,無界	Polarity=Neg	2	advmod	_	Gloss=not|SpaceAfter=No
2			VERB	v,動詞,行為,移動	_	0	root	_	Gloss=enter|SpaceAfter=No
3			NOUN	n,名詞,主体,動物	_	4	nmod	_	Gloss=tiger|SpaceAfter=No
4			NOUN	n,名詞,固定物,地形	Case=Loc	2	obj	_	Gloss=cave|SpaceAfter=No
5			ADV	v,副詞,否定,無界	Polarity=Neg	6	advmod	_	Gloss=not|SpaceAfter=No
6			VERB	v,動詞,行為,得失	_	2	parataxis	_	Gloss=get|SpaceAfter=No
7			NOUN	n,名詞,主体,動物	_	8	nmod	_	Gloss=tiger|SpaceAfter=No
8			NOUN	n,名詞,,関係	_	6	obj	_	Gloss=child|SpaceAfter=No

>>> t=s[1]
>>> print(t.id,t.form,t.lemma,t.upos,t.xpos,t.feats,t.head.id,t.deprel,t.deps,t.misc)
1   ADV v,副詞,否定,無界 Polarity=Neg 2 advmod _ Gloss=not|SpaceAfter=No

>>> print(s.kaeriten())
虎穴虎子>>> print(s.to_tree())
 <════╗   advmod
 ═══╗═╝═╗ root
 <╗ ║   ║ nmod
 ═╝<╝   ║ obj
 <════╗ ║ advmod
 ═══╗═╝<parataxis
 <╗ ║     nmod
 ═╝<obj

>>> f=open("trial.svg","w")
>>> f.write(s.to_svg())
>>> f.close()

trial.svg udkanbun.load() has three options udkanbun.load(MeCab=True,Danku=False). By default, the UD-Kanbun pipeline uses MeCab for tokenizer and POS-tagger, then uses UDPipe for dependency-parser. With the option MeCab=False the pipeline uses UDPipe for all through the processing. With the option Danku=True the pipeline tries to segment sentences automatically.

udkanbun.UDKanbunEntry.to_tree() has an option to_tree(BoxDrawingWidth=2) for old terminals, whose Box Drawing characters are "fullwidth". to_tree(kaeriten=True,Japanese=True) is convenient for Japanese users.

You can simply use udkanbun on the command line:

echo 不入虎穴不得虎子 | udkanbun

Usage via spaCy

If you have already installed spaCy 2.1.0 or later, you can use UD-Kanbun via spaCy Language pipeline.

>>> import udkanbun.spacy
>>> lzh=udkanbun.spacy.load()
>>> d=lzh("不入虎穴不得虎子")
>>> print(type(d))
<class 'spacy.tokens.doc.Doc'>
>>> print(udkanbun.spacy.to_conllu(d))
# text = 不入虎穴不得虎子
1			ADV	v,副詞,否定,無界	_	2	advmod	_	Gloss=not|SpaceAfter=No
2			VERB	v,動詞,行為,移動	_	0	root	_	Gloss=enter|SpaceAfter=No
3			NOUN	n,名詞,主体,動物	_	4	nmod	_	Gloss=tiger|SpaceAfter=No
4			NOUN	n,名詞,固定物,地形	_	2	obj	_	Gloss=cave|SpaceAfter=No
5			ADV	v,副詞,否定,無界	_	6	advmod	_	Gloss=not|SpaceAfter=No
6			VERB	v,動詞,行為,得失	_	2	parataxis	_	Gloss=get|SpaceAfter=No
7			NOUN	n,名詞,主体,動物	_	8	nmod	_	Gloss=tiger|SpaceAfter=No
8			NOUN	n,名詞,,関係	_	6	obj	_	Gloss=child|SpaceAfter=No

>>> t=d[0]
>>> print(t.i+1,t.orth_,t.lemma_,t.pos_,t.tag_,t.head.i+1,t.dep_,t.whitespace_,t.norm_)
1   ADV v,副詞,否定,無界 2 advmod  not

Installation for Linux

Tar-ball is available for Linux, and is installed by default when you use pip:

pip install udkanbun

Installation for Cygwin

Make sure to get gcc-g++ python37-pip python37-devel packages, and then:

pip3.7 install udkanbun

Use python3.7 command in Cygwin instead of python.

Installation for Jupyter Notebook (Google Colaboratory)

!pip install udkanbun

Try notebook for Google Colaboratory.

Author

Koichi Yasuoka (安岡孝一)

References