FST Lookup

Implements lookup for Foma finite state transducers.

Supports Python 3.5 and up.

Install

pip install fst-lookup

Usage

Import the library, and load an FST from a file:

Hint: Test this module by downloading the eat FST!

>>> from fst_lookup import FST
>>> fst = FST.from_file('eat.fomabin')

Assumed format of the FSTs

fst_lookup assumes that the lower label corresponds to the surface form, while the upper label corresponds to the lemma, and linguistic tags and features: e.g., your LEXC will look something like this—note what is on each side of the colon (:):

Multichar_Symbols +N +Sg +Pl
Lexicon Root
    cow+N+Sg:cow #;
    cow+N+Pl:cows #;
    goose+N+Sg:goose #;
    goose+N+Pl:geese #;
    sheep+N+Sg:sheep #;
    sheep+N+Pl:sheep #;

If your FST has labels on the opposite sides—e.g., the upper label corresponds to the surface form and the upper label corresponds to the lemma and linguistic tags—then instantiate the FST by providing the labels="invert" keyword argument:

fst = FST.from_file('eat-inverted.fomabin', labels="invert")

Hint: FSTs originating from the HFST suite are often inverted, so try to loading the FST inverted first if .generate() or .analyze() aren't working correctly!

Analyze a word form

To analyze a form (take a word form, and get its linguistic analyzes) call the analyze() function:

def analyze(self, surface_form: str) -> Iterator[Analysis]

This will yield all possible linguistic analyses produced by the FST.

An analysis is a tuple of strings. The strings are either linguistic tags, or the lemma (base form of the word).

FST.analyze() is a generator, so you must call list() to get a list.

>>> list(sorted(fst.analyze('eats')))
[('eat', '+N', '+Mass'),
 ('eat', '+V', '+3P', '+Sg')]

Generate a word form

To generate a form (take a linguistic analysis, and get its concrete word forms), call the generate() function:

def generate(self, analysis: str) -> Iterator[str]

FST.generate() is a Python generator, so you must call list() to get a list.

>>> list(fst.generate('eat+V+Past')))
['ate']

Contributing

If you plan to contribute code, it is recommended you use Poetry. Fork and clone this repository, then install development dependencies by typing:

poetry install

Then, do all your development within a virtual environment, managed by Poetry:

poetry shell

Type-checking

This project uses mypy to check static types. To invoke it on this package, type the following:

mypy -p fst_lookup

Running tests

To run this project's tests, we use py.test:

poetry run pytest

C Extension

Building the C extension is handled in build.py

To disable building the C extension, add the following line to .env:

export FST_LOOKUP_BUILD_EXT=False

(by default, this is True).

To enable debugging flags while working on the C extension, add the following line to .env:

export FST_LOOKUP_DEBUG=TRUE

(by default, this is False).

Fixtures

If you are creating or modifying existing test fixtures (i.e., mostly pre-built FSTs used for testing), you will need the following dependencies:

• GNU make
• Foma

Fixtures are stored in tests/data/. Here, you will use make to compile all pre-built FSTs from source:

make

License

Copyright © 2019–2021 National Research Council Canada.

Licensed under the MIT license.