Graph based Semantic Role Labeller

This is a semantic role laballer based on a graph convolutional network. The goal is to make something reasonably state-of-the-art for the Dutch language.

This is work in progress.

Training data

SoNaR contains a annotated dataset of 500K words. The annotation of the frames is quite basic:

• the semantic annotations were made on top of a syntactic dependency tree
• only (some) verbs are marked
• no linking to FrameNet or any frame identification is made

The original annotations were made on AlpinoXML; the files are converted to conllu using a script from Groningen. Conversion scripts from Alpino and Lassy to the UD format available here.

I am planning to use universal dependencies schema for the syntax, and assume input files are in conllu format. This way the labeller can be used as part of the Newsreader annotation pipeline.

Approach

The training set was made as annotations on top of a syntactic tree. We'll use the same tree, as given by the HEAD and DEPREL fields from the conll files. Words form the nodes, and we add three kinds of edges:

• from dependent to head (weighted by the number of dependents)
• from the head to the dependent
• from the node to itself

At each node, we add a GRU cell. The initial state is made using one-hot encoding of a number of features. The output of the cell is then passed to two classifiers One to indicate if this word is a frame, and one to indicate if the word is the head of an arguments (and which argument).

As node features we can use the information in the conllu file. We also added pre-trained word vectors form either fasttext, or BERTje. Finally, we add a positional encoding (ie. 'first descendant').

Installation

Setup a virtual env with python3, and install dependencies:

python3 -m venv env
. env/bin/activate
pip install -r requirements.txt

Plans

• Train frames and roles separately, to see effect of multi-task learning;
• Reconsider how we use the BERT wordvectors. We now first encode a whole sentence, and sum the resulting vectors per bert-token to our tokeninzation. Alternatively, per word we find the sentence part it is the head of, and encode that.

Best model until now

Model layers

Net(
  (embedding): Sequential(
    (0): Linear(in_features=189, out_features=100, bias=True)
    (1): BatchNorm1d(100, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (2): ReLU()
  )
  (gru): RGRUGCN(
    in_feats=100, out_feats=100
    (gru): GRU(100, 100)
    (batchnorm): BatchNorm1d(100, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  )
  (task_a): MLP(
    (fc): Sequential(
      (0): Linear(in_features=100, out_features=100, bias=True)
      (1): BatchNorm1d(100, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (2): ReLU()
      (3): Linear(in_features=100, out_features=2, bias=True)
    )
  )
  (task_b): MLP(
    (fc): Sequential(
      (0): Linear(in_features=100, out_features=100, bias=True)
      (1): BatchNorm1d(100, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (2): ReLU()
      (3): Linear(in_features=100, out_features=21, bias=True)
    )
  )
)

Training settings

• Features used UPOS, DEPREL, FEATS, RID, FastText100
• Size of hidden state 100
• Number of GRU iterations 4
• Activation functions relu
• Batch size 50 sentences
• Adam optimizer ADAM
• Initial learning rate 1e-02
• Learning rate scheduler StepLR, gamma=0.9
• Focal Loss, gamma=5.
• The two loss functions were added, both with weight 1

Results

2 classes are so rare, they are not in our 10% evaluation set, and were not predicted by the model. (Arg5 and ArgM-STR).

The confustion matrix and statistics (classification_report) were made with scikit-learn.

The best model was after 6628920 words, or 15 epochs.

Frames

Roles

Ensemble prediction

We trained several models, most of which are similarly good (or bad). An ensemble is a method to further increase the model performance. For this we need a way to combine the indenpendent classifications in a single classifier. A simple way is to use marjority voting: the label with the most votes wins. But this does not make the best use of the separate classifiers. We'll combine the individual label distributions in a single distribution by multiplying the distributions. This is known as 'conflation' (in physics), or 'product-of-experts', or a logarithmic opinion pool.

unweighted

First we'll try an unweighted product.

weighted

We are also looking into using weight factors, following the references below. The idea is to minimize the KL divergence of ensemble.

References

1. Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling
2. Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics
3. Focal Loss for Dense Object Detection
4. GaAN: Gated Attention Networks for Learning on Large and Spatiotemporal Graphs
5. Look Again at the Syntax: Relational Graph Convolutional Network for Gendered Ambiguous Pronoun Resolution code
6. Deep Graph Library
7. The CoNLL-2009 shared task: syntactic and semantic dependencies in multiple languages
8. 2009 Shared task evaluation script
9. Super-Convergence: Very Fast Training of NeuralNetworks Using Large Learning Rates
10. Label Distribution Learning
11. On Loss Functions for Deep Neural Networks in Classification
12. Training Products of Experts by Minimizing Contrastive Divergence
13. Selecting weighting factors in logarithmic opinion pools.pdf

best srl:

runs_srl/ADAM_1e-02_FL1.50e+00cst_50b_100d_2l_reluUPOS_FEATS_DEPREL_WVEC_FT100/model_005925154.pt.eval

best mention:

runs_mentions/ADAMv3.32_1e-03_150b_HL_64d_2lUPOS_FEATS_DEPREL_WVEC_FT50/model_030325091.pt

best coref:

On the test set:

|------|------|------|---------|

(runs_coref/ADAMv4.11_1e-03_50b_HL_100d_2l_WVEC_DEPREL_FT50/model_005840868.pt)

best entity:

On the test set:

|------|------|------|---------|

(runs_entity/entity_v0.52_30_stat/model_000022850.pt)

try it:

python3 run_stanza.py --output jenj.conll jip_en_janneke.txt python3 postprocess_srl.py --output jenj_srl.conll jenj.conll python3 run_mentions.py --output jenj_mentions.conll jenj_srl.conll python3 run_coref.py --html coref.html --output jenj_coref.conll jenj_mentions.conll python3 run_entity.py --html entity.html --output jenj_entity.conll jenj_mentions.conll google-chrome coref.html entity.html