Intent and Slot-filling on the ATIS dataset task.

Adapted for tensorflow 2.0 and shortened in comparison to original
https://github.com/mohammedterry/slots_intents

Model architecture

For this task, a model was trained to jointly predict a sentence's Intent and Slots (entities). Each word is embedded (using pre-defined word vectors) to capture the word's meaning while a character-level bidirection Long Short-Term Memory(LSTM) Network encodes the word's letters to capture its lexical structure.

The word vector and outputs of the character-level bi-LSTM are then fed into the word-level bi-LSTM which predicts the Intent. The second layer feeds into a Conditional Random Fields (CRF) layer to predict the individual slots (entities).

The model is stored in intents_slots/model.py

Word Vectors

The newly released Poincare word embeddings (100 dimensional) were used as they have been reported to better encode the hierarchical relationships inherent between words you can find the word vectors used in word_vectors/poincare.txt

Demo

you can run a demo of the pre-trained model by running intents_slots/demo.py

Training

The model was trained for 50 epochs and stored in the pretrained_models directory

• pretrained_models/dataset_info contains all the vocabularies used by the model (character, word, intent, entity) and their mappings to numbers for encoding/decoding
• pretrained_models/model.h5 are the weights to the model

The model's loss during training over the epochs are shown below:

The model's accuracy at predicting intents and entities (slots) over time are shown below:

You can retrain the model by running intents_slots/train.py

Tests

Joint:

Intents only:

Entites only:

to above results can be obtained by running intents_slots/evaluate.py

Improvements:

To improve the robustness of the model to out of vocab words, the training data was lemmatised prior to training and the model was retrained. Numbers were also masked using a placeholder (e.g. *) to avoid out-of-vocab times appearing (e.g. 9:30 may appear in training but not 9:29).

The results were slightly improved given the above tweaks. Precision, Recall and F1 scores improved across the board (for both intents, entities).

Perfect scores were achieved using the validation set!?? data/atis-2-dev.csv

Future Improvements (TO DO):

• Balance out training data (its clear that the intent ATIS_FLIGHT dominates the training set)

(and 'O' dominates the entity tags)

(or if we discount this as an entity tag - then "to/fromloc.city_name" tags)

• e.g. this can be achieved by subsampling or artificially perterbing data to generate more samples (e.g. increase training instance by sliding each sentences one,two,three,etc places)

• Investigate the Intents & Entities which are scoring relatively low F1 scores

e.g. (intents such as ATIS_DAY_NAME, ATIS_MEAL, ATIS_FLIGHT_TIME, etc)

e.g. (entities such as compartment, booking_class, meal_code, etc)

• Preprocess intent labels with #?
• Embed unknown words too (if possible) rather than giving them (1)
• convert word numbers (e.g. "one") into digits
• improve slot extraction using additional pre-trained Named Entity Recognition (NER)s from various libraries