bert-multitask-learning

BERT for Multi-task Learning


Keywords
bert, cws, encoder-decoder, multi-task-learning, multitask-learning, named-entity-recognition, ner, nlp, part-of-speech, pretrained-models, text-classification, transformer, word-segmentation
License
MIT
Install
pip install bert-multitask-learning==0.7.0

Documentation

python tensorflow PyPI version fury.io PyPI license

Bert for Multi-task Learning

中文文档

Note: Since 0.4.0, tf version >= 2.1 is required.

Install

pip install bert-multitask-learning

What is it

This a project that uses BERT to do multi-task learning with multiple GPU support.

Why do I need this

In the original BERT code, neither multi-task learning or multiple GPU training is possible. Plus, the original purpose of this project is NER which dose not have a working script in the original BERT code.

To sum up, compared to the original bert repo, this repo has the following features:

  1. Multi-task learning(major reason of re-writing the majority of code).
  2. Multiple GPU training
  3. Support sequence labeling (for example, NER) and Encoder-Decoder Seq2Seq(with transformer decoder).

What type of problems are supported?

  • Masked LM and next sentence prediction Pre-train(pretrain)
  • Classification(cls)
  • Sequence Labeling(seq_tag)
  • Seq2seq Labeling(seq2seq_tag)
  • Seq2seq Text Generation(seq2seq_text)
  • Multi-Label Classification(multi_cls)

How to run pre-defined problems

There are two types of chaining operations can be used to chain problems.

  • &. If two problems have the same inputs, they can be chained using &. Problems chained by & will be trained at the same time.
  • |. If two problems don't have the same inputs, they need to be chained using |. Problems chained by | will be sampled to train at every instance.

For example, cws|NER|weibo_ner&weibo_cws, one problem will be sampled at each turn, say weibo_ner&weibo_cws, then weibo_ner and weibo_cws will trained for this turn together. Therefore, in a particular batch, some tasks might not be sampled, and their loss could be 0 in this batch.

Please see the examples in notebooks for more details about training, evaluation and export models.

Bert多任务学习

注意:版本0.4.0后要求tf>=2.1

安装

pip install bert-multitask-learning

这是什么

这是利用BERT进行多任务学习并且支持多GPU训练的项目.

我为什么需要这个项目

在原始的BERT代码中, 是没有办法直接用多GPU进行多任务学习的. 另外, BERT并没有给出序列标注和Seq2seq的训练代码.

因此, 和原来的BERT相比, 这个项目具有以下特点:

  1. 多任务学习
  2. 多GPU训练
  3. 序列标注以及Encoder-decoder seq2seq的支持(用transformer decoder)

目前支持的任务类型

  • Masked LM和next sentence prediction预训练(pretrain)
  • 单标签分类(cls)
  • 序列标注(seq_tag)
  • 序列到序列标签标注(seq2seq_tag)
  • 序列到序列文本生成(seq2seq_text)
  • 多标签分类(multi_cls)

如何运行预定义任务

目前支持的任务

  • 中文命名实体识别
  • 中文分词
  • 中文词性标注

可以用两种方法来将多个任务连接起来.

  • &. 如果两个任务有相同的输入, 不同标签的话, 那么他们可以&来连接. 被&连接起来的任务会被同时训练.
  • |. 如果两个任务为不同的输入, 那么他们必须|来连接. 被|连接起来的任务会被随机抽取来训练.

例如, 我们定义任务cws|NER|weibo_ner&weibo_cws, 那么在生成每一条数据时, 一个任务块会被随机抽取出来, 例如在这一次抽样中, weibo_ner&weibo_cws被选中. 那么这次weibo_nerweibo_cws会被同时训练. 因此, 在一个batch中, 有可能某些任务没有被抽中, loss为0.

训练, eval和导出模型请见notebooks