结合主题模型与自监督学习的可控文本生成技术研究

Controllable text generation technology based on topic model and self supervised learning

摘要:

基于大型预训练语言模型的有监督学习方法在可控文本生成任务上取得了优秀的成果，但这些研究都着重于控制生成文本的高级属性(比如情感与主题),而忽略了泛化性问题.现有的基于自监督学习的研究方法则通过句子级别的训练来使模型获得补全整句的能力，使模型做到单词和短语级别的控制生成，但生成与特定属性强相关句子的能力依旧待提升.所以本文提出了一种单词级别(细粒度)与句子(粗粒度)级别相结合的多粒度训练方式：单词级别的主题模型让模型学习主题层面的语义以获得主题到文本的生成能力，句子级别的自监督训练让模型学习整句的表征以获得补全句子的能力.通过主题模型与自监督学习的结合，使模型在单词与短语级别的可控生成阶段取得了更好的效果.实验表明，本文提出的模型在主题契合度以及常规文本生成指标方面优于现有的基线模型.

Abstract:

Supervised learning methods based on large-scale pre-trained language models have achieved excellent results in controllable text generation tasks, but current approaches mainly focus on controlling the high-level attributes of the generated text such as emotion and theme, neglecting the generalization problem. The existing research methods based on self-supervised learning use sentence-level training to enable the model to obtain the ability to complete the entire sentence, so that the model can control the generation of words and phrases, but the generation is strongly related to specific attributes. To address this problem, this paper proposes a multi granularity training method combining word level (fine granularity) and sentence level (coarse granularity): word level topic model lets the model learn the semantics of the topic level to obtain the ability to generate topic to text, and sentence level self-monitoring training lets the model learn the representation of the whole sentence to obtain the ability to complete the sentence. Through the combination of topic model and self supervised learning, the model achieve better results in controlled generation at the word and phrase level. Experiments show that the proposed model is superior to the existing baseline model in terms of topic fit and conventional text generation metrics.