基于BERT的长文本分类方法

BERT-based approach for long document classification

摘要:

由于预训练模型输入分词数量限制，基于 BERT 的长文本分类任务效果与长文本分割后的文本段集合的处理及特征融合密切相关，现有的长文本分类研究在融合文本段特征时更关注文本段之间原始的顺序关系，而本文提出了一种基于 BERT 和集合神经网络的长文本分类模型.该方法以 BERT 为基础，可处理从同一文本样本分割得到的任意数量文本段，经过 BERT 后得到文本段特征，再将所有文本段特征输入到具有置换不变性的集合神经网络层中，提取出集合级别特征来优化长文本的特征表达.通过在三个数据上的实验分析，论文在平均分词长度较长的数据集上取得了90.82%的准确率，高出目前最优方法4.37%.

Abstract:

Concerning the input limitation of pre-training model, a long document needs to be spit into a set of text segments. The performance of long document classification is closely related to the further processing of the segment set and feature fusion. Existing document classification models keep more attention on the sequential of segments in the text segment set. However, the athors consider that the sequential order of segments have a mild influence on drawing the feature of a long document. The authors propose a BERT based long document classification model, which utilizes deep sets to obtain the collectionlevel feature from the segment set. In the model, the authors obtain a set of text segment features after BERT, and this proposed network which is immune to permutation learns the identical feature of the set to represent the long document feature. The accuracy of our model on the 20 newsgroups dataset achieved 90.82%, which outperforms the state-of-the-art method by 4.37%.