Machine Learning Project On Bitcoin Volatility

Machine Learning Project On Bitcoin Volatility

Time-Series & Market Analytics · ML Project

Python · Data Preprocessing · Model Development · Evaluation

Project focus: prediction and forecasting using timestamped market prices, volumes and derived indicators.

How to Fine-Tune BERT for Text Classification?

Chi Sun, Xipeng Qiu∗, Yige Xu, Xuanjing Huang Shanghai Key Laboratory of Intelligent Information Processing, Fudan University School of Computer Science, Fudan University 825 Zhangheng Road, Shanghai, China {sunc17,xpqiu,ygxu18,xjhuang}@fudan.edu.cn

Abstract 2018). These word embeddings are often used as additional features for the main task. An- Language model pre-training has proven to be other kind of pre-training models is sentence- useful in learning universal language represen- level. Howard and Ruder (2018) propose ULM- arXiv:1905.05583v3 [cs.CL] 5 Feb 2020

tations. As a state-of-the-art language model pre-training model, BERT (Bidirectional En- FiT, a fine-tuning method for pre-trained language coder Representations from Transformers) has model that achieves state-of-the-art results on six achieved amazing results in many language widely studied text classification datasets. More understanding tasks. In this paper, we con- recently, pre-trained language models have shown

duct exhaustive experiments to investigate dif- to be useful in learning common language rep- ferent fine-tuning methods of BERT on text resentations by utilizing a large amount of unla- classification task and provide a general solu- tion for BERT fine-tuning. Finally, the pro- beled data: e.g., OpenAI GPT (Radford et al., posed solution obtains new state-of-the-art re- 2018) and BERT (Devlin et al., 2018). BERT is sults on eight widely-studied text classification based on a multi-layer bidirectional Transformer

datasets.1 (Vaswani et al., 2017) and is trained on plain text for masked word prediction and next sentence pre- 1 Introduction diction tasks. Text classification is a classic problem in Natural Although BERT has achieved amazing results Language Processing (NLP). The task is to assign in many natural language understanding (NLU) predefined categories to a given text sequence. An tasks, its potential has yet to be fully explored.

important intermediate step is the text representa- There is little research to enhance BERT to im- tion. Previous work uses various neural models prove the performance on target tasks further. to learn text representation, including convolution In this paper, we investigate how to maximize models (Kalchbrenner et al., 2014; Zhang et al., the utilization of BERT for the text classifica- 2015; Conneau et al., 2016; Johnson and Zhang, tion task. We explore several ways of fine-tuning

2017; Zhang et al., 2017; Shen et al., 2018), re- BERT to enhance its performance on text classifi- current models (Liu et al., 2016; Yogatama et al., cation task. We design exhaustive experiments to 2017; Seo et al., 2017), and attention mechanisms make a detailed analysis of BERT. (Yang et al., 2016; Lin et al., 2017). The contributions of our paper are as follows: Alternatively, substantial work has shown that

• We propose a general solution to fine-tune pre-trained models on large corpus are beneficial the pre-trained BERT model, which includes for text classification and other NLP tasks, which three steps: (1) further pre-train BERT on can avoid training a new model from scratch. One within-task training data or in-domain data; kind of pre-trained models is the word embed- (2) optional fine-tuning BERT with multi- dings, such as word2vec (Mikolov et al., 2013)

task learning if several related tasks are avail- and GloVe (Pennington et al., 2014), or the con- able; (3) fine-tune BERT for the target task. textualized word embeddings, such as CoVe (Mc- Cann et al., 2017) and ELMo (Peters et al., • We also investigate the fine-tuning meth- ∗ ods for BERT on target task, including pre- Corresponding author 1 The source codes are available at https://github. process of long text, layer selection, layer-

com/xuyige/BERT4doc-Classification. wise learning rate, catastrophic forgetting,

and low-shot learning problems. the-art results for a range of NLP tasks, demon- • We achieve the new state-of-the-art results on strating the enormous potential of the fine-tuning seven widely-studied English text classifica- method. In this paper, we have further explored tion datasets and one Chinese news classifi- the BERT fine-tuning method for text classifica- cation dataset. tion.

2 Related Work 2.2 Multi-task learning Multi-task learning (Caruana, 1993; Collobert and Borrowing the learned knowledge from the other Weston, 2008) is another relevant direction. Rei tasks has a rising interest in the field of NLP. We (2017) and Liu et al. (2018) use this method to briefly review two related approaches: language train the language model and the main task model model pre-training and multi-task Learning. jointly. Liu et al. (2019) extend the MT-DNN

2.1 Language Model Pre-training model originally proposed in Liu et al. (2015) by incorporating BERT as its shared text encoding Pre-trained word embeddings (Mikolov et al., layers. MTL requires training tasks from scratch 2013; Pennington et al., 2014), as an important every time, which makes it inefficient and it usu- component of modern NLP systems can offer sig- ally requires careful weighing of task-specific ob-

nificant improvements over embeddings learned jective functions (Chen et al., 2017). However, from scratch. The generalization of word embed- we can use multi-task BERT fine-tuning to avoid dings, such as sentence embeddings (Kiros et al., this problem by making full use of the shared pre- 2015; Logeswaran and Lee, 2018) or paragraph trained model. embeddings (Le and Mikolov, 2014), are also used as features in downstream models. 3 BERT for Text Classification

Peters et al. (2018) concatenate embeddings de- rived from language model as additional features BERT-base model contains an encoder with 12 for the main task and advance the state-of-the- Transformer blocks, 12 self-attention heads, and art for several major NLP benchmarks. In addi- the hidden size of 768. BERT takes an input of tion to pre-training with unsupervised data, trans- a sequence of no more than 512 tokens and out-

fer learning with a large amount of supervised data puts the representation of the sequence. The se- can also achieve good performance, such as natu- quence has one or two segments that the first token ral language inference (Conneau et al., 2017) and of the sequence is always [CLS] which contains machine translation (McCann et al., 2017). the special classification embedding and another More recently, the method of pre-training lan- special token [SEP] is used for separating seg-

guage models on a large network with a large ments. amount of unlabeled data and fine-tuning in down- For text classification tasks, BERT takes the fi- stream tasks has made a breakthrough in sev- nal hidden state h of the first token [CLS] as the eral natural language understanding tasks, such as representation of the whole sequence. A simple OpenAI GPT (Radford et al., 2018) and BERT softmax classifier is added to the top of BERT to

(Devlin et al., 2018). Dai and Le (2015) use lan- predict the probability of label c: guage model fine-tuning but overfit with 10k la- beled examples while Howard and Ruder (2018) p(c|h) = softmax(W h), (1) propose ULMFiT and achieve state-of-the-art re- where W is the task-specific parameter matrix. sults in the text classification task. BERT is We fine-tune all the parameters from BERT as well pre-trained on Masked Language Model Task and

as W jointly by maximizing the log-probability of Next Sentence Prediction Task via a large cross- the correct label. domain corpus. Unlike previous bidirectional lan- guage models (biLM) limited to a combination of 4 Methodology two unidirectional language models (i.e., left-to- right and right-to-left), BERT uses a Masked Lan- When we adapt BERT to NLP tasks in a target guage Model to predict words which are randomly domain, a proper fine-tuning strategy is desired.

masked or replaced. BERT is the first fine-tuning In this paper, we look for the proper fine-tuning based representation model that achieves state-of- methods in the following three ways.

Single-Task tains the parameters of the l-th layer of BERT. Fine-Tuning Then the parameters are updated as follows: Further BERT Pre-training Multi-Task θtl = θt−1 l − η l · ∇θl J(θ), (2) Fine-Tuning where η l represents the learning rate of the l-th shown with different colors. We set the base learning rate to η L and use η k−1 = ξ · η k , where ξ is a decay factor and less

than or equal to 1. When ξ < 1, the lower layer 1) Fine-Tuning Strategies: When we fine-tune has a lower learning rate than the higher layer. BERT for a target task, there are many ways to When ξ = 1, all layers have the same learning utilize BERT. For example, the different layers rate, which is equivalent to the regular stochastic of BERT capture different levels of semantic and gradient descent (SGD). We will investigate these syntactic information, which layer is better for a

factors in Sec. 5.3. target task? How we choose a better optimization algorithm and learning rate? 4.2 Further Pre-training 2) Further Pre-training: BERT is trained in The BERT model is pre-trained in the general- the general domain, which has a different data dis- domain corpus. For a text classification task in a tribution from the target domain. A natural idea is specific domain, such as movie reviews, its data to further pre-train BERT with target domain data.

distribution may be different from BERT. There- 3) Multi-Task Fine-Tuning: Without pre- fore, we can further pre-train BERT with masked trained LM models, multi-task learning has shown language model and next sentence prediction tasks its effectiveness of exploiting the shared knowl- on the domain-specific data. Three further pre- edge among the multiple tasks. When there are training approaches are performed: several available tasks in a target domain, an inter-

1) Within-task pre-training, in which BERT is esting question is whether it still bring benefits to further pre-trained on the training data of a target fine-tune BERT on all the tasks simultaneously. task. Our general methodology of fine-tuning BERT 2) In-domain pre-training, in which the pre- is shown in Figure 1. training data is obtained from the same domain of a target task. For example, there are several dif- 4.1 Fine-Tuning Strategies ferent sentiment classification tasks, which have a

Different layers of a neural network can capture similar data distribution. We can further pre-train different levels of syntactic and semantic infor- BERT on the combined training data from these mation (Yosinski et al., 2014; Howard and Ruder, tasks. 2018). 3) Cross-domain pre-training, in which the pre- To adapt BERT to a target task, we need to con- training data is obtained from both the same and

sider several factors: 1) The first factor is the pre- other different domains to a target task. processing of long text since the maximum se- We will investigate these different approaches quence length of BERT is 512. 2) The second to further pre-training in Sec. 5.4. factor is layer selection. The official BERT-base model consists of an embedding layer, a 12-layer 4.3 Multi-Task Fine-Tuning encoder, and a pooling layer. We need to select the Multi-task Learning is also an effective approach

most effective layer for the text classification task. to share the knowledge obtained from several re- 3) The third factor is the overfitting problem. A lated supervised tasks. Similar to Liu et al. (2019), better optimizer with an appropriate learning rate we also use fine-tune BERT in multi-task learning is desired. framework for text classification. Intuitively, the lower layer of the BERT model All the tasks share the BERT layers and the em-

may contain more general information. We can bedding layer. The only layer that does not share fine-tune them with different learning rates. is the final classification layer, which means that Following Howard and Ruder (2018), we split each task has a private classifier layer. The exper- the parameters θ into {θ1 , · · · , θL } where θl con- imental analysis is in Sec. 5.5.

Average Max Exceeding Train Test Dataset Classes Type lengths lengths ratio samples samples IMDb 2 Sentiment 292 3,045 12.69% 25,000 25,000 Yelp P. 2 Sentiment 177 2,066 4.60% 560,000 38,000 Yelp F. 5 Sentiment 179 2,342 4.60% 650,000 50,000

TREC 6 Question 11 39 0.00% 5,452 500 Yahoo! Answers 10 Question 131 4,018 2.65% 1,400,000 60,000 AG’s News 4 Topic 44 221 0.00% 120,000 7,600 DBPedia 14 Topic 67 3,841 0.00% 560,000 70,000 Sogou News 6 Topic 737 47,988 46.23% 54,000 6,000

samples with a length exceeding 512.

5 Experiments fectiveness of BERT for Chinese text, we cre- ate the Chinese training and test datasets for So- We investigate the different fine-tuning methods gou news corpus. Unlike Zhang et al. (2015), for seven English and one Chinese text classifica- we use the Chinese character directly rather than tion tasks. We use the base BERT models: the un- Pinyin. The dataset is a combination of the So-

cased BERT-base model2 and the Chinese BERT- gouCA and SogouCS news corpora (Wang et al., base model3 respectively. 2008). We determine the category of the news 5.1 Datasets based on the URL, such as “sports” correspond- ing to “http://sports.sohu.com”. We choose 6 cate- We evaluate our approach on eight widely-studied gories – “sports”, “house”, “business”, “entertain- datasets. These datasets have varying numbers of

ment”, “women” and “technology”. The number documents and varying document lengths, cover- of training samples selected for each class is 9,000 ing three common text classification tasks: sen- and testing 1,000. timent analysis, question classification, and topic Data preprocessing Following Devlin et al. classification. We show the statistics for each (2018), we use WordPiece embeddings (Wu et al., dataset in Table 1. 2016) with a 30,000 token vocabulary and de-

Sentiment analysis For sentiment analysis, note split word pieces with ##. So the statis- we use the binary film review IMDb dataset (Maas tics of the length of the documents in the datasets et al., 2011) and the binary and five-class version are based on the word pieces. For further pre- of the Yelp review dataset built by Zhang et al. training with BERT, we use spaCy4 to perform (2015). sentence segmentation in English datasets and we

Question classification For question classifi- use “。”,“？” and “！” as separators when deal- cation, we evaluate our method on the six-class ing with the Chinese Sogou News dataset. version of the TREC dataset (Voorhees and Tice, 1999) and Yahoo! Answers dataset created by 5.2 Hyperparameters Zhang et al. (2015). TREC dataset is dataset for We use the BERT-base model (Devlin et al., 2018)

question classification consisting of open-domain, with a hidden size of 768, 12 Transformer blocks fact-based questions divided into broad seman- (Vaswani et al., 2017) and 12 self-attention heads. tic categories. Compared to other document-level We further pre-train with BERT on 1 TITAN Xp datasets, TREC dataset is sentence-level, and there GPU, with a batch size of 32, max squence length are fewer training examples for it. Yahoo! An- of 128, learning rate of 5e-5, train steps of 100,000

swers dataset is a big dataset with 1,400k train and warm-up steps of 10,000. samples. We fine-tune the BERT model on 4 TITAN Xp Topic classification For topic classification, we GPUs and set the batch size to 24 to ensure that the use large-scale AG’s News and DBPedia cre- GPU memory is fully utilized. The dropout prob- ated by Zhang et al. (2015). To test the ef- ability is always kept at 0.1. We use Adam with

2 https://storage.googleapis.com/bert models/2018 10 18/ β1 = 0.9 and β2 = 0.999. We use slanted triangu- uncased L-12 H-768 A-12.zip lar learning rates (Howard and Ruder, 2018), the 3 https://storage.googleapis.com/bert models/2018 11 03/ 4 chinese L-12 H-768 A-12.zip https://spacy.io/

base learning rate is 2e-5, and the warm-up pro- Method IMDb Sogou portion is 0.1. We empirically set the max number head-only 5.63 2.58 of the epoch to 4 and save the best model on the tail-only 5.44 3.17 head+tail 5.42 2.43 validation set for testing. hier. mean 5.89 2.83

hier. max 5.71 2.47 5.3 Exp-I: Investigating Different hier. self-attention 5.49 2.65 Fine-Tuning Strategies In this subsection, we use the IMDb dataset to in- Table 2: Test error rates (%) on IMDb and Chinese So- gou News datasets. vestigate the different fine-tuning strategies. The official pre-trained model is set as the initial en- coder5 . last layer of BERT gives the best performance.

5.3.1 Dealing with long texts Therefore, we use this setting for the following ex- The maximum sequence length of BERT is 512. periments. The first problem of applying BERT to text clas- sification is how processing the text with a length Layer Test error rates(%) larger than 512. We try the following ways for Layer-0 11.07 dealing with long articles. Layer-1 9.81

Layer-2 9.29 Truncation methods Usually, the key informa- Layer-3 8.66 tion of an article is at the beginning and end. We Layer-4 7.83 Layer-5 6.83 use three different methods of truncate text to per- Layer-6 6.83 form BERT fine-tuning. Layer-7 6.41

Layer-8 6.04 1. head-only: keep the first 510 tokens6 ; Layer-9 5.70 2. tail-only: keep the last 510 tokens; Layer-10 5.46 Layer-11 5.42 3. head+tail: empirically select the first 128 First 4 Layers + concat 8.69 and the last 382 tokens. First 4 Layers + mean 9.09

First 4 Layers + max 8.76 Hierarchical methods The input text is firstly divided into k = L/510 fractions, which is fed Last 4 Layers + concat 5.43 Last 4 Layers + mean 5.44 into BERT to obtain the representation of the k Last 4 Layers + max 5.42 text fractions. The representation of each fraction All 12 Layers + concat 5.44 is the hidden state of the [CLS] tokens of the last

layer. Then we use mean pooling, max pooling Table 3: Fine-tuning BERT with different layers on and self-attention to combine the representations IMDb dataset. of all the fractions. methods. The truncation method of head+tail achieves the best performance on IMDb and So- 5.3.3 Catastrophic Forgetting gou datasets. Therefore, we use this method to Catastrophic forgetting (McCloskey and Cohen, deal with the long text in the following experi- 1989) is usually a common problem in trans-

ments. fer learning, which means the pre-trained knowl- 5.3.2 Features from Different layers edge is erased during learning of new knowledge. Each layer of BERT captures the different features Therefore, we also investigate whether BERT suf- of the input text. We investigate the effectiveness fers from the catastrophic forgetting problem. of features from different layers. We then fine- We fine-tune BERT with different learning

Continued Discussion and Extended Analysis

How to Fine-Tune BERT for Text Classification?

Chi Sun, Xipeng Qiu∗, Yige Xu, Xuanjing Huang Shanghai Key Laboratory of Intelligent Information Processing, Fudan University School of Computer Science, Fudan University 825 Zhangheng Road, Shanghai, China {sunc17,xpqiu,ygxu18,xjhuang}@fudan.edu.cn

Abstract 2018). These word embeddings are often used as additional features for the main task. An- Language model pre-training has proven to be other kind of pre-training models is sentence- useful in learning universal language represen- level. Howard and Ruder (2018) propose ULM- arXiv:1905.05583v3 [cs.CL] 5 Feb 2020

tations. As a state-of-the-art language model pre-training model, BERT (Bidirectional En- FiT, a fine-tuning method for pre-trained language coder Representations from Transformers) has model that achieves state-of-the-art results on six achieved amazing results in many language widely studied text classification datasets. More understanding tasks. In this paper, we con- recently, pre-trained language models have shown

duct exhaustive experiments to investigate dif- to be useful in learning common language rep- ferent fine-tuning methods of BERT on text resentations by utilizing a large amount of unla- classification task and provide a general solu- tion for BERT fine-tuning. Finally, the pro- beled data: e.g., OpenAI GPT (Radford et al., posed solution obtains new state-of-the-art re- 2018) and BERT (Devlin et al., 2018). BERT is sults on eight widely-studied text classification based on a multi-layer bidirectional Transformer

datasets.1 (Vaswani et al., 2017) and is trained on plain text for masked word prediction and next sentence pre- 1 Introduction diction tasks. Text classification is a classic problem in Natural Although BERT has achieved amazing results Language Processing (NLP). The task is to assign in many natural language understanding (NLU) predefined categories to a given text sequence. An tasks, its potential has yet to be fully explored.

important intermediate step is the text representa- There is little research to enhance BERT to im- tion. Previous work uses various neural models prove the performance on target tasks further. to learn text representation, including convolution In this paper, we investigate how to maximize models (Kalchbrenner et al., 2014; Zhang et al., the utilization of BERT for the text classifica- 2015; Conneau et al., 2016; Johnson and Zhang, tion task. We explore several ways of fine-tuning

2017; Zhang et al., 2017; Shen et al., 2018), re- BERT to enhance its performance on text classifi- current models (Liu et al., 2016; Yogatama et al., cation task. We design exhaustive experiments to 2017; Seo et al., 2017), and attention mechanisms make a detailed analysis of BERT. (Yang et al., 2016; Lin et al., 2017). The contributions of our paper are as follows: Alternatively, substantial work has shown that

• We propose a general solution to fine-tune pre-trained models on large corpus are beneficial the pre-trained BERT model, which includes for text classification and other NLP tasks, which three steps: (1) further pre-train BERT on can avoid training a new model from scratch. One within-task training data or in-domain data; kind of pre-trained models is the word embed- (2) optional fine-tuning BERT with multi- dings, such as word2vec (Mikolov et al., 2013)

task learning if several related tasks are avail- and GloVe (Pennington et al., 2014), or the con- able; (3) fine-tune BERT for the target task. textualized word embeddings, such as CoVe (Mc- Cann et al., 2017) and ELMo (Peters et al., • We also investigate the fine-tuning meth- ∗ ods for BERT on target task, including pre- Corresponding author 1 The source codes are available at https://github. process of long text, layer selection, layer-

com/xuyige/BERT4doc-Classification. wise learning rate, catastrophic forgetting,

and low-shot learning problems. the-art results for a range of NLP tasks, demon- • We achieve the new state-of-the-art results on strating the enormous potential of the fine-tuning seven widely-studied English text classifica- method. In this paper, we have further explored tion datasets and one Chinese news classifi- the BERT fine-tuning method for text classifica- cation dataset. tion.

2 Related Work 2.2 Multi-task learning Multi-task learning (Caruana, 1993; Collobert and Borrowing the learned knowledge from the other Weston, 2008) is another relevant direction. Rei tasks has a rising interest in the field of NLP. We (2017) and Liu et al. (2018) use this method to briefly review two related approaches: language train the language model and the main task model model pre-training and multi-task Learning. jointly. Liu et al. (2019) extend the MT-DNN

2.1 Language Model Pre-training model originally proposed in Liu et al. (2015) by incorporating BERT as its shared text encoding Pre-trained word embeddings (Mikolov et al., layers. MTL requires training tasks from scratch 2013; Pennington et al., 2014), as an important every time, which makes it inefficient and it usu- component of modern NLP systems can offer sig- ally requires careful weighing of task-specific ob-

nificant improvements over embeddings learned jective functions (Chen et al., 2017). However, from scratch. The generalization of word embed- we can use multi-task BERT fine-tuning to avoid dings, such as sentence embeddings (Kiros et al., this problem by making full use of the shared pre- 2015; Logeswaran and Lee, 2018) or paragraph trained model. embeddings (Le and Mikolov, 2014), are also used as features in downstream models. 3 BERT for Text Classification

Peters et al. (2018) concatenate embeddings de- rived from language model as additional features BERT-base model contains an encoder with 12 for the main task and advance the state-of-the- Transformer blocks, 12 self-attention heads, and art for several major NLP benchmarks. In addi- the hidden size of 768. BERT takes an input of tion to pre-training with unsupervised data, trans- a sequence of no more than 512 tokens and out-

fer learning with a large amount of supervised data puts the representation of the sequence. The se- can also achieve good performance, such as natu- quence has one or two segments that the first token ral language inference (Conneau et al., 2017) and of the sequence is always [CLS] which contains machine translation (McCann et al., 2017). the special classification embedding and another More recently, the method of pre-training lan- special token [SEP] is used for separating seg-

guage models on a large network with a large ments. amount of unlabeled data and fine-tuning in down- For text classification tasks, BERT takes the fi- stream tasks has made a breakthrough in sev- nal hidden state h of the first token [CLS] as the eral natural language understanding tasks, such as representation of the whole sequence. A simple OpenAI GPT (Radford et al., 2018) and BERT softmax classifier is added to the top of BERT to

(Devlin et al., 2018). Dai and Le (2015) use lan- predict the probability of label c: guage model fine-tuning but overfit with 10k la- beled examples while Howard and Ruder (2018) p(c|h) = softmax(W h), (1) propose ULMFiT and achieve state-of-the-art re- where W is the task-specific parameter matrix. sults in the text classification task. BERT is We fine-tune all the parameters from BERT as well pre-trained on Masked Language Model Task and

as W jointly by maximizing the log-probability of Next Sentence Prediction Task via a large cross- the correct label. domain corpus. Unlike previous bidirectional lan- guage models (biLM) limited to a combination of 4 Methodology two unidirectional language models (i.e., left-to- right and right-to-left), BERT uses a Masked Lan- When we adapt BERT to NLP tasks in a target guage Model to predict words which are randomly domain, a proper fine-tuning strategy is desired.

masked or replaced. BERT is the first fine-tuning In this paper, we look for the proper fine-tuning based representation model that achieves state-of- methods in the following three ways.

Single-Task tains the parameters of the l-th layer of BERT. Fine-Tuning Then the parameters are updated as follows: Further BERT Pre-training Multi-Task θtl = θt−1 l − η l · ∇θl J(θ), (2) Fine-Tuning where η l represents the learning rate of the l-th shown with different colors. We set the base learning rate to η L and use η k−1 = ξ · η k , where ξ is a decay factor and less

than or equal to 1. When ξ < 1, the lower layer 1) Fine-Tuning Strategies: When we fine-tune has a lower learning rate than the higher layer. BERT for a target task, there are many ways to When ξ = 1, all layers have the same learning utilize BERT. For example, the different layers rate, which is equivalent to the regular stochastic of BERT capture different levels of semantic and gradient descent (SGD). We will investigate these syntactic information, which layer is better for a

factors in Sec. 5.3. target task? How we choose a better optimization algorithm and learning rate? 4.2 Further Pre-training 2) Further Pre-training: BERT is trained in The BERT model is pre-trained in the general- the general domain, which has a different data dis- domain corpus. For a text classification task in a tribution from the target domain. A natural idea is specific domain, such as movie reviews, its data to further pre-train BERT with target domain data.

distribution may be different from BERT. There- 3) Multi-Task Fine-Tuning: Without pre- fore, we can further pre-train BERT with masked trained LM models, multi-task learning has shown language model and next sentence prediction tasks its effectiveness of exploiting the shared knowl- on the domain-specific data. Three further pre- edge among the multiple tasks. When there are training approaches are performed: several available tasks in a target domain, an inter-

1) Within-task pre-training, in which BERT is esting question is whether it still bring benefits to further pre-trained on the training data of a target fine-tune BERT on all the tasks simultaneously. task. Our general methodology of fine-tuning BERT 2) In-domain pre-training, in which the pre- is shown in Figure 1. training data is obtained from the same domain of a target task. For example, there are several dif- 4.1 Fine-Tuning Strategies ferent sentiment classification tasks, which have a

Different layers of a neural network can capture similar data distribution. We can further pre-train different levels of syntactic and semantic infor- BERT on the combined training data from these mation (Yosinski et al., 2014; Howard and Ruder, tasks. 2018). 3) Cross-domain pre-training, in which the pre- To adapt BERT to a target task, we need to con- training data is obtained from both the same and

sider several factors: 1) The first factor is the pre- other different domains to a target task. processing of long text since the maximum se- We will investigate these different approaches quence length of BERT is 512. 2) The second to further pre-training in Sec. 5.4. factor is layer selection. The official BERT-base model consists of an embedding layer, a 12-layer 4.3 Multi-Task Fine-Tuning encoder, and a pooling layer. We need to select the Multi-task Learning is also an effective approach

most effective layer for the text classification task. to share the knowledge obtained from several re- 3) The third factor is the overfitting problem. A lated supervised tasks. Similar to Liu et al. (2019), better optimizer with an appropriate learning rate we also use fine-tune BERT in multi-task learning is desired. framework for text classification. Intuitively, the lower layer of the BERT model All the tasks share the BERT layers and the em-

may contain more general information. We can bedding layer. The only layer that does not share fine-tune them with different learning rates. is the final classification layer, which means that Following Howard and Ruder (2018), we split each task has a private classifier layer. The exper- the parameters θ into {θ1 , · · · , θL } where θl con- imental analysis is in Sec. 5.5.

Average Max Exceeding Train Test Dataset Classes Type lengths lengths ratio samples samples IMDb 2 Sentiment 292 3,045 12.69% 25,000 25,000 Yelp P. 2 Sentiment 177 2,066 4.60% 560,000 38,000 Yelp F. 5 Sentiment 179 2,342 4.60% 650,000 50,000

TREC 6 Question 11 39 0.00% 5,452 500 Yahoo! Answers 10 Question 131 4,018 2.65% 1,400,000 60,000 AG’s News 4 Topic 44 221 0.00% 120,000 7,600 DBPedia 14 Topic 67 3,841 0.00% 560,000 70,000 Sogou News 6 Topic 737 47,988 46.23% 54,000 6,000

samples with a length exceeding 512.

5 Experiments fectiveness of BERT for Chinese text, we cre- ate the Chinese training and test datasets for So- We investigate the different fine-tuning methods gou news corpus. Unlike Zhang et al. (2015), for seven English and one Chinese text classifica- we use the Chinese character directly rather than tion tasks. We use the base BERT models: the un- Pinyin. The dataset is a combination of the So-

cased BERT-base model2 and the Chinese BERT- gouCA and SogouCS news corpora (Wang et al., base model3 respectively. 2008). We determine the category of the news 5.1 Datasets based on the URL, such as “sports” correspond- ing to “http://sports.sohu.com”. We choose 6 cate- We evaluate our approach on eight widely-studied gories – “sports”, “house”, “business”, “entertain- datasets. These datasets have varying numbers of

ment”, “women” and “technology”. The number documents and varying document lengths, cover- of training samples selected for each class is 9,000 ing three common text classification tasks: sen- and testing 1,000. timent analysis, question classification, and topic Data preprocessing Following Devlin et al. classification. We show the statistics for each (2018), we use WordPiece embeddings (Wu et al., dataset in Table 1. 2016) with a 30,000 token vocabulary and de-

Sentiment analysis For sentiment analysis, note split word pieces with ##. So the statis- we use the binary film review IMDb dataset (Maas tics of the length of the documents in the datasets et al., 2011) and the binary and five-class version are based on the word pieces. For further pre- of the Yelp review dataset built by Zhang et al. training with BERT, we use spaCy4 to perform (2015). sentence segmentation in English datasets and we

Question classification For question classifi- use “。”,“？” and “！” as separators when deal- cation, we evaluate our method on the six-class ing with the Chinese Sogou News dataset. version of the TREC dataset (Voorhees and Tice, 1999) and Yahoo! Answers dataset created by 5.2 Hyperparameters Zhang et al. (2015). TREC dataset is dataset for We use the BERT-base model (Devlin et al., 2018)

question classification consisting of open-domain, with a hidden size of 768, 12 Transformer blocks fact-based questions divided into broad seman- (Vaswani et al., 2017) and 12 self-attention heads. tic categories. Compared to other document-level We further pre-train with BERT on 1 TITAN Xp datasets, TREC dataset is sentence-level, and there GPU, with a batch size of 32, max squence length are fewer training examples for it. Yahoo! An- of 128, learning rate of 5e-5, train steps of 100,000

swers dataset is a big dataset with 1,400k train and warm-up steps of 10,000. samples. We fine-tune the BERT model on 4 TITAN Xp Topic classification For topic classification, we GPUs and set the batch size to 24 to ensure that the use large-scale AG’s News and DBPedia cre- GPU memory is fully utilized. The dropout prob- ated by Zhang et al. (2015). To test the ef- ability is always kept at 0.1. We use Adam with

2 https://storage.googleapis.com/bert models/2018 10 18/ β1 = 0.9 and β2 = 0.999. We use slanted triangu- uncased L-12 H-768 A-12.zip lar learning rates (Howard and Ruder, 2018), the 3 https://storage.googleapis.com/bert models/2018 11 03/ 4 chinese L-12 H-768 A-12.zip https://spacy.io/

base learning rate is 2e-5, and the warm-up pro- Method IMDb Sogou portion is 0.1. We empirically set the max number head-only 5.63 2.58 of the epoch to 4 and save the best model on the tail-only 5.44 3.17 head+tail 5.42 2.43 validation set for testing. hier. mean 5.89 2.83

hier. max 5.71 2.47 5.3 Exp-I: Investigating Different hier. self-attention 5.49 2.65 Fine-Tuning Strategies In this subsection, we use the IMDb dataset to in- Table 2: Test error rates (%) on IMDb and Chinese So- gou News datasets. vestigate the different fine-tuning strategies. The official pre-trained model is set as the initial en- coder5 . last layer of BERT gives the best performance.

5.3.1 Dealing with long texts Therefore, we use this setting for the following ex- The maximum sequence length of BERT is 512. periments. The first problem of applying BERT to text clas- sification is how processing the text with a length Layer Test error rates(%) larger than 512. We try the following ways for Layer-0 11.07 dealing with long articles. Layer-1 9.81

Layer-2 9.29 Truncation methods Usually, the key informa- Layer-3 8.66 tion of an article is at the beginning and end. We Layer-4 7.83 Layer-5 6.83 use three different methods of truncate text to per- Layer-6 6.83 form BERT fine-tuning. Layer-7 6.41

Layer-8 6.04 1. head-only: keep the first 510 tokens6 ; Layer-9 5.70 2. tail-only: keep the last 510 tokens; Layer-10 5.46 Layer-11 5.42 3. head+tail: empirically select the first 128 First 4 Layers + concat 8.69 and the last 382 tokens. First 4 Layers + mean 9.09

First 4 Layers + max 8.76 Hierarchical methods The input text is firstly divided into k = L/510 fractions, which is fed Last 4 Layers + concat 5.43 Last 4 Layers + mean 5.44 into BERT to obtain the representation of the k Last 4 Layers + max 5.42 text fractions. The representation of each fraction All 12 Layers + concat 5.44 is the hidden state of the [CLS] tokens of the last

layer. Then we use mean pooling, max pooling Table 3: Fine-tuning BERT with different layers on and self-attention to combine the representations IMDb dataset. of all the fractions. methods. The truncation method of head+tail achieves the best performance on IMDb and So- 5.3.3 Catastrophic Forgetting gou datasets. Therefore, we use this method to Catastrophic forgetting (McCloskey and Cohen, deal with the long text in the following experi- 1989) is usually a common problem in trans-

ments. fer learning, which means the pre-trained knowl- 5.3.2 Features from Different layers edge is erased during learning of new knowledge. Each layer of BERT captures the different features Therefore, we also investigate whether BERT suf- of the input text. We investigate the effectiveness fers from the catastrophic forgetting problem. of features from different layers. We then fine- We fine-tune BERT with different learning

Continued Discussion and Extended Analysis

How to Fine-Tune BERT for Text Classification?

Chi Sun, Xipeng Qiu∗, Yige Xu, Xuanjing Huang Shanghai Key Laboratory of Intelligent Information Processing, Fudan University School of Computer Science, Fudan University 825 Zhangheng Road, Shanghai, China {sunc17,xpqiu,ygxu18,xjhuang}@fudan.edu.cn

Abstract 2018). These word embeddings are often used as additional features for the main task. An- Language model pre-training has proven to be other kind of pre-training models is sentence- useful in learning universal language represen- level. Howard and Ruder (2018) propose ULM- arXiv:1905.05583v3 [cs.CL] 5 Feb 2020

tations. As a state-of-the-art language model pre-training model, BERT (Bidirectional En- FiT, a fine-tuning method for pre-trained language coder Representations from Transformers) has model that achieves state-of-the-art results on six achieved amazing results in many language widely studied text classification datasets. More understanding tasks. In this paper, we con- recently, pre-trained language models have shown

duct exhaustive experiments to investigate dif- to be useful in learning common language rep- ferent fine-tuning methods of BERT on text resentations by utilizing a large amount of unla- classification task and provide a general solu- tion for BERT fine-tuning. Finally, the pro- beled data: e.g., OpenAI GPT (Radford et al., posed solution obtains new state-of-the-art re- 2018) and BERT (Devlin et al., 2018). BERT is sults on eight widely-studied text classification based on a multi-layer bidirectional Transformer

datasets.1 (Vaswani et al., 2017) and is trained on plain text for masked word prediction and next sentence pre- 1 Introduction diction tasks. Text classification is a classic problem in Natural Although BERT has achieved amazing results Language Processing (NLP). The task is to assign in many natural language understanding (NLU) predefined categories to a given text sequence. An tasks, its potential has yet to be fully explored.

important intermediate step is the text representa- There is little research to enhance BERT to im- tion. Previous work uses various neural models prove the performance on target tasks further. to learn text representation, including convolution In this paper, we investigate how to maximize models (Kalchbrenner et al., 2014; Zhang et al., the utilization of BERT for the text classifica- 2015; Conneau et al., 2016; Johnson and Zhang, tion task. We explore several ways of fine-tuning

2017; Zhang et al., 2017; Shen et al., 2018), re- BERT to enhance its performance on text classifi- current models (Liu et al., 2016; Yogatama et al., cation task. We design exhaustive experiments to 2017; Seo et al., 2017), and attention mechanisms make a detailed analysis of BERT. (Yang et al., 2016; Lin et al., 2017). The contributions of our paper are as follows: Alternatively, substantial work has shown that

• We propose a general solution to fine-tune pre-trained models on large corpus are beneficial the pre-trained BERT model, which includes for text classification and other NLP tasks, which three steps: (1) further pre-train BERT on can avoid training a new model from scratch. One within-task training data or in-domain data; kind of pre-trained models is the word embed- (2) optional fine-tuning BERT with multi- dings, such as word2vec (Mikolov et al., 2013)

task learning if several related tasks are avail- and GloVe (Pennington et al., 2014), or the con- able; (3) fine-tune BERT for the target task. textualized word embeddings, such as CoVe (Mc- Cann et al., 2017) and ELMo (Peters et al., • We also investigate the fine-tuning meth- ∗ ods for BERT on target task, including pre- Corresponding author 1 The source codes are available at https://github. process of long text, layer selection, layer-

com/xuyige/BERT4doc-Classification. wise learning rate, catastrophic forgetting,

and low-shot learning problems. the-art results for a range of NLP tasks, demon- • We achieve the new state-of-the-art results on strating the enormous potential of the fine-tuning seven widely-studied English text classifica- method. In this paper, we have further explored tion datasets and one Chinese news classifi- the BERT fine-tuning method for text classifica- cation dataset. tion.

2 Related Work 2.2 Multi-task learning Multi-task learning (Caruana, 1993; Collobert and Borrowing the learned knowledge from the other Weston, 2008) is another relevant direction. Rei tasks has a rising interest in the field of NLP. We (2017) and Liu et al. (2018) use this method to briefly review two related approaches: language train the language model and the main task model model pre-training and multi-task Learning. jointly. Liu et al. (2019) extend the MT-DNN

2.1 Language Model Pre-training model originally proposed in Liu et al. (2015) by incorporating BERT as its shared text encoding Pre-trained word embeddings (Mikolov et al., layers. MTL requires training tasks from scratch 2013; Pennington et al., 2014), as an important every time, which makes it inefficient and it usu- component of modern NLP systems can offer sig- ally requires careful weighing of task-specific ob-

nificant improvements over embeddings learned jective functions (Chen et al., 2017). However, from scratch. The generalization of word embed- we can use multi-task BERT fine-tuning to avoid dings, such as sentence embeddings (Kiros et al., this problem by making full use of the shared pre- 2015; Logeswaran and Lee, 2018) or paragraph trained model. embeddings (Le and Mikolov, 2014), are also used as features in downstream models. 3 BERT for Text Classification

Peters et al. (2018) concatenate embeddings de- rived from language model as additional features BERT-base model contains an encoder with 12 for the main task and advance the state-of-the- Transformer blocks, 12 self-attention heads, and art for several major NLP benchmarks. In addi- the hidden size of 768. BERT takes an input of tion to pre-training with unsupervised data, trans- a sequence of no more than 512 tokens and out-

fer learning with a large amount of supervised data puts the representation of the sequence. The se- can also achieve good performance, such as natu- quence has one or two segments that the first token ral language inference (Conneau et al., 2017) and of the sequence is always [CLS] which contains machine translation (McCann et al., 2017). the special classification embedding and another More recently, the method of pre-training lan- special token [SEP] is used for separating seg-

guage models on a large network with a large ments. amount of unlabeled data and fine-tuning in down- For text classification tasks, BERT takes the fi- stream tasks has made a breakthrough in sev- nal hidden state h of the first token [CLS] as the eral natural language understanding tasks, such as representation of the whole sequence. A simple OpenAI GPT (Radford et al., 2018) and BERT softmax classifier is added to the top of BERT to

(Devlin et al., 2018). Dai and Le (2015) use lan- predict the probability of label c: guage model fine-tuning but overfit with 10k la- beled examples while Howard and Ruder (2018) p(c|h) = softmax(W h), (1) propose ULMFiT and achieve state-of-the-art re- where W is the task-specific parameter matrix. sults in the text classification task. BERT is We fine-tune all the parameters from BERT as well pre-trained on Masked Language Model Task and

as W jointly by maximizing the log-probability of Next Sentence Prediction Task via a large cross- the correct label. domain corpus. Unlike previous bidirectional lan- guage models (biLM) limited to a combination of 4 Methodology two unidirectional language models (i.e., left-to- right and right-to-left), BERT uses a Masked Lan- When we adapt BERT to NLP tasks in a target guage Model to predict words which are randomly domain, a proper fine-tuning strategy is desired.

masked or replaced. BERT is the first fine-tuning In this paper, we look for the proper fine-tuning based representation model that achieves state-of- methods in the following three ways.

Single-Task tains the parameters of the l-th layer of BERT. Fine-Tuning Then the parameters are updated as follows: Further BERT Pre-training Multi-Task θtl = θt−1 l − η l · ∇θl J(θ), (2) Fine-Tuning where η l represents the learning rate of the l-th shown with different colors. We set the base learning rate to η L and use η k−1 = ξ · η k , where ξ is a decay factor and less

than or equal to 1. When ξ < 1, the lower layer 1) Fine-Tuning Strategies: When we fine-tune has a lower learning rate than the higher layer. BERT for a target task, there are many ways to When ξ = 1, all layers have the same learning utilize BERT. For example, the different layers rate, which is equivalent to the regular stochastic of BERT capture different levels of semantic and gradient descent (SGD). We will investigate these syntactic information, which layer is better for a

factors in Sec. 5.3. target task? How we choose a better optimization algorithm and learning rate? 4.2 Further Pre-training 2) Further Pre-training: BERT is trained in The BERT model is pre-trained in the general- the general domain, which has a different data dis- domain corpus. For a text classification task in a tribution from the target domain. A natural idea is specific domain, such as movie reviews, its data to further pre-train BERT with target domain data.

distribution may be different from BERT. There- 3) Multi-Task Fine-Tuning: Without pre- fore, we can further pre-train BERT with masked trained LM models, multi-task learning has shown language model and next sentence prediction tasks its effectiveness of exploiting the shared knowl- on the domain-specific data. Three further pre- edge among the multiple tasks. When there are training approaches are performed: several available tasks in a target domain, an inter-

1) Within-task pre-training, in which BERT is esting question is whether it still bring benefits to further pre-trained on the training data of a target fine-tune BERT on all the tasks simultaneously. task. Our general methodology of fine-tuning BERT 2) In-domain pre-training, in which the pre- is shown in Figure 1. training data is obtained from the same domain of a target task. For example, there are several dif- 4.1 Fine-Tuning Strategies ferent sentiment classification tasks, which have a

Different layers of a neural network can capture similar data distribution. We can further pre-train different levels of syntactic and semantic infor- BERT on the combined training data from these mation (Yosinski et al., 2014; Howard and Ruder, tasks. 2018). 3) Cross-domain pre-training, in which the pre- To adapt BERT to a target task, we need to con- training data is obtained from both the same and

sider several factors: 1) The first factor is the pre- other different domains to a target task. processing of long text since the maximum se- We will investigate these different approaches quence length of BERT is 512. 2) The second to further pre-training in Sec. 5.4. factor is layer selection. The official BERT-base model consists of an embedding layer, a 12-layer 4.3 Multi-Task Fine-Tuning encoder, and a pooling layer. We need to select the Multi-task Learning is also an effective approach

most effective layer for the text classification task. to share the knowledge obtained from several re- 3) The third factor is the overfitting problem. A lated supervised tasks. Similar to Liu et al. (2019), better optimizer with an appropriate learning rate we also use fine-tune BERT in multi-task learning is desired. framework for text classification. Intuitively, the lower layer of the BERT model All the tasks share the BERT layers and the em-

may contain more general information. We can bedding layer. The only layer that does not share fine-tune them with different learning rates. is the final classification layer, which means that Following Howard and Ruder (2018), we split each task has a private classifier layer. The exper- the parameters θ into {θ1 , · · · , θL } where θl con- imental analysis is in Sec. 5.5.

Average Max Exceeding Train Test Dataset Classes Type lengths lengths ratio samples samples IMDb 2 Sentiment 292 3,045 12.69% 25,000 25,000 Yelp P. 2 Sentiment 177 2,066 4.60% 560,000 38,000 Yelp F. 5 Sentiment 179 2,342 4.60% 650,000 50,000

TREC 6 Question 11 39 0.00% 5,452 500 Yahoo! Answers 10 Question 131 4,018 2.65% 1,400,000 60,000 AG’s News 4 Topic 44 221 0.00% 120,000 7,600 DBPedia 14 Topic 67 3,841 0.00% 560,000 70,000 Sogou News 6 Topic 737 47,988 46.23% 54,000 6,000

samples with a length exceeding 512.

5 Experiments fectiveness of BERT for Chinese text, we cre- ate the Chinese training and test datasets for So- We investigate the different fine-tuning methods gou news corpus. Unlike Zhang et al. (2015), for seven English and one Chinese text classifica- we use the Chinese character directly rather than tion tasks. We use the base BERT models: the un- Pinyin. The dataset is a combination of the So-

cased BERT-base model2 and the Chinese BERT- gouCA and SogouCS news corpora (Wang et al., base model3 respectively. 2008). We determine the category of the news 5.1 Datasets based on the URL, such as “sports” correspond- ing to “http://sports.sohu.com”. We choose 6 cate- We evaluate our approach on eight widely-studied gories – “sports”, “house”, “business”, “entertain- datasets. These datasets have varying numbers of

ment”, “women” and “technology”. The number documents and varying document lengths, cover- of training samples selected for each class is 9,000 ing three common text classification tasks: sen- and testing 1,000. timent analysis, question classification, and topic Data preprocessing Following Devlin et al. classification. We show the statistics for each (2018), we use WordPiece embeddings (Wu et al., dataset in Table 1. 2016) with a 30,000 token vocabulary and de-

Sentiment analysis For sentiment analysis, note split word pieces with ##. So the statis- we use the binary film review IMDb dataset (Maas tics of the length of the documents in the datasets et al., 2011) and the binary and five-class version are based on the word pieces. For further pre- of the Yelp review dataset built by Zhang et al. training with BERT, we use spaCy4 to perform (2015). sentence segmentation in English datasets and we

Question classification For question classifi- use “。”,“？” and “！” as separators when deal- cation, we evaluate our method on the six-class ing with the Chinese Sogou News dataset. version of the TREC dataset (Voorhees and Tice, 1999) and Yahoo! Answers dataset created by 5.2 Hyperparameters Zhang et al. (2015). TREC dataset is dataset for We use the BERT-base model (Devlin et al., 2018)

question classification consisting of open-domain, with a hidden size of 768, 12 Transformer blocks fact-based questions divided into broad seman- (Vaswani et al., 2017) and 12 self-attention heads. tic categories. Compared to other document-level We further pre-train with BERT on 1 TITAN Xp datasets, TREC dataset is sentence-level, and there GPU, with a batch size of 32, max squence length are fewer training examples for it. Yahoo! An- of 128, learning rate of 5e-5, train steps of 100,000

swers dataset is a big dataset with 1,400k train and warm-up steps of 10,000. samples. We fine-tune the BERT model on 4 TITAN Xp Topic classification For topic classification, we GPUs and set the batch size to 24 to ensure that the use large-scale AG’s News and DBPedia cre- GPU memory is fully utilized. The dropout prob- ated by Zhang et al. (2015). To test the ef- ability is always kept at 0.1. We use Adam with

2 https://storage.googleapis.com/bert models/2018 10 18/ β1 = 0.9 and β2 = 0.999. We use slanted triangu- uncased L-12 H-768 A-12.zip lar learning rates (Howard and Ruder, 2018), the 3 https://storage.googleapis.com/bert models/2018 11 03/ 4 chinese L-12 H-768 A-12.zip https://spacy.io/

base learning rate is 2e-5, and the warm-up pro- Method IMDb Sogou portion is 0.1. We empirically set the max number head-only 5.63 2.58 of the epoch to 4 and save the best model on the tail-only 5.44 3.17 head+tail 5.42 2.43 validation set for testing. hier. mean 5.89 2.83

hier. max 5.71 2.47 5.3 Exp-I: Investigating Different hier. self-attention 5.49 2.65 Fine-Tuning Strategies In this subsection, we use the IMDb dataset to in- Table 2: Test error rates (%) on IMDb and Chinese So- gou News datasets. vestigate the different fine-tuning strategies. The official pre-trained model is set as the initial en- coder5 . last layer of BERT gives the best performance.

5.3.1 Dealing with long texts Therefore, we use this setting for the following ex- The maximum sequence length of BERT is 512. periments. The first problem of applying BERT to text clas- sification is how processing the text with a length Layer Test error rates(%) larger than 512. We try the following ways for Layer-0 11.07 dealing with long articles. Layer-1 9.81

Layer-2 9.29 Truncation methods Usually, the key informa- Layer-3 8.66 tion of an article is at the beginning and end. We Layer-4 7.83 Layer-5 6.83 use three different methods of truncate text to per- Layer-6 6.83 form BERT fine-tuning. Layer-7 6.41

Layer-8 6.04 1. head-only: keep the first 510 tokens6 ; Layer-9 5.70 2. tail-only: keep the last 510 tokens; Layer-10 5.46 Layer-11 5.42 3. head+tail: empirically select the first 128 First 4 Layers + concat 8.69 and the last 382 tokens. First 4 Layers + mean 9.09

First 4 Layers + max 8.76 Hierarchical methods The input text is firstly divided into k = L/510 fractions, which is fed Last 4 Layers + concat 5.43 Last 4 Layers + mean 5.44 into BERT to obtain the representation of the k Last 4 Layers + max 5.42 text fractions. The representation of each fraction All 12 Layers + concat 5.44 is the hidden state of the [CLS] tokens of the last

layer. Then we use mean pooling, max pooling Table 3: Fine-tuning BERT with different layers on and self-attention to combine the representations IMDb dataset. of all the fractions. methods. The truncation method of head+tail achieves the best performance on IMDb and So- 5.3.3 Catastrophic Forgetting gou datasets. Therefore, we use this method to Catastrophic forgetting (McCloskey and Cohen, deal with the long text in the following experi- 1989) is usually a common problem in trans-

ments. fer learning, which means the pre-trained knowl- 5.3.2 Features from Different layers edge is erased during learning of new knowledge. Each layer of BERT captures the different features Therefore, we also investigate whether BERT suf- of the input text. We investigate the effectiveness fers from the catastrophic forgetting problem. of features from different layers. We then fine- We fine-tune BERT with different learning

Continued Discussion and Extended Analysis

How to Fine-Tune BERT for Text Classification?

Chi Sun, Xipeng Qiu∗, Yige Xu, Xuanjing Huang Shanghai Key Laboratory of Intelligent Information Processing, Fudan University School of Computer Science, Fudan University 825 Zhangheng Road, Shanghai, China {sunc17,xpqiu,ygxu18,xjhuang}@fudan.edu.cn

Abstract 2018). These word embeddings are often used as additional features for the main task. An- Language model pre-training has proven to be other kind of pre-training models is sentence- useful in learning universal language represen- level. Howard and Ruder (2018) propose ULM- arXiv:1905.05583v3 [cs.CL] 5 Feb 2020

tations. As a state-of-the-art language model pre-training model, BERT (Bidirectional En- FiT, a fine-tuning method for pre-trained language coder Representations from Transformers) has model that achieves state-of-the-art results on six achieved amazing results in many language widely studied text classification datasets. More understanding tasks. In this paper, we con- recently, pre-trained language models have shown

duct exhaustive experiments to investigate dif- to be useful in learning common language rep- ferent fine-tuning methods of BERT on text resentations by utilizing a large amount of unla- classification task and provide a general solu- tion for BERT fine-tuning. Finally, the pro- beled data: e.g., OpenAI GPT (Radford et al., posed solution obtains new state-of-the-art re- 2018) and BERT (Devlin et al., 2018). BERT is sults on eight widely-studied text classification based on a multi-layer bidirectional Transformer

datasets.1 (Vaswani et al., 2017) and is trained on plain text for masked word prediction and next sentence pre- 1 Introduction diction tasks. Text classification is a classic problem in Natural Although BERT has achieved amazing results Language Processing (NLP). The task is to assign in many natural language understanding (NLU) predefined categories to a given text sequence. An tasks, its potential has yet to be fully explored.

important intermediate step is the text representa- There is little research to enhance BERT to im- tion. Previous work uses various neural models prove the performance on target tasks further. to learn text representation, including convolution In this paper, we investigate how to maximize models (Kalchbrenner et al., 2014; Zhang et al., the utilization of BERT for the text classifica- 2015; Conneau et al., 2016; Johnson and Zhang, tion task. We explore several ways of fine-tuning

2017; Zhang et al., 2017; Shen et al., 2018), re- BERT to enhance its performance on text classifi- current models (Liu et al., 2016; Yogatama et al., cation task. We design exhaustive experiments to 2017; Seo et al., 2017), and attention mechanisms make a detailed analysis of BERT. (Yang et al., 2016; Lin et al., 2017). The contributions of our paper are as follows: Alternatively, substantial work has shown that

• We propose a general solution to fine-tune pre-trained models on large corpus are beneficial the pre-trained BERT model, which includes for text classification and other NLP tasks, which three steps: (1) further pre-train BERT on can avoid training a new model from scratch. One within-task training data or in-domain data; kind of pre-trained models is the word embed- (2) optional fine-tuning BERT with multi- dings, such as word2vec (Mikolov et al., 2013)

task learning if several related tasks are avail- and GloVe (Pennington et al., 2014), or the con- able; (3) fine-tune BERT for the target task. textualized word embeddings, such as CoVe (Mc- Cann et al., 2017) and ELMo (Peters et al., • We also investigate the fine-tuning meth- ∗ ods for BERT on target task, including pre- Corresponding author 1 The source codes are available at https://github. process of long text, layer selection, layer-

com/xuyige/BERT4doc-Classification. wise learning rate, catastrophic forgetting,

and low-shot learning problems. the-art results for a range of NLP tasks, demon- • We achieve the new state-of-the-art results on strating the enormous potential of the fine-tuning seven widely-studied English text classifica- method. In this paper, we have further explored tion datasets and one Chinese news classifi- the BERT fine-tuning method for text classifica- cation dataset. tion.

2 Related Work 2.2 Multi-task learning Multi-task learning (Caruana, 1993; Collobert and Borrowing the learned knowledge from the other Weston, 2008) is another relevant direction. Rei tasks has a rising interest in the field of NLP. We (2017) and Liu et al. (2018) use this method to briefly review two related approaches: language train the language model and the main task model model pre-training and multi-task Learning. jointly. Liu et al. (2019) extend the MT-DNN

2.1 Language Model Pre-training model originally proposed in Liu et al. (2015) by incorporating BERT as its shared text encoding Pre-trained word embeddings (Mikolov et al., layers. MTL requires training tasks from scratch 2013; Pennington et al., 2014), as an important every time, which makes it inefficient and it usu- component of modern NLP systems can offer sig- ally requires careful weighing of task-specific ob-

nificant improvements over embeddings learned jective functions (Chen et al., 2017). However, from scratch. The generalization of word embed- we can use multi-task BERT fine-tuning to avoid dings, such as sentence embeddings (Kiros et al., this problem by making full use of the shared pre- 2015; Logeswaran and Lee, 2018) or paragraph trained model. embeddings (Le and Mikolov, 2014), are also used as features in downstream models. 3 BERT for Text Classification

Peters et al. (2018) concatenate embeddings de- rived from language model as additional features BERT-base model contains an encoder with 12 for the main task and advance the state-of-the- Transformer blocks, 12 self-attention heads, and art for several major NLP benchmarks. In addi- the hidden size of 768. BERT takes an input of tion to pre-training with unsupervised data, trans- a sequence of no more than 512 tokens and out-

fer learning with a large amount of supervised data puts the representation of the sequence. The se- can also achieve good performance, such as natu- quence has one or two segments that the first token ral language inference (Conneau et al., 2017) and of the sequence is always [CLS] which contains machine translation (McCann et al., 2017). the special classification embedding and another More recently, the method of pre-training lan- special token [SEP] is used for separating seg-

guage models on a large network with a large ments. amount of unlabeled data and fine-tuning in down- For text classification tasks, BERT takes the fi- stream tasks has made a breakthrough in sev- nal hidden state h of the first token [CLS] as the eral natural language understanding tasks, such as representation of the whole sequence. A simple OpenAI GPT (Radford et al., 2018) and BERT softmax classifier is added to the top of BERT to

(Devlin et al., 2018). Dai and Le (2015) use lan- predict the probability of label c: guage model fine-tuning but overfit with 10k la- beled examples while Howard and Ruder (2018) p(c|h) = softmax(W h), (1) propose ULMFiT and achieve state-of-the-art re- where W is the task-specific parameter matrix. sults in the text classification task. BERT is We fine-tune all the parameters from BERT as well pre-trained on Masked Language Model Task and

as W jointly by maximizing the log-probability of Next Sentence Prediction Task via a large cross- the correct label. domain corpus. Unlike previous bidirectional lan- guage models (biLM) limited to a combination of 4 Methodology two unidirectional language models (i.e., left-to- right and right-to-left), BERT uses a Masked Lan- When we adapt BERT to NLP tasks in a target guage Model to predict words which are randomly domain, a proper fine-tuning strategy is desired.

masked or replaced. BERT is the first fine-tuning In this paper, we look for the proper fine-tuning based representation model that achieves state-of- methods in the following three ways.

Single-Task tains the parameters of the l-th layer of BERT. Fine-Tuning Then the parameters are updated as follows: Further BERT Pre-training Multi-Task θtl = θt−1 l − η l · ∇θl J(θ), (2) Fine-Tuning where η l represents the learning rate of the l-th shown with different colors. We set the base learning rate to η L and use η k−1 = ξ · η k , where ξ is a decay factor and less

than or equal to 1. When ξ < 1, the lower layer 1) Fine-Tuning Strategies: When we fine-tune has a lower learning rate than the higher layer. BERT for a target task, there are many ways to When ξ = 1, all layers have the same learning utilize BERT. For example, the different layers rate, which is equivalent to the regular stochastic of BERT capture different levels of semantic and gradient descent (SGD). We will investigate these syntactic information, which layer is better for a

factors in Sec. 5.3. target task? How we choose a better optimization algorithm and learning rate? 4.2 Further Pre-training 2) Further Pre-training: BERT is trained in The BERT model is pre-trained in the general- the general domain, which has a different data dis- domain corpus. For a text classification task in a tribution from the target domain. A natural idea is specific domain, such as movie reviews, its data to further pre-train BERT with target domain data.

distribution may be different from BERT. There- 3) Multi-Task Fine-Tuning: Without pre- fore, we can further pre-train BERT with masked trained LM models, multi-task learning has shown language model and next sentence prediction tasks its effectiveness of exploiting the shared knowl- on the domain-specific data. Three further pre- edge among the multiple tasks. When there are training approaches are performed: several available tasks in a target domain, an inter-

1) Within-task pre-training, in which BERT is esting question is whether it still bring benefits to further pre-trained on the training data of a target fine-tune BERT on all the tasks simultaneously. task. Our general methodology of fine-tuning BERT 2) In-domain pre-training, in which the pre- is shown in Figure 1. training data is obtained from the same domain of a target task. For example, there are several dif- 4.1 Fine-Tuning Strategies ferent sentiment classification tasks, which have a

Different layers of a neural network can capture similar data distribution. We can further pre-train different levels of syntactic and semantic infor- BERT on the combined training data from these mation (Yosinski et al., 2014; Howard and Ruder, tasks. 2018). 3) Cross-domain pre-training, in which the pre- To adapt BERT to a target task, we need to con- training data is obtained from both the same and

sider several factors: 1) The first factor is the pre- other different domains to a target task. processing of long text since the maximum se- We will investigate these different approaches quence length of BERT is 512. 2) The second to further pre-training in Sec. 5.4. factor is layer selection. The official BERT-base model consists of an embedding layer, a 12-layer 4.3 Multi-Task Fine-Tuning encoder, and a pooling layer. We need to select the Multi-task Learning is also an effective approach

most effective layer for the text classification task. to share the knowledge obtained from several re- 3) The third factor is the overfitting problem. A lated supervised tasks. Similar to Liu et al. (2019), better optimizer with an appropriate learning rate we also use fine-tune BERT in multi-task learning is desired. framework for text classification. Intuitively, the lower layer of the BERT model All the tasks share the BERT layers and the em-

may contain more general information. We can bedding layer. The only layer that does not share fine-tune them with different learning rates. is the final classification layer, which means that Following Howard and Ruder (2018), we split each task has a private classifier layer. The exper- the parameters θ into {θ1 , · · · , θL } where θl con- imental analysis is in Sec. 5.5.

Average Max Exceeding Train Test Dataset Classes Type lengths lengths ratio samples samples IMDb 2 Sentiment 292 3,045 12.69% 25,000 25,000 Yelp P. 2 Sentiment 177 2,066 4.60% 560,000 38,000 Yelp F. 5 Sentiment 179 2,342 4.60% 650,000 50,000

TREC 6 Question 11 39 0.00% 5,452 500 Yahoo! Answers 10 Question 131 4,018 2.65% 1,400,000 60,000 AG’s News 4 Topic 44 221 0.00% 120,000 7,600 DBPedia 14 Topic 67 3,841 0.00% 560,000 70,000 Sogou News 6 Topic 737 47,988 46.23% 54,000 6,000

samples with a length exceeding 512.

5 Experiments fectiveness of BERT for Chinese text, we cre- ate the Chinese training and test datasets for So- We investigate the different fine-tuning methods gou news corpus. Unlike Zhang et al. (2015), for seven English and one Chinese text classifica- we use the Chinese character directly rather than tion tasks. We use the base BERT models: the un- Pinyin. The dataset is a combination of the So-

cased BERT-base model2 and the Chinese BERT- gouCA and SogouCS news corpora (Wang et al., base model3 respectively. 2008). We determine the category of the news 5.1 Datasets based on the URL, such as “sports” correspond- ing to “http://sports.sohu.com”. We choose 6 cate- We evaluate our approach on eight widely-studied gories – “sports”, “house”, “business”, “entertain- datasets. These datasets have varying numbers of

ment”, “women” and “technology”. The number documents and varying document lengths, cover- of training samples selected for each class is 9,000 ing three common text classification tasks: sen- and testing 1,000. timent analysis, question classification, and topic Data preprocessing Following Devlin et al. classification. We show the statistics for each (2018), we use WordPiece embeddings (Wu et al., dataset in Table 1. 2016) with a 30,000 token vocabulary and de-

Sentiment analysis For sentiment analysis, note split word pieces with ##. So the statis- we use the binary film review IMDb dataset (Maas tics of the length of the documents in the datasets et al., 2011) and the binary and five-class version are based on the word pieces. For further pre- of the Yelp review dataset built by Zhang et al. training with BERT, we use spaCy4 to perform (2015). sentence segmentation in English datasets and we

Question classification For question classifi- use “。”,“？” and “！” as separators when deal- cation, we evaluate our method on the six-class ing with the Chinese Sogou News dataset. version of the TREC dataset (Voorhees and Tice, 1999) and Yahoo! Answers dataset created by 5.2 Hyperparameters Zhang et al. (2015). TREC dataset is dataset for We use the BERT-base model (Devlin et al., 2018)

question classification consisting of open-domain, with a hidden size of 768, 12 Transformer blocks fact-based questions divided into broad seman- (Vaswani et al., 2017) and 12 self-attention heads. tic categories. Compared to other document-level We further pre-train with BERT on 1 TITAN Xp datasets, TREC dataset is sentence-level, and there GPU, with a batch size of 32, max squence length are fewer training examples for it. Yahoo! An- of 128, learning rate of 5e-5, train steps of 100,000

swers dataset is a big dataset with 1,400k train and warm-up steps of 10,000. samples. We fine-tune the BERT model on 4 TITAN Xp Topic classification For topic classification, we GPUs and set the batch size to 24 to ensure that the use large-scale AG’s News and DBPedia cre- GPU memory is fully utilized. The dropout prob- ated by Zhang et al. (2015). To test the ef- ability is always kept at 0.1. We use Adam with

2 https://storage.googleapis.com/bert models/2018 10 18/ β1 = 0.9 and β2 = 0.999. We use slanted triangu- uncased L-12 H-768 A-12.zip lar learning rates (Howard and Ruder, 2018), the 3 https://storage.googleapis.com/bert models/2018 11 03/ 4 chinese L-12 H-768 A-12.zip https://spacy.io/

base learning rate is 2e-5, and the warm-up pro- Method IMDb Sogou portion is 0.1. We empirically set the max number head-only 5.63 2.58 of the epoch to 4 and save the best model on the tail-only 5.44 3.17 head+tail 5.42 2.43 validation set for testing. hier. mean 5.89 2.83

hier. max 5.71 2.47 5.3 Exp-I: Investigating Different hier. self-attention 5.49 2.65 Fine-Tuning Strategies In this subsection, we use the IMDb dataset to in- Table 2: Test error rates (%) on IMDb and Chinese So- gou News datasets. vestigate the different fine-tuning strategies. The official pre-trained model is set as the initial en- coder5 . last layer of BERT gives the best performance.

5.3.1 Dealing with long texts Therefore, we use this setting for the following ex- The maximum sequence length of BERT is 512. periments. The first problem of applying BERT to text clas- sification is how processing the text with a length Layer Test error rates(%) larger than 512. We try the following ways for Layer-0 11.07 dealing with long articles. Layer-1 9.81

Layer-2 9.29 Truncation methods Usually, the key informa- Layer-3 8.66 tion of an article is at the beginning and end. We Layer-4 7.83 Layer-5 6.83 use three different methods of truncate text to per- Layer-6 6.83 form BERT fine-tuning. Layer-7 6.41

Layer-8 6.04 1. head-only: keep the first 510 tokens6 ; Layer-9 5.70 2. tail-only: keep the last 510 tokens; Layer-10 5.46 Layer-11 5.42 3. head+tail: empirically select the first 128 First 4 Layers + concat 8.69 and the last 382 tokens. First 4 Layers + mean 9.09

First 4 Layers + max 8.76 Hierarchical methods The input text is firstly divided into k = L/510 fractions, which is fed Last 4 Layers + concat 5.43 Last 4 Layers + mean 5.44 into BERT to obtain the representation of the k Last 4 Layers + max 5.42 text fractions. The representation of each fraction All 12 Layers + concat 5.44 is the hidden state of the [CLS] tokens of the last

layer. Then we use mean pooling, max pooling Table 3: Fine-tuning BERT with different layers on and self-attention to combine the representations IMDb dataset. of all the fractions. methods. The truncation method of head+tail achieves the best performance on IMDb and So- 5.3.3 Catastrophic Forgetting gou datasets. Therefore, we use this method to Catastrophic forgetting (McCloskey and Cohen, deal with the long text in the following experi- 1989) is usually a common problem in trans-

ments. fer learning, which means the pre-trained knowl- 5.3.2 Features from Different layers edge is erased during learning of new knowledge. Each layer of BERT captures the different features Therefore, we also investigate whether BERT suf- of the input text. We investigate the effectiveness fers from the catastrophic forgetting problem. of features from different layers. We then fine- We fine-tune BERT with different learning