Ai Waste Management

This project focuses on ai waste management using modern AI and machine learning techniques. The content below is adapted from research literature and practical implementation notes.

How to Fine-Tune BERT for Text Classification?

Chi Sun, Xipeng Qiu∗, Yige Xu, Xuanjing Huang

825 Zhangheng Road, Shanghai, China

Abstract 2018). These word embeddings are often used

as additional features for the main task. An-

Language model pre-training has proven to be

other kind of pre-training models is sentence- useful in learning universal language represen- level. Howard and Ruder (2018) propose ULM-

tations. As a state-of-the-art language model pre-training model, BERT (Bidirectional En- FiT, a fine-tuning method for pre-trained language coder Representations from Transformers) has model that achieves state-of-the-art results on six achieved amazing results in many language widely studied text classification datasets. More understanding tasks. In this paper, we con- recently, pre-trained language models have shown duct exhaustive experiments to investigate dif- to be useful in learning common language rep-

ferent fine-tuning methods of BERT on text resentations by utilizing a large amount of unla- classification task and provide a general solu- tion for BERT fine-tuning. Finally, the pro- sults on eight widely-studied text classification based on a multi-layer bidirectional Transformer for masked word prediction and next sentence pre-

1 Introduction diction tasks.

Text classification is a classic problem in Natural Although BERT has achieved amazing results

Language Processing (NLP). The task is to assign in many natural language understanding (NLU)

predefined categories to a given text sequence. An tasks, its potential has yet to be fully explored. important intermediate step is the text representa- There is little research to enhance BERT to im- tion. Previous work uses various neural models prove the performance on target tasks further. to learn text representation, including convolution In this paper, we investigate how to maximize

Alternatively, substantial work has shown that

• We propose a general solution to fine-tune pre-trained models on large corpus are beneficial the pre-trained BERT model, which includes for text classification and other NLP tasks, which three steps: (1) further pre-train BERT on can avoid training a new model from scratch. One within-task training data or in-domain data; kind of pre-trained models is the word embed- (2) optional fine-tuning BERT with multi- task learning if several related tasks are avail- able; (3) fine-tune BERT for the target task.

textualized word embeddings, such as CoVe (Mc- ∗ ods for BERT on target task, including pre-

Corresponding author

The source codes are available at https://github. process of long text, layer selection, layer-

com/xuyige/BERT4doc-Classification. wise learning rate, catastrophic forgetting,

and low-shot learning problems. the-art results for a range of NLP tasks, demon- • We achieve the new state-of-the-art results on strating the enormous potential of the fine-tuning seven widely-studied English text classifica- method. In this paper, we have further explored tion datasets and one Chinese news classifi- the BERT fine-tuning method for text classifica- cation dataset. tion.

2 Related Work 2.2 Multi-task learning

Multi-task learning (Caruana, 1993; Collobert and

Borrowing the learned knowledge from the other Weston, 2008) is another relevant direction. Rei

briefly review two related approaches: language train the language model and the main task model

incorporating BERT as its shared text encoding component of modern NLP systems can offer sig- ally requires careful weighing of task-specific ob- from scratch. The generalization of word embed- we can use multi-task BERT fine-tuning to avoid 2015; Logeswaran and Lee, 2018) or paragraph trained model. embeddings (Le and Mikolov, 2014), are also used as features in downstream models. 3 BERT for Text Classification rived from language model as additional features BERT-base model contains an encoder with 12

for the main task and advance the state-of-the- Transformer blocks, 12 self-attention heads, and art for several major NLP benchmarks. In addi- the hidden size of 768. BERT takes an input of tion to pre-training with unsupervised data, trans- a sequence of no more than 512 tokens and out- fer learning with a large amount of supervised data puts the representation of the sequence. The se- can also achieve good performance, such as natu- quence has one or two segments that the first token

More recently, the method of pre-training lan- special token [SEP] is used for separating seg- guage models on a large network with a large ments. amount of unlabeled data and fine-tuning in down- For text classification tasks, BERT takes the fi- stream tasks has made a breakthrough in sev- nal hidden state h of the first token [CLS] as the eral natural language understanding tasks, such as representation of the whole sequence. A simple guage model fine-tuning but overfit with 10k la-

beled examples while Howard and Ruder (2018) p(c|h) = softmax(W h), (1) propose ULMFiT and achieve state-of-the-art re- where W is the task-specific parameter matrix. sults in the text classification task. BERT is

We fine-tune all the parameters from BERT as well

pre-trained on Masked Language Model Task and as W jointly by maximizing the log-probability of

Next Sentence Prediction Task via a large cross-

the correct label. domain corpus. Unlike previous bidirectional lan- guage models (biLM) limited to a combination of 4 Methodology two unidirectional language models (i.e., left-to- right and right-to-left), BERT uses a Masked Lan- When we adapt BERT to NLP tasks in a target guage Model to predict words which are randomly domain, a proper fine-tuning strategy is desired. masked or replaced. BERT is the first fine-tuning In this paper, we look for the proper fine-tuning

based representation model that achieves state-of- methods in the following three ways.

Single-Task tains the parameters of the l-th layer of BERT. Fine-Tuning Then the parameters are updated as follows: Further

BERT Pre-training

Multi-Task θtl = θt−1

l − η l · ∇θl J(θ), (2) Fine-Tuning where η l represents the learning rate of the l-th shown with different colors. We set the base learning rate to η L and use η k−1 = ξ · η k , where ξ is a decay factor and less than or equal to 1. When ξ < 1, the lower layer 1) Fine-Tuning Strategies: When we fine-tune has a lower learning rate than the higher layer.

BERT for a target task, there are many ways to

When ξ = 1, all layers have the same learning

utilize BERT. For example, the different layers rate, which is equivalent to the regular stochastic of BERT capture different levels of semantic and gradient descent (SGD). We will investigate these syntactic information, which layer is better for a factors in Sec. 5.3. target task? How we choose a better optimization algorithm and learning rate? 4.2 Further Pre-training 2) Further Pre-training: BERT is trained in

The BERT model is pre-trained in the general-

the general domain, which has a different data dis- domain corpus. For a text classification task in a tribution from the target domain. A natural idea is specific domain, such as movie reviews, its data to further pre-train BERT with target domain data. distribution may be different from BERT. There- 3) Multi-Task Fine-Tuning: Without pre- fore, we can further pre-train BERT with masked trained LM models, multi-task learning has shown language model and next sentence prediction tasks

its effectiveness of exploiting the shared knowl- on the domain-specific data. Three further pre- edge among the multiple tasks. When there are training approaches are performed: several available tasks in a target domain, an inter- 1) Within-task pre-training, in which BERT is esting question is whether it still bring benefits to further pre-trained on the training data of a target fine-tune BERT on all the tasks simultaneously. task.

Our general methodology of fine-tuning BERT

2) In-domain pre-training, in which the pre- is shown in Figure 1. training data is obtained from the same domain of a target task. For example, there are several dif-

4.1 Fine-Tuning Strategies

ferent sentiment classification tasks, which have a Different layers of a neural network can capture similar data distribution. We can further pre-train different levels of syntactic and semantic infor- BERT on the combined training data from these 2018). 3) Cross-domain pre-training, in which the pre- To adapt BERT to a target task, we need to con- training data is obtained from both the same and sider several factors: 1) The first factor is the pre- other different domains to a target task.

processing of long text since the maximum se- We will investigate these different approaches quence length of BERT is 512. 2) The second to further pre-training in Sec. 5.4. factor is layer selection. The official BERT-base model consists of an embedding layer, a 12-layer 4.3 Multi-Task Fine-Tuning encoder, and a pooling layer. We need to select the Multi-task Learning is also an effective approach most effective layer for the text classification task. to share the knowledge obtained from several re-

better optimizer with an appropriate learning rate we also use fine-tune BERT in multi-task learning is desired. framework for text classification. Intuitively, the lower layer of the BERT model All the tasks share the BERT layers and the em- may contain more general information. We can bedding layer. The only layer that does not share fine-tune them with different learning rates. is the final classification layer, which means that

Following Howard and Ruder (2018), we split each task has a private classifier layer. The exper- the parameters θ into {θ1 , · · · , θL } where θl con- imental analysis is in Sec. 5.5.

Average Max Exceeding Train Test

Dataset Classes Type

lengths lengths ratio samples samples

IMDb 2 Sentiment 292 3,045 12.69% 25,000 25,000

Yelp P. 2 Sentiment 177 2,066 4.60% 560,000 38,000

Yelp F. 5 Sentiment 179 2,342 4.60% 650,000 50,000

TREC 6 Question 11 39 0.00% 5,452 500

Yahoo! Answers 10 Question 131 4,018 2.65% 1,400,000 60,000

AG’s News 4 Topic 44 221 0.00% 120,000 7,600

DBPedia 14 Topic 67 3,841 0.00% 560,000 70,000

Sogou News 6 Topic 737 47,988 46.23% 54,000 6,000

samples with a length exceeding 512.

5 Experiments fectiveness of BERT for Chinese text, we cre-

ate the Chinese training and test datasets for So-

We investigate the different fine-tuning methods

for seven English and one Chinese text classifica- we use the Chinese character directly rather than tion tasks. We use the base BERT models: the un-

Pinyin. The dataset is a combination of the So-

cased BERT-base model2 and the Chinese BERT- base model3 respectively. 2008). We determine the category of the news

5.1 Datasets based on the URL, such as “sports” correspond-

ing to “http://sports.sohu.com”. We choose 6 cate-

We evaluate our approach on eight widely-studied

gories – “sports”, “house”, “business”, “entertain- datasets. These datasets have varying numbers of ment”, “women” and “technology”. The number documents and varying document lengths, cover- of training samples selected for each class is 9,000 ing three common text classification tasks: sen- and testing 1,000. timent analysis, question classification, and topic classification. We show the statistics for each dataset in Table 1. 2016) with a 30,000 token vocabulary and de-

Sentiment analysis For sentiment analysis, note split word pieces with ##. So the statis-

we use the binary film review IMDb dataset (Maas tics of the length of the documents in the datasets (2015). sentence segmentation in English datasets and we

Question classification For question classifi- use “。”,“？” and “！” as separators when deal-

cation, we evaluate our method on the six-class ing with the Chinese Sogou News dataset. version of the TREC dataset (Voorhees and Tice, 1999) and Yahoo! Answers dataset created by 5.2 Hyperparameters question classification consisting of open-domain, with a hidden size of 768, 12 Transformer blocks tic categories. Compared to other document-level We further pre-train with BERT on 1 TITAN Xp datasets, TREC dataset is sentence-level, and there GPU, with a batch size of 32, max squence length

are fewer training examples for it. Yahoo! An- of 128, learning rate of 5e-5, train steps of 100,000 swers dataset is a big dataset with 1,400k train and warm-up steps of 10,000. samples. We fine-tune the BERT model on 4 TITAN Xp Topic classification For topic classification, we GPUs and set the batch size to 24 to ensure that the use large-scale AG’s News and DBPedia cre- GPU memory is fully utilized. The dropout prob-

https://storage.googleapis.com/bert models/2018 10 18/ β1 = 0.9 and β2 = 0.999. We use slanted triangu- uncased L-12 H-768 A-12.zip lar learning rates (Howard and Ruder, 2018), the https://storage.googleapis.com/bert models/2018 11 03/ chinese L-12 H-768 A-12.zip https://spacy.io/

base learning rate is 2e-5, and the warm-up pro- Method IMDb Sogou portion is 0.1. We empirically set the max number head-only 5.63 2.58 of the epoch to 4 and save the best model on the tail-only 5.44 3.17 head+tail 5.42 2.43 validation set for testing. hier. mean 5.89 2.83 hier. max 5.71 2.47

5.3 Exp-I: Investigating Different hier. self-attention 5.49 2.65

Fine-Tuning Strategies

In this subsection, we use the IMDb dataset to in- Table 2: Test error rates (%) on IMDb and Chinese So- gou News datasets. vestigate the different fine-tuning strategies. The official pre-trained model is set as the initial en- coder5 . last layer of BERT gives the best performance.

5.3.1 Dealing with long texts Therefore, we use this setting for the following ex-

The maximum sequence length of BERT is 512. periments.

The first problem of applying BERT to text clas-

sification is how processing the text with a length Layer Test error rates(%) larger than 512. We try the following ways for

Layer-0 11.07

dealing with long articles. Layer-1 9.81

Layer-2 9.29

Truncation methods Usually, the key informa- Layer-3 8.66

tion of an article is at the beginning and end. We Layer-4 7.83

Layer-5 6.83

use three different methods of truncate text to per- Layer-6 6.83 form BERT fine-tuning. Layer-7 6.41

Layer-8 6.04

1. head-only: keep the first 510 tokens6 ; Layer-9 5.70

2. tail-only: keep the last 510 tokens; Layer-10 5.46

Layer-11 5.42

3. head+tail: empirically select the first 128

First 4 Layers + concat 8.69

and the last 382 tokens. First 4 Layers + mean 9.09

First 4 Layers + max 8.76

Hierarchical methods The input text is firstly

divided into k = L/510 fractions, which is fed Last 4 Layers + concat 5.43

Last 4 Layers + mean 5.44

into BERT to obtain the representation of the k Last 4 Layers + max 5.42 text fractions. The representation of each fraction

All 12 Layers + concat 5.44

is the hidden state of the [CLS] tokens of the last layer. Then we use mean pooling, max pooling Table 3: Fine-tuning BERT with different layers on and self-attention to combine the representations IMDb dataset. of all the fractions. methods. The truncation method of head+tail achieves the best performance on IMDb and So- 5.3.3 Catastrophic Forgetting gou datasets. Therefore, we use this method to Catastrophic forgetting (McCloskey and Cohen,

deal with the long text in the following experi- 1989) is usually a common problem in trans- ments. fer learning, which means the pre-trained knowl-

5.3.2 Features from Different layers edge is erased during learning of new knowledge.

Each layer of BERT captures the different features Therefore, we also investigate whether BERT suf- of the input text. We investigate the effectiveness fers from the catastrophic forgetting problem. of features from different layers. We then fine- We fine-tune BERT with different learning tune the model and record the performance on test rates, and the learning curves of error rates on error rates. IMDb are shown in Figure 2.

BERT with different layers. The feature from the is necessary to make BERT overcome the catas- https://github.com/google-research/bert trophic forgetting problem. With an aggressive

512 to subtract the [CLS] and [SEP] tokens. learn rate of 4e-4, the training set fails to converge.

(a) lr=2e-5 (b) lr=5e-5 (c) lr=1e-4 (d) lr=4e-4

5.3.4 Layer-wise Decreasing Layer Rate BERT-ITPT-FiT

Test error rate(%)

learning rate and decay factors (see Eq. (2)) on

IMDb dataset. We find that assign a lower learn- 5

ing rate to the lower layer is effective to fine-tuning

BERT, and an appropriate setting is ξ=0.95 and

lr=2.0e-5. 4.5

Learning rate Decay factor ξ Test error rates(%) 0 100 200 300 400 500

2.5e-5 1.00 5.52 Within-Task Pre-training Steps (thousand) 2.5e-5 0.95 5.46 2.5e-5 0.90 5.44 Figure 3: Benefit of different further pre-training steps 2.5e-5 0.85 5.58 on IMDb datasets. BERT-ITPT-FiT means “BERT + 2.0e-5 1.00 5.42 withIn-Task Pre-Training + Fine-Tuning”. 2.0e-5 0.95 5.40

2.0e-5 0.90 5.52 2.0e-5 0.85 5.65

5.4.2 In-Domain and Cross-Domain Further

Besides the training data of a target task, we

can further pre-train BERT on the data from the

5.4 Exp-II: Investigating the Further same domain. In this subsection, we investigate

Pretraining whether further pre-training BERT w

Frequently Asked Questions

What is this project about?

This project covers practical implementation and research aspects of the topic using AI/ML techniques.

This section provides additional detailed analysis and supporting information derived from the research paper content to ensure comprehensive coverage of the topic with expanded discussion on key concepts, methods, and findings.

Speaker Verification Xvector