Ai Learning Path Recommendation

This project focuses on ai learning path recommendation using modern AI and machine learning techniques. The content below is adapted from research literature and practical implementation notes.

BERT: Pre-training of Deep Bidirectional Transformers for

Language Understanding

Jacob Devlin

Ming-Wei Chang Kenton Lee Kristina Toutanova

Google AI Language

Abstract There are two existing strategies for apply-

ing pre-trained language representations to down-

We introduce a new language representa-

stream tasks: feature-based and fine-tuning. The

tion model called BERT, which stands for

Bidirectional Encoder Representations from feature-based approach, such as ELMo (Peters

train deep bidirectional representations from the Generative Pre-trained Transformer (OpenAI left and right context in all layers. As a re- task-specific parameters, and is trained on the sult, the pre-trained BERT model can be fine- tuned with just one additional output layer downstream tasks by simply fine-tuning all pre- to create state-of-the-art models for a wide trained parameters. The two approaches share the range of tasks, such as question answering and same objective function during pre-training, where

language inference, without substantial task- they use unidirectional language models to learn specific architecture modifications. general language representations.

BERT is conceptually simple and empirically We argue that current techniques restrict the

powerful. It obtains new state-of-the-art re- power of the pre-trained representations, espe- sults on eleven natural language processing cially for the fine-tuning approaches. The ma- tasks, including pushing the GLUE score to jor limitation is that standard language models are 80.5% (7.7% point absolute improvement),

MultiNLI accuracy to 86.7% (4.6% absolute

unidirectional, and this limits the choice of archi- improvement), SQuAD v1.1 question answer- tectures that can be used during pre-training. For ing Test F1 to 93.2 (1.5 point absolute im- example, in OpenAI GPT, the authors use a left-to- provement) and SQuAD v2.0 Test F1 to 83.1 right architecture, where every token can only at- (5.1 point absolute improvement). tend to previous tokens in the self-attention layers

1 Introduction

strictions are sub-optimal for sentence-level tasks, Language model pre-training has been shown to and could be very harmful when applying fine- be effective for improving many natural language tuning based approaches to token-level tasks such 2018). These include sentence-level tasks such as In this paper, we improve the fine-tuning based and Brockett, 2005), which aim to predict the re- BERT alleviates the previously mentioned unidi- lationships between sentences by analyzing them rectionality constraint by using a “masked lan-

holistically, as well as token-level tasks such as guage model” (MLM) pre-training objective, in- named entity recognition and question answering, spired by the Cloze task (Taylor, 1953). The where models are required to produce fine-grained masked language model randomly masks some of output at the token level (Tjong Kim Sang and the tokens from the input, and the objective is to

word based only on its context. Unlike left-to- These approaches have been generalized to right language model pre-training, the MLM ob- coarser granularities, such as sentence embed- and the right context, which allows us to pre- 2018) or paragraph embeddings (Le and Mikolov, train a deep bidirectional Transformer. In addi- 2014). To train sentence representations, prior tion to the masked language model, we also use work has used objectives to rank candidate next

trains text-pair representations. The contributions Lee, 2018), left-to-right generation of next sen- of our paper are as follows: tence words given a representation of the previous • We demonstrate the importance of bidirectional pre-training for language representations. Un- tional language models for pre-training, BERT 2018a) generalize traditional word embedding re- uses masked language models to enable pre- search along a different dimension. They extract

trained deep bidirectional representations. This context-sensitive features from a left-to-right and a uses a shallow concatenation of independently resentation of each token is the concatenation of trained left-to-right and right-to-left LMs. the left-to-right and right-to-left representations.

When integrating contextual word embeddings

• We show that pre-trained representations reduce with existing task-specific architectures, ELMo the need for many heavily-engineered task- advances the state of the art for several major NLP of sentence-level and token-level tasks, outper- recognition (Tjong Kim Sang and De Meulder, contextual representations through a task to pre- • BERT advances the state of the art for eleven dict a single word from both left and right context

NLP tasks. The code and pre-trained mod-

using LSTMs. Similar to ELMo, their model is els are available at https://github.com/ feature-based and not deeply bidirectional. Fedus google-research/bert.

2 Related Work to improve the robustness of text generation mod-

There is a long history of pre-training general lan-

guage representations, and we briefly review the 2.2 Unsupervised Fine-tuning Approaches most widely-used approaches in this section.

As with the feature-based approaches, the first

2.1 Unsupervised Feature-based Approaches works in this direction only pre-trained word em-

Learning widely applicable representations of bedding parameters from unlabeled text (Col-

words has been an active area of research for lobert and Weston, 2008). 2014) methods. Pre-trained word embeddings fine-tuned for a supervised downstream task (Dai are an integral part of modern NLP systems, of- and Le, 2015; Howard and Ruder, 2018; Radford train word embedding vectors, left-to-right lan- scratch. At least partly due to this advantage, and Hinton, 2009), as well as objectives to dis- viously state-of-the-art results on many sentence-

criminate correct from incorrect words in left and level tasks from the GLUE benchmark (Wang

NSP Mask LM Mask LM MNLI NER SQuAD

Start/End Span

C T1 ... TN T[SEP] T1’ ... TM’ C T1 ... TN T[SEP] T1’ ... TM’

BERT BERT BERT

E[CLS] E1 ... EN E[SEP] E1’ ... EM’ E[CLS] E1 ... EN E[SEP] E1’ ... EM’

[CLS] Tok 1 ... Tok N [SEP] Tok 1 ... TokM [CLS] Tok 1 ... Tok N [SEP] Tok 1 ... TokM

Masked Sentence A Masked Sentence B Question Paragraph

Unlabeled Sentence A and B Pair Question Answer Pair

Pre-training Fine-Tuning

tures are used in both pre-training and fine-tuning. The same pre-trained model parameters are used to initialize models for different down-stream tasks. During fine-tuning, all parameters are fine-tuned. [CLS] is a special symbol added in front of every input example, and [SEP] is a special separator token (e.g. separating ques- tions/answers).

ing and auto-encoder objectives have been used mal difference between the pre-trained architec- for pre-training such models (Howard and Ruder, ture and the final downstream architecture.

Model Architecture BERT’s model architec-

2.3 Transfer Learning from Supervised Data ture is a multi-layer bidirectional Transformer en-

There has also been work showing effective trans- coder based on the original implementation de- 2017). Computer vision research has also demon- plementation is almost identical to the original, strated the importance of transfer learning from we will omit an exhaustive background descrip- large pre-trained models, where an effective recipe tion of the model architecture and refer readers to

In this work, we denote the number of layers

3 BERT (i.e., Transformer blocks) as L, the hidden size as

H, and the number of self-attention heads as A.3

We introduce BERT and its detailed implementa- We primarily report results on two model sizes: tion in this section. There are two steps in our BERTBASE (L=12, H=768, A=12, Total Param- framework: pre-training and fine-tuning. Dur- eters=110M) and BERTLARGE (L=24, H=1024, ing pre-training, the model is trained on unlabeled A=16, Total Parameters=340M). data over different pre-training tasks. For fine- BERTBASE was chosen to have the same model

tuning, the BERT model is first initialized with size as OpenAI GPT for comparison purposes. the pre-trained parameters, and all of the param- Critically, however, the BERT Transformer uses eters are fine-tuned using labeled data from the bidirectional self-attention, while the GPT Trans- downstream tasks. Each downstream task has sep- former uses constrained self-attention where every

arate fine-tuned models, even though they are ini- token can only attend to context to its left.4 tialized with the same pre-trained parameters. The question-answering example in Figure 1 will serve https://github.com/tensorflow/tensor2tensor as a running example for this section. http://nlp.seas.harvard.edu/2018/04/03/attention.html In all cases we set the feed-forward/filter size to be 4H,

A distinctive feature of BERT is its unified ar- i.e., 3072 for the H = 768 and 4096 for the H = 1024. chitecture across different tasks. There is mini- We note that in the literature the bidirectional Trans-

Input/Output Representations To make BERT In order to train a deep bidirectional representa-

handle a variety of down-stream tasks, our input tion, we simply mask some percentage of the input representation is able to unambiguously represent tokens at random, and then predict those masked both a single sentence and a pair of sentences tokens. We refer to this procedure as a “masked (e.g., h Question, Answer i) in one token sequence. LM” (MLM), although it is often referred to as a Throughout this work, a “sentence” can be an arbi- Cloze task in the literature (Taylor, 1953). In this

trary span of contiguous text, rather than an actual case, the final hidden vectors corresponding to the linguistic sentence. A “sequence” refers to the in- mask tokens are fed into an output softmax over put token sequence to BERT, which may be a sin- the vocabulary, as in a standard LM. In all of our gle sentence or two sentences packed together. experiments, we mask 15% of all WordPiece to- token of every sequence is always a special clas- only predict the masked words rather than recon-

sification token ([CLS]). The final hidden state structing the entire input. corresponding to this token is used as the ag- Although this allows us to obtain a bidirec- gregate sequence representation for classification tional pre-trained model, a downside is that we tasks. Sentence pairs are packed together into a are creating a mismatch between pre-training and single sequence. We differentiate the sentences in fine-tuning, since the [MASK] token does not ap-

two ways. First, we separate them with a special pear during fine-tuning. To mitigate this, we do token ([SEP]). Second, we add a learned embed- not always replace “masked” words with the ac- ding to every token indicating whether it belongs tual [MASK] token. The training data generator to sentence A or sentence B. As shown in Figure 1, chooses 15% of the token positions at random for we denote input embedding as E, the final hidden prediction. If the i-th token is chosen, we replace

vector of the special [CLS] token as C ∈ RH , the i-th token with (1) the [MASK] token 80% of and the final hidden vector for the ith input token the time (2) a random token 10% of the time (3) as Ti ∈ RH . the unchanged i-th token 10% of the time. Then, For a given token, its input representation is Ti will be used to predict the original token with constructed by summing the corresponding token, cross entropy loss. We compare variations of this

segment, and position embeddings. A visualiza- procedure in Appendix C.2. tion of this construction can be seen in Figure 2.

Task #2: Next Sentence Prediction (NSP)

3.1 Pre-training BERT Many important downstream tasks such as Ques-

(2018), we do not use traditional left-to-right or ence (NLI) are based on understanding the rela- right-to-left language models to pre-train BERT. tionship between two sentences, which is not di-

Instead, we pre-train BERT using two unsuper- rectly captured by language modeling. In order

vised tasks, described in this section. This step to train a model that understands sentence rela- is presented in the left part of Figure 1. tionships, we pre-train for a binarized next sen- tence prediction task that can be trivially gener- Task #1: Masked LM Intuitively, it is reason- ated from any monolingual corpus. Specifically, able to believe that a deep bidirectional model is when choosing the sentences A and B for each pre-

strictly more powerful than either a left-to-right training example, 50% of the time B is the actual model or the shallow concatenation of a left-to- next sentence that follows A (labeled as IsNext), right and a right-to-left model. Unfortunately, and 50% of the time it is a random sentence from standard conditional language models can only be the corpus (labeled as NotNext). As we show trained left-to-right or right-to-left, since bidirec- in Figure 1, C is used for next sentence predic-

tional conditioning would allow each word to in- tion (NSP).5 Despite its simplicity, we demon- directly “see itself”, and the model could trivially strate in Section 5.1 that pre-training towards this predict the target word in a multi-layered context. task is very beneficial to both QA and NLI. 6 former is often referred to as a “Transformer encoder” while The final model achieves 97%-98% accuracy on NSP. the left-context-only version is referred to as a “Transformer The vector C is not a meaningful sentence representation

decoder” since it can be used for text generation. without fine-tuning, since it was trained with NSP.

Input [CLS] my dog is cute [SEP] he likes play ##ing [SEP]

E[CLS] Emy Edog Eis Ecute E[SEP] Ehe Elikes Eplay E##ing E[SEP]

Embeddings EA EA EA EA EA EA EB EB EB EB EB

Embeddings E0 E1 E2 E3 E4 E5 E6 E7 E8 E9 E10

tion embeddings and the position embeddings.

The NSP task is closely related to representation- (4) a degenerate text-∅ pair in text classification Logeswaran and Lee (2018). However, in prior resentations are fed into an output layer for token- work, only sentence embeddings are transferred to level tasks, such as sequence tagging or question down-stream tasks, where BERT transfers all pa- answering, and the [CLS] representation is fed rameters to initialize end-task model parameters. into an output layer for classification, such as en-

tailment or sentiment analysis.

Pre-training data The pre-training procedure Compared to pre-training, fine-tuning is rela-

largely follows the existing literature on language tively inexpensive. All of the results in the pa- model pre-training. For the pre-training corpus we per can be replicated in at most 1 hour on a sin- 2015) and English Wikipedia (2,500M words). from the exact same pre-trained model.7 We de- For Wikipedia we extract only the text passages scribe the task-specific details in the correspond- and ignore lists, tables, and headers. It is criti- ing subsections of Section 4. More details can be

cal to use a document-level corpus rather than a found in Appendix A.5. shuffled sentence-level corpus such as the Billion extract long contiguous sequences.

In this section, we present BERT fine-tuning re-

3.2 Fine-tuning BERT sults on 11 NLP tasks.

Fine-tuning is straightforward since the self- 4.1 GLUE

attention mechanism in the Transformer al-

The General Language Understanding Evaluation

lows BERT to model many downstream tasks— whether they involve single text or text pairs—by lection of diverse natural language understanding swapping out the appropriate inputs and outputs. tasks. Detailed descriptions of GLUE datasets are

For applications involving text pairs, a common

included in Appendix B.1. pattern is to independently encode text pairs be-

To fine-tune on GLUE, we represent the input

fore applying bidirectional cross attention, such sequence (for single sentence or sentence pairs) as described in Section 3, and use the final hid- instead uses the self-attention mechanism to unify den vector C ∈ RH corresponding to the first these two stages, as encoding a concatenated text input token ([CLS]) as the aggregate representa- pair with self-attention effectively includes bidi- tion. The only new parameters introduced during rectional cross attention between two sentences.

fine-tuning are classification layer weights W ∈ For each task, we simply plug in the task- RK×H , where K is the number of labels. We com- specific inputs and outputs into BERT and fine- pute a standard classification loss with C and W , tune all the parameters end-to-end. At the in- i.e., log(softmax(CW T )). put, sentence A and sentence B from pre-training are analogous to (1) sentence pairs in paraphras- For example, the BERT SQuAD model can be trained in

around 30 minutes on a single Cloud TPU to achieve a Dev ing, (2) hypothesis-premise pairs in entailment, (3) F1 score of 91.0%. question-passage pairs in question answering, and See (10) in https://gluebenchmark.com/faq.

System MNLI-(m/mm) QQP QNLI SST-2 CoLA STS-B MRPC RTE Average

392k 363k 108k 67k 8.5k 5.7k 3.5k 2.5k -

Pre-OpenAI SOTA 80.6/80.1 66.1 82.3 93.2 35.0 81.0 86.0 61.7 74.0

BiLSTM+ELMo+Attn 76.4/76.1 64.8 79.8 90.4 36.0 73.3 84.9 56.8 71.0

OpenAI GPT 82.1/81.4 70.3 87.4 91.3 45.4 80.0 82.3 56.0 75.1

BERTBASE 84.6/83.4 71.2 90.5 93.5 52.1 85.8 88.9 66.4 79.6

BERTLARGE 86.7/85.9 72.1 92.7 94.9 60.5 86.5 89.3 70.1 82.1

The number below each task denotes the number of training examples. The “Average” column is slightly different than the official GLUE score, since we exclude the problematic WNLI set.8 BERT and OpenAI GPT are single- model, single task. F1 scores are reported for QQP and MRPC, Spearman correlations are reported for STS-B, and accuracy scores are reported for the other tasks. We exclude entries that use BERT as one of their components.

We use a batch size of 32 and fine-tune for 3 Wikipedia containing the answer, the task is to epochs over the data for all GLUE tasks. For each predict the answer text span in the passage. task, we selected the best fine-tuning learning rate As shown in Figure 1, in the question answer- (among 5e-5, 4e-5, 3e-5, and 2e-5) on the Dev set. ing task, we represent the input question and pas- Additionally, for BERTLARGE we found that fine- sage as a single packed sequence, with the ques-

tuning was sometimes unstable on small datasets, tion using the A embedding and the passage using so we ran several random restarts and selected the the B embedding. We only introduce a start vec- best model on the Dev set. With random restarts, tor S ∈ RH and an end vector E ∈ RH during we use the same pre-trained checkpoint but per- fine-tuning. The probability of word i being the form different fine-tuning data shuffling and clas- start of the answer span is computed as a dot prod-

sifier layer initialization.9 uct between Ti and S followed by a softmax over S·Ti

Results are presented in Table 1. Both all of the words in the paragraph: Pi = Pe S·T j

BERTBASE and BERTLARGE outperform all sys-

The analogous formula is used for the end of the

Frequently Asked Questions

What is this project about?

This project covers practical implementation and research aspects of the topic using AI/ML techniques.

This section provides additional detailed analysis and supporting information derived from the research paper content to ensure comprehensive coverage of the topic with expanded discussion on key concepts, methods, and findings.

Speaker Verification Xvector