Book Image

Getting Started with Google BERT

By : Sudharsan Ravichandiran
Book Image

Getting Started with Google BERT

By: Sudharsan Ravichandiran

Overview of this book

BERT (bidirectional encoder representations from transformer) has revolutionized the world of natural language processing (NLP) with promising results. This book is an introductory guide that will help you get to grips with Google's BERT architecture. With a detailed explanation of the transformer architecture, this book will help you understand how the transformer’s encoder and decoder work. You’ll explore the BERT architecture by learning how the BERT model is pre-trained and how to use pre-trained BERT for downstream tasks by fine-tuning it for NLP tasks such as sentiment analysis and text summarization with the Hugging Face transformers library. As you advance, you’ll learn about different variants of BERT such as ALBERT, RoBERTa, and ELECTRA, and look at SpanBERT, which is used for NLP tasks like question answering. You'll also cover simpler and faster BERT variants based on knowledge distillation such as DistilBERT and TinyBERT. The book takes you through MBERT, XLM, and XLM-R in detail and then introduces you to sentence-BERT, which is used for obtaining sentence representation. Finally, you'll discover domain-specific BERT models such as BioBERT and ClinicalBERT, and discover an interesting variant called VideoBERT. By the end of this BERT book, you’ll be well-versed with using BERT and its variants for performing practical NLP tasks.

Related Content you might be interested in

Current Title:

Small book image
Getting Started with Google BERT
Sudharsan Ravichandiran
Jan, 2021 | 352 pages
Small book image
Mastering Transformers
Meysam Asgari Chenaghlu | Savaş Yıldırım
Sep, 2021 | 374 pages
Small book image
Transformers for Natural Language Processing
Denis Rothman
Jan, 2021 | 384 pages
Small book image
Advanced Natural Language Processing with TensorFlow 2
Ashish Bansal
Feb, 2021 | 380 pages
Table of Contents (15 chapters)
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Get in touch
1
Section 1 - Starting Off with BERT
Section 1 - Starting Off with BERT
Free Chapter
2
A Primer on Transformers
A Primer on Transformers
Introduction to the transformer
Understanding the encoder of the transformer
Understanding the decoder of a transformer
Putting the encoder and decoder together
Training the transformer
Summary
Questions
Further reading
3
Understanding the BERT Model
Understanding the BERT Model
Basic idea of BERT
Working of BERT
Configurations of BERT
Pre-training the BERT model
Subword tokenization algorithms
Summary
Questions
Further reading
4
Getting Hands-On with BERT
Getting Hands-On with BERT
Exploring the pre-trained BERT model
Extracting embeddings from pre-trained BERT
Extracting embeddings from all encoder layers of BERT
Fine-tuning BERT for downstream tasks
Summary
Questions
Further reading
5
Section 2 - Exploring BERT Variants
Section 2 - Exploring BERT Variants
6
BERT Variants I - ALBERT, RoBERTa, ELECTRA, and SpanBERT
BERT Variants I - ALBERT, RoBERTa, ELECTRA, and SpanBERT
A Lite version of BERT
Extracting embeddings with ALBERT
Robustly Optimized BERT pre-training Approach
Understanding ELECTRA
Predicting span with SpanBERT
Summary
Questions
Further reading
7
BERT Variants II - Based on Knowledge Distillation
BERT Variants II - Based on Knowledge Distillation
Introducing knowledge distillation
DistilBERT – the distilled version of BERT
Introducing TinyBERT
Transferring knowledge from BERT to neural networks
Summary
Questions
Further reading
8
Section 3 - Applications of BERT
Section 3 - Applications of BERT
9
Exploring BERTSUM for Text Summarization
Exploring BERTSUM for Text Summarization
Text summarization
Fine-tuning BERT for text summarization
Understanding ROUGE evaluation metrics
The performance of the BERTSUM model
Training the BERTSUM model
Summary
Questions
Further reading
10
Applying BERT to Other Languages
Applying BERT to Other Languages
Understanding multilingual BERT
How multilingual is multilingual BERT?
The cross-lingual language model
Understanding XLM-R
Language-specific BERT
Summary
Questions
Further reading
11
Exploring Sentence and Domain-Specific BERT
Exploring Sentence and Domain-Specific BERT
Learning about sentence representation with Sentence-BERT
Exploring the sentence-transformers library
Learning multilingual embeddings through knowledge distillation
Domain-specific BERT
Summary
Questions
Further reading
12
Working with VideoBERT, BART, and More
Working with VideoBERT, BART, and More
Learning language and video representations with VideoBERT
Understanding BART
Exploring BERT libraries
Summary
Questions
Further reading
13
Assessments
Assessments
Chapter 1, A Primer on Transformers
Chapter 2, Understanding the BERT Model
Chapter 3, Getting Hands-On with BERT
Chapter 4, BERT Variants I – ALBERT, RoBERTa, ELECTRA, SpanBERT
Chapter 5, BERT Variants II – Based on Knowledge Distillation
Chapter 6, Exploring BERTSUM for Text Summarization
Chapter 7, Applying BERT to Other Languages
Chapter 8, Exploring Sentence- and Domain-Specific BERT
Chapter 9, Working with VideoBERT, BART, and More
14
Other Books You May Enjoy
Other Books You May Enjoy
Leave a review - let other readers know what you think

How multilingual is multilingual BERT?

In the previous section, we learned about M-BERT. We learned that M-BERT is trained on the Wikipedia text of 104 different languages. We also evaluated M-BERT by fine-tuning it on the XNLI dataset. But how multilingual is our M-BERT? How is a single model able to transfer knowledge across multiple languages? To understand this, in this section, let's investigate the multilingual ability of M-BERT in more detail.

Effect of vocabulary overlap

We learned that M-BERT is trained on the Wikipedia text of 104 languages and that it consists of a shared vocabulary of 110k tokens. In this section, let's investigate whether the multilingual knowledge transfer of M-BERT depends on the vocabulary overlap.

We learned that M-BERT is good at zero-shot transfer, that is, we can fine-tune M-BERT in one language and use the fine-tuned M-BERT model in other languages. Let's say we are performing an NER task. Suppose we fine-tune M-BERT for the NER task...

Previous Section
End of Section 3
Next Section