BERT Variants II - Based on Knowledge Distillation | Getting Started with Google BERT

Book Overview & Buying
Table Of Contents

Getting Started with Google BERT

By : Sudharsan Ravichandiran

4.2 (50)

Buy this Book

Getting Started with Google BERT

4.2 (50)

By: Sudharsan Ravichandiran

Buy this Book

Overview of this book

BERT (bidirectional encoder representations from transformer) has revolutionized the world of natural language processing (NLP) with promising results. This book is an introductory guide that will help you get to grips with Google's BERT architecture. With a detailed explanation of the transformer architecture, this book will help you understand how the transformer’s encoder and decoder work. You’ll explore the BERT architecture by learning how the BERT model is pre-trained and how to use pre-trained BERT for downstream tasks by fine-tuning it for NLP tasks such as sentiment analysis and text summarization with the Hugging Face transformers library. As you advance, you’ll learn about different variants of BERT such as ALBERT, RoBERTa, and ELECTRA, and look at SpanBERT, which is used for NLP tasks like question answering. You'll also cover simpler and faster BERT variants based on knowledge distillation such as DistilBERT and TinyBERT. The book takes you through MBERT, XLM, and XLM-R in detail and then introduces you to sentence-BERT, which is used for obtaining sentence representation. Finally, you'll discover domain-specific BERT models such as BioBERT and ClinicalBERT, and discover an interesting variant called VideoBERT. By the end of this BERT book, you’ll be well-versed with using BERT and its variants for performing practical NLP tasks.

Preface

Who this book is for

What this book covers

To get the most out of this book

Get in touch

Section 1 - Starting Off with BERT

Free Chapter

A Primer on Transformers

Introduction to the transformer

Understanding the encoder of the transformer

Understanding the decoder of a transformer

Putting the encoder and decoder together

Training the transformer

Summary

Questions

Further reading

Understanding the BERT Model

Basic idea of BERT

Working of BERT

Configurations of BERT

Pre-training the BERT model

Subword tokenization algorithms

Summary

Questions

Further reading

Getting Hands-On with BERT

Exploring the pre-trained BERT model

Extracting embeddings from pre-trained BERT

Extracting embeddings from all encoder layers of BERT

Fine-tuning BERT for downstream tasks

Summary

Questions

Further reading

Section 2 - Exploring BERT Variants

BERT Variants I - ALBERT, RoBERTa, ELECTRA, and SpanBERT

A Lite version of BERT

Extracting embeddings with ALBERT

Robustly Optimized BERT pre-training Approach

Understanding ELECTRA

Predicting span with SpanBERT

Summary

Questions

Further reading

BERT Variants II - Based on Knowledge Distillation

Introducing knowledge distillation

DistilBERT – the distilled version of BERT

Introducing TinyBERT

Transferring knowledge from BERT to neural networks

Summary

Questions

Further reading

Section 3 - Applications of BERT

Exploring BERTSUM for Text Summarization

Text summarization

Fine-tuning BERT for text summarization

Understanding ROUGE evaluation metrics

The performance of the BERTSUM model

Training the BERTSUM model

Summary

Questions

Further reading

Applying BERT to Other Languages

Understanding multilingual BERT

How multilingual is multilingual BERT?

The cross-lingual language model

Understanding XLM-R

Language-specific BERT

Summary

Questions

Further reading

Exploring Sentence and Domain-Specific BERT

Learning about sentence representation with Sentence-BERT

Exploring the sentence-transformers library

Learning multilingual embeddings through knowledge distillation

Domain-specific BERT

Summary

Questions

Further reading

Working with VideoBERT, BART, and More

Learning language and video representations with VideoBERT

Understanding BART

Exploring BERT libraries

Summary

Questions

Further reading

Assessments

Chapter 1, A Primer on Transformers

Chapter 2, Understanding the BERT Model

Chapter 3, Getting Hands-On with BERT

Chapter 4, BERT Variants I – ALBERT, RoBERTa, ELECTRA, SpanBERT

Chapter 5, BERT Variants II – Based on Knowledge Distillation

Chapter 6, Exploring BERTSUM for Text Summarization

Chapter 7, Applying BERT to Other Languages

Chapter 8, Exploring Sentence- and Domain-Specific BERT

Chapter 9, Working with VideoBERT, BART, and More

Other Books You May Enjoy

Leave a review - let other readers know what you think

BERT Variants II - Based on Knowledge Distillation

In the previous chapters, we learned how BERT works, and we also looked into different variants of BERT. We learned that we don't have to train BERT from scratch; instead, we can fine-tune the pre-trained BERT model on downstream tasks. However, one of the challenges with using the pre-trained BERT model is that it is computationally expensive and it is very difficult to run the model with limited resources. The pre-trained BERT model has a large number of parameters and also high inference time, which makes it harder to use it on edge devices such as mobile phones.

To alleviate this issue, we transfer knowledge from a large pre-trained BERT to a small BERT using knowledge distillation. In this chapter, we will learn about several variants of the BERT model that are based on knowledge distillation.

We will begin the chapter...