BERT Variants I - ALBERT, RoBERTa, ELECTRA, and SpanBERT

Book Overview & Buying
Table Of Contents

Getting Started with Google BERT

By : Sudharsan Ravichandiran

4.2 (50)

Buy this Book

Getting Started with Google BERT

4.2 (50)

By: Sudharsan Ravichandiran

Buy this Book

Overview of this book

BERT (bidirectional encoder representations from transformer) has revolutionized the world of natural language processing (NLP) with promising results. This book is an introductory guide that will help you get to grips with Google's BERT architecture. With a detailed explanation of the transformer architecture, this book will help you understand how the transformer’s encoder and decoder work. You’ll explore the BERT architecture by learning how the BERT model is pre-trained and how to use pre-trained BERT for downstream tasks by fine-tuning it for NLP tasks such as sentiment analysis and text summarization with the Hugging Face transformers library. As you advance, you’ll learn about different variants of BERT such as ALBERT, RoBERTa, and ELECTRA, and look at SpanBERT, which is used for NLP tasks like question answering. You'll also cover simpler and faster BERT variants based on knowledge distillation such as DistilBERT and TinyBERT. The book takes you through MBERT, XLM, and XLM-R in detail and then introduces you to sentence-BERT, which is used for obtaining sentence representation. Finally, you'll discover domain-specific BERT models such as BioBERT and ClinicalBERT, and discover an interesting variant called VideoBERT. By the end of this BERT book, you’ll be well-versed with using BERT and its variants for performing practical NLP tasks.

Preface

Who this book is for

What this book covers

To get the most out of this book

Get in touch

Section 1 - Starting Off with BERT

Free Chapter

A Primer on Transformers

Introduction to the transformer

Understanding the encoder of the transformer

Understanding the decoder of a transformer

Putting the encoder and decoder together

Training the transformer

Summary

Questions

Further reading

Understanding the BERT Model

Basic idea of BERT

Working of BERT

Configurations of BERT

Pre-training the BERT model

Subword tokenization algorithms

Summary

Questions

Further reading

Getting Hands-On with BERT

Exploring the pre-trained BERT model

Extracting embeddings from pre-trained BERT

Extracting embeddings from all encoder layers of BERT

Fine-tuning BERT for downstream tasks

Summary

Questions

Further reading

Section 2 - Exploring BERT Variants

BERT Variants I - ALBERT, RoBERTa, ELECTRA, and SpanBERT

A Lite version of BERT

Extracting embeddings with ALBERT

Robustly Optimized BERT pre-training Approach

Understanding ELECTRA

Predicting span with SpanBERT

Summary

Questions

Further reading

BERT Variants II - Based on Knowledge Distillation

Introducing knowledge distillation

DistilBERT – the distilled version of BERT

Introducing TinyBERT

Transferring knowledge from BERT to neural networks

Summary

Questions

Further reading

Section 3 - Applications of BERT

Exploring BERTSUM for Text Summarization

Text summarization

Fine-tuning BERT for text summarization

Understanding ROUGE evaluation metrics

The performance of the BERTSUM model

Training the BERTSUM model

Summary

Questions

Further reading

Applying BERT to Other Languages

Understanding multilingual BERT

How multilingual is multilingual BERT?

The cross-lingual language model

Understanding XLM-R

Language-specific BERT

Summary

Questions

Further reading

Exploring Sentence and Domain-Specific BERT

Learning about sentence representation with Sentence-BERT

Exploring the sentence-transformers library

Learning multilingual embeddings through knowledge distillation

Domain-specific BERT

Summary

Questions

Further reading

Working with VideoBERT, BART, and More

Learning language and video representations with VideoBERT

Understanding BART

Exploring BERT libraries

Summary

Questions

Further reading

Assessments

Chapter 1, A Primer on Transformers

Chapter 2, Understanding the BERT Model

Chapter 3, Getting Hands-On with BERT

Chapter 4, BERT Variants I – ALBERT, RoBERTa, ELECTRA, SpanBERT

Chapter 5, BERT Variants II – Based on Knowledge Distillation

Chapter 6, Exploring BERTSUM for Text Summarization

Chapter 7, Applying BERT to Other Languages

Chapter 8, Exploring Sentence- and Domain-Specific BERT

Chapter 9, Working with VideoBERT, BART, and More

Other Books You May Enjoy

Leave a review - let other readers know what you think

BERT Variants I - ALBERT, RoBERTa, ELECTRA, and SpanBERT

In this chapter, we will understand different variants of BERT, such as ALBERT, RoBERTa, ELECTRA, and SpanBERT. We will start with understanding how ALBERT works. ALBERT is basically A Lite version of BERT model. The ALBERT model includes few architectural changes to the BERT to minimize the training time. We will cover how ALBERT works and how it differs from BERT in detail.

Moving on, we will learn about the RoBERTa model, which stands for a Robustly Optimized BERT pre-training Approach. RoBERTa is one of the most popular variants of the BERT and it is used in many state-of-the-art systems. RoBERTa works similar to BERT but with a few changes in the pre-training steps. We will explore how RoBERTa works and how it differs from the BERT model in detail.

Going ahead, we will learn about the ELECTRA model, which stands for...