Understanding the BERT Model | Getting Started with Google BERT

Book Overview & Buying
Table Of Contents

Getting Started with Google BERT

By : Sudharsan Ravichandiran

4.2 (50)

Buy this Book

Getting Started with Google BERT

4.2 (50)

By: Sudharsan Ravichandiran

Buy this Book

Overview of this book

BERT (bidirectional encoder representations from transformer) has revolutionized the world of natural language processing (NLP) with promising results. This book is an introductory guide that will help you get to grips with Google's BERT architecture. With a detailed explanation of the transformer architecture, this book will help you understand how the transformer’s encoder and decoder work. You’ll explore the BERT architecture by learning how the BERT model is pre-trained and how to use pre-trained BERT for downstream tasks by fine-tuning it for NLP tasks such as sentiment analysis and text summarization with the Hugging Face transformers library. As you advance, you’ll learn about different variants of BERT such as ALBERT, RoBERTa, and ELECTRA, and look at SpanBERT, which is used for NLP tasks like question answering. You'll also cover simpler and faster BERT variants based on knowledge distillation such as DistilBERT and TinyBERT. The book takes you through MBERT, XLM, and XLM-R in detail and then introduces you to sentence-BERT, which is used for obtaining sentence representation. Finally, you'll discover domain-specific BERT models such as BioBERT and ClinicalBERT, and discover an interesting variant called VideoBERT. By the end of this BERT book, you’ll be well-versed with using BERT and its variants for performing practical NLP tasks.

Preface

Who this book is for

What this book covers

To get the most out of this book

Get in touch

Section 1 - Starting Off with BERT

Free Chapter

A Primer on Transformers

Introduction to the transformer

Understanding the encoder of the transformer

Understanding the decoder of a transformer

Putting the encoder and decoder together

Training the transformer

Summary

Questions

Further reading

Understanding the BERT Model

Basic idea of BERT

Working of BERT

Configurations of BERT

Pre-training the BERT model

Subword tokenization algorithms

Summary

Questions

Further reading

Getting Hands-On with BERT

Exploring the pre-trained BERT model

Extracting embeddings from pre-trained BERT

Extracting embeddings from all encoder layers of BERT

Fine-tuning BERT for downstream tasks

Summary

Questions

Further reading

Section 2 - Exploring BERT Variants

BERT Variants I - ALBERT, RoBERTa, ELECTRA, and SpanBERT

A Lite version of BERT

Extracting embeddings with ALBERT

Robustly Optimized BERT pre-training Approach

Understanding ELECTRA

Predicting span with SpanBERT

Summary

Questions

Further reading

BERT Variants II - Based on Knowledge Distillation

Introducing knowledge distillation

DistilBERT – the distilled version of BERT

Introducing TinyBERT

Transferring knowledge from BERT to neural networks

Summary

Questions

Further reading

Section 3 - Applications of BERT

Exploring BERTSUM for Text Summarization

Text summarization

Fine-tuning BERT for text summarization

Understanding ROUGE evaluation metrics

The performance of the BERTSUM model

Training the BERTSUM model

Summary

Questions

Further reading

Applying BERT to Other Languages

Understanding multilingual BERT

How multilingual is multilingual BERT?

The cross-lingual language model

Understanding XLM-R

Language-specific BERT

Summary

Questions

Further reading

Exploring Sentence and Domain-Specific BERT

Learning about sentence representation with Sentence-BERT

Exploring the sentence-transformers library

Learning multilingual embeddings through knowledge distillation

Domain-specific BERT

Summary

Questions

Further reading

Working with VideoBERT, BART, and More

Learning language and video representations with VideoBERT

Understanding BART

Exploring BERT libraries

Summary

Questions

Further reading

Assessments

Chapter 1, A Primer on Transformers

Chapter 2, Understanding the BERT Model

Chapter 3, Getting Hands-On with BERT

Chapter 4, BERT Variants I – ALBERT, RoBERTa, ELECTRA, SpanBERT

Chapter 5, BERT Variants II – Based on Knowledge Distillation

Chapter 6, Exploring BERTSUM for Text Summarization

Chapter 7, Applying BERT to Other Languages

Chapter 8, Exploring Sentence- and Domain-Specific BERT

Chapter 9, Working with VideoBERT, BART, and More

Other Books You May Enjoy

Leave a review - let other readers know what you think

Understanding the BERT Model

In this chapter, we will get started with one of the most popularly used state-of-the-art text embedding models called BERT. BERT has revolutionized the world of NLP by providing state-of-the-art results on many NLP tasks. We will begin the chapter by understanding what BERT is and how it differs from the other embedding models. We will then look into the working of BERT and its configuration in detail.

Moving on, we will learn how the BERT model is pre-trained using two tasks, called masked language modeling and next sentence prediction, in detail. We will then look into the pre-training procedure of BERT. At the end of the chapter, we will learn about several interesting subword tokenization algorithms, including byte pair encoding, byte-level byte pair encoding, and WordPiece.

In this chapter, we will cover the following topics: