Getting Started with Google BERT

By : Sudharsan Ravichandiran

Getting Started with Google BERT

By: Sudharsan Ravichandiran

Overview of this book

BERT (bidirectional encoder representations from transformer) has revolutionized the world of natural language processing (NLP) with promising results. This book is an introductory guide that will help you get to grips with Google's BERT architecture. With a detailed explanation of the transformer architecture, this book will help you understand how the transformer’s encoder and decoder work. You’ll explore the BERT architecture by learning how the BERT model is pre-trained and how to use pre-trained BERT for downstream tasks by fine-tuning it for NLP tasks such as sentiment analysis and text summarization with the Hugging Face transformers library. As you advance, you’ll learn about different variants of BERT such as ALBERT, RoBERTa, and ELECTRA, and look at SpanBERT, which is used for NLP tasks like question answering. You'll also cover simpler and faster BERT variants based on knowledge distillation such as DistilBERT and TinyBERT. The book takes you through MBERT, XLM, and XLM-R in detail and then introduces you to sentence-BERT, which is used for obtaining sentence representation. Finally, you'll discover domain-specific BERT models such as BioBERT and ClinicalBERT, and discover an interesting variant called VideoBERT. By the end of this BERT book, you’ll be well-versed with using BERT and its variants for performing practical NLP tasks.

Preface

Who this book is for

What this book covers

To get the most out of this book

Get in touch

Section 1 - Starting Off with BERT

Free Chapter

A Primer on Transformers

Introduction to the transformer

Understanding the encoder of the transformer

Understanding the decoder of a transformer

Putting the encoder and decoder together

Training the transformer

Summary

Questions

Further reading

Understanding the BERT Model

Basic idea of BERT

Working of BERT

Configurations of BERT

Pre-training the BERT model

Subword tokenization algorithms

Summary

Questions

Further reading

Getting Hands-On with BERT

Exploring the pre-trained BERT model

Extracting embeddings from pre-trained BERT

Extracting embeddings from all encoder layers of BERT

Fine-tuning BERT for downstream tasks

Summary

Questions

Further reading

Section 2 - Exploring BERT Variants

BERT Variants I - ALBERT, RoBERTa, ELECTRA, and SpanBERT

A Lite version of BERT

Extracting embeddings with ALBERT

Robustly Optimized BERT pre-training Approach

Understanding ELECTRA

Predicting span with SpanBERT

Summary

Questions

Further reading

BERT Variants II - Based on Knowledge Distillation

Introducing knowledge distillation

DistilBERT – the distilled version of BERT

Introducing TinyBERT

Transferring knowledge from BERT to neural networks

Summary

Questions

Further reading

Section 3 - Applications of BERT

Exploring BERTSUM for Text Summarization

Text summarization

Fine-tuning BERT for text summarization

Understanding ROUGE evaluation metrics

The performance of the BERTSUM model

Training the BERTSUM model

Summary

Questions

Further reading

Applying BERT to Other Languages

Understanding multilingual BERT

How multilingual is multilingual BERT?

The cross-lingual language model

Understanding XLM-R

Language-specific BERT

Summary

Questions

Further reading

Exploring Sentence and Domain-Specific BERT

Learning about sentence representation with Sentence-BERT

Exploring the sentence-transformers library

Learning multilingual embeddings through knowledge distillation

Domain-specific BERT

Summary

Questions

Further reading

Working with VideoBERT, BART, and More

Learning language and video representations with VideoBERT

Understanding BART

Exploring BERT libraries

Summary

Questions

Further reading

Assessments

Chapter 1, A Primer on Transformers

Chapter 2, Understanding the BERT Model

Chapter 3, Getting Hands-On with BERT

Chapter 4, BERT Variants I – ALBERT, RoBERTa, ELECTRA, SpanBERT

Chapter 5, BERT Variants II – Based on Knowledge Distillation

Chapter 6, Exploring BERTSUM for Text Summarization

Chapter 7, Applying BERT to Other Languages

Chapter 8, Exploring Sentence- and Domain-Specific BERT

Chapter 9, Working with VideoBERT, BART, and More

Other Books You May Enjoy

Leave a review - let other readers know what you think

Customer Reviews

5 star

4 star

3 star

2 star

1 star

Understanding the encoder of the transformer

The transformer consists of a stack of number of encoders. The output of one encoder is sent as input to the encoder above it. As shown in the following figure, we have a stack of number of encoders. Each encoder sends its output to the encoder above it. The final encoder returns the representation of the given source sentence as output. We feed the source sentence as input to the encoder and get the representation of the source sentence as output:

Figure 1.2 – A stack of N number of encoders

Note that in the transformer paper Attention Is All You Need, the authors have used , meaning that they stacked up six encoders one above the another. However, we can try out different values of . For simplicity and better understanding, let's keep :

Figure 1.3 – A stack of encoders

Okay, the question is how exactly does the encoder work? How is it generating the representation for the given source sentence (input sentence)...

Getting Started with Google BERT

By : Sudharsan Ravichandiran

Getting Started with Google BERT

By: Sudharsan Ravichandiran

Overview of this book

Related Content you might be interested in

Current Title:

Getting Started with Google BERT

Mastering Transformers

Transformers for Natural Language Processing

Advanced Natural Language Processing with TensorFlow 2

Understanding the encoder of the transformer