Book Image

Getting Started with Google BERT

By : Sudharsan Ravichandiran

Book Image

Getting Started with Google BERT

By: Sudharsan Ravichandiran

Overview of this book

BERT (bidirectional encoder representations from transformer) has revolutionized the world of natural language processing (NLP) with promising results. This book is an introductory guide that will help you get to grips with Google's BERT architecture. With a detailed explanation of the transformer architecture, this book will help you understand how the transformer’s encoder and decoder work. You’ll explore the BERT architecture by learning how the BERT model is pre-trained and how to use pre-trained BERT for downstream tasks by fine-tuning it for NLP tasks such as sentiment analysis and text summarization with the Hugging Face transformers library. As you advance, you’ll learn about different variants of BERT such as ALBERT, RoBERTa, and ELECTRA, and look at SpanBERT, which is used for NLP tasks like question answering. You'll also cover simpler and faster BERT variants based on knowledge distillation such as DistilBERT and TinyBERT. The book takes you through MBERT, XLM, and XLM-R in detail and then introduces you to sentence-BERT, which is used for obtaining sentence representation. Finally, you'll discover domain-specific BERT models such as BioBERT and ClinicalBERT, and discover an interesting variant called VideoBERT. By the end of this BERT book, you’ll be well-versed with using BERT and its variants for performing practical NLP tasks.

Preface

Who this book is for

What this book covers

To get the most out of this book

Section 1 - Starting Off with BERT

Section 1 - Starting Off with BERT

Free Chapter

A Primer on Transformers

A Primer on Transformers

Introduction to the transformer

Understanding the encoder of the transformer

Understanding the decoder of a transformer

Putting the encoder and decoder together

Training the transformer

Further reading

Understanding the BERT Model

Understanding the BERT Model

Basic idea of BERT

Working of BERT

Configurations of BERT

Pre-training the BERT model

Subword tokenization algorithms

Further reading

Getting Hands-On with BERT

Getting Hands-On with BERT

Exploring the pre-trained BERT model

Extracting embeddings from pre-trained BERT

Extracting embeddings from all encoder layers of BERT

Fine-tuning BERT for downstream tasks

Further reading

Section 2 - Exploring BERT Variants

Section 2 - Exploring BERT Variants

BERT Variants I - ALBERT, RoBERTa, ELECTRA, and SpanBERT

BERT Variants I - ALBERT, RoBERTa, ELECTRA, and SpanBERT

A Lite version of BERT

Extracting embeddings with ALBERT

Robustly Optimized BERT pre-training Approach

Understanding ELECTRA

Predicting span with SpanBERT

Further reading

BERT Variants II - Based on Knowledge Distillation

BERT Variants II - Based on Knowledge Distillation

Introducing knowledge distillation

DistilBERT – the distilled version of BERT

Introducing TinyBERT

Transferring knowledge from BERT to neural networks

Further reading

Section 3 - Applications of BERT

Section 3 - Applications of BERT

Exploring BERTSUM for Text Summarization

Exploring BERTSUM for Text Summarization

Text summarization

Fine-tuning BERT for text summarization

Understanding ROUGE evaluation metrics

The performance of the BERTSUM model

Training the BERTSUM model

Further reading

Applying BERT to Other Languages

Applying BERT to Other Languages

Understanding multilingual BERT

How multilingual is multilingual BERT?

The cross-lingual language model

Understanding XLM-R

Language-specific BERT

Further reading

Exploring Sentence and Domain-Specific BERT

Exploring Sentence and Domain-Specific BERT

Learning about sentence representation with Sentence-BERT

Exploring the sentence-transformers library

Learning multilingual embeddings through knowledge distillation

Domain-specific BERT

Further reading

Working with VideoBERT, BART, and More

Working with VideoBERT, BART, and More

Learning language and video representations with VideoBERT

Understanding BART

Exploring BERT libraries

Further reading

Assessments

Chapter 1, A Primer on Transformers

Chapter 2, Understanding the BERT Model

Chapter 3, Getting Hands-On with BERT

Chapter 4, BERT Variants I – ALBERT, RoBERTa, ELECTRA, SpanBERT

Chapter 5, BERT Variants II – Based on Knowledge Distillation

Chapter 6, Exploring BERTSUM for Text Summarization

Chapter 7, Applying BERT to Other Languages

Chapter 8, Exploring Sentence- and Domain-Specific BERT

Chapter 9, Working with VideoBERT, BART, and More

Other Books You May Enjoy

Other Books You May Enjoy

Leave a review - let other readers know what you think

Customer Reviews

5 star

0

4 star

0

3 star

0

2 star

0

1 star

0

Further reading

For more information, refer to the following papers:

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations by Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut, available at https://arxiv.org/pdf/1909.11942.pdf
RoBERTa: A Robustly Optimized BERT Pre-training Approach by Yinhan Liu, Myle Ott, et al., available at https://arxiv.org/pdf/1907.11692.pdf
ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators by Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning, available at https://arxiv.org/pdf/2003.10555.pdf
SpanBERT: Improving Pre-training by Representing and Predicting Spans by Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy, available at https://arxiv.org/pdf/1907.10529v3.pdf