Book Image

Getting Started with Google BERT

By : Sudharsan Ravichandiran
Book Image

Getting Started with Google BERT

By: Sudharsan Ravichandiran

Overview of this book

BERT (bidirectional encoder representations from transformer) has revolutionized the world of natural language processing (NLP) with promising results. This book is an introductory guide that will help you get to grips with Google's BERT architecture. With a detailed explanation of the transformer architecture, this book will help you understand how the transformer’s encoder and decoder work. You’ll explore the BERT architecture by learning how the BERT model is pre-trained and how to use pre-trained BERT for downstream tasks by fine-tuning it for NLP tasks such as sentiment analysis and text summarization with the Hugging Face transformers library. As you advance, you’ll learn about different variants of BERT such as ALBERT, RoBERTa, and ELECTRA, and look at SpanBERT, which is used for NLP tasks like question answering. You'll also cover simpler and faster BERT variants based on knowledge distillation such as DistilBERT and TinyBERT. The book takes you through MBERT, XLM, and XLM-R in detail and then introduces you to sentence-BERT, which is used for obtaining sentence representation. Finally, you'll discover domain-specific BERT models such as BioBERT and ClinicalBERT, and discover an interesting variant called VideoBERT. By the end of this BERT book, you’ll be well-versed with using BERT and its variants for performing practical NLP tasks.
Table of Contents (15 chapters)
1
Section 1 - Starting Off with BERT
5
Section 2 - Exploring BERT Variants
8
Section 3 - Applications of BERT

Summary

We started off the chapter by understanding what knowledge distillation is and how it works. We learned that knowledge distillation is a model compression technique in which a small model is trained to reproduce the behavior of a large pre-trained model. It is also referred to as teacher-student learning, where the large pre-trained model is the teacher and the small model is the student.

Next, we learned about DistilBERT where we take a large pre-trained BERT as a teacher and transfer its knowledge to a small BERT through knowledge distillation.

Following DistilBERT, we learned how TinyBERT works. In TinyBERT, apart from transferring knowledge from the output layer of the teacher, we also transfer knowledge from other layers, such as the embedding layer, transformer layer, and prediction layer.

Moving on, at the end of the chapter, we learned how to transfer task-specific knowledge from BERT to a simple neural network. In the next chapter, we will learn how to fine-tune the...