Book Image

Getting Started with Google BERT

By : Sudharsan Ravichandiran
Book Image

Getting Started with Google BERT

By: Sudharsan Ravichandiran

Overview of this book

BERT (bidirectional encoder representations from transformer) has revolutionized the world of natural language processing (NLP) with promising results. This book is an introductory guide that will help you get to grips with Google's BERT architecture. With a detailed explanation of the transformer architecture, this book will help you understand how the transformer’s encoder and decoder work. You’ll explore the BERT architecture by learning how the BERT model is pre-trained and how to use pre-trained BERT for downstream tasks by fine-tuning it for NLP tasks such as sentiment analysis and text summarization with the Hugging Face transformers library. As you advance, you’ll learn about different variants of BERT such as ALBERT, RoBERTa, and ELECTRA, and look at SpanBERT, which is used for NLP tasks like question answering. You'll also cover simpler and faster BERT variants based on knowledge distillation such as DistilBERT and TinyBERT. The book takes you through MBERT, XLM, and XLM-R in detail and then introduces you to sentence-BERT, which is used for obtaining sentence representation. Finally, you'll discover domain-specific BERT models such as BioBERT and ClinicalBERT, and discover an interesting variant called VideoBERT. By the end of this BERT book, you’ll be well-versed with using BERT and its variants for performing practical NLP tasks.

Related Content you might be interested in

Current Title:

Small book image
Getting Started with Google BERT
Sudharsan Ravichandiran
Jan, 2021 | 352 pages
Small book image
Mastering Transformers
Meysam Asgari Chenaghlu | Savaş Yıldırım
Sep, 2021 | 374 pages
Small book image
Transformers for Natural Language Processing
Denis Rothman
Jan, 2021 | 384 pages
Small book image
Advanced Natural Language Processing with TensorFlow 2
Ashish Bansal
Feb, 2021 | 380 pages
Table of Contents (15 chapters)
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Get in touch
1
Section 1 - Starting Off with BERT
Section 1 - Starting Off with BERT
Free Chapter
2
A Primer on Transformers
A Primer on Transformers
Introduction to the transformer
Understanding the encoder of the transformer
Understanding the decoder of a transformer
Putting the encoder and decoder together
Training the transformer
Summary
Questions
Further reading
3
Understanding the BERT Model
Understanding the BERT Model
Basic idea of BERT
Working of BERT
Configurations of BERT
Pre-training the BERT model
Subword tokenization algorithms
Summary
Questions
Further reading
4
Getting Hands-On with BERT
Getting Hands-On with BERT
Exploring the pre-trained BERT model
Extracting embeddings from pre-trained BERT
Extracting embeddings from all encoder layers of BERT
Fine-tuning BERT for downstream tasks
Summary
Questions
Further reading
5
Section 2 - Exploring BERT Variants
Section 2 - Exploring BERT Variants
6
BERT Variants I - ALBERT, RoBERTa, ELECTRA, and SpanBERT
BERT Variants I - ALBERT, RoBERTa, ELECTRA, and SpanBERT
A Lite version of BERT
Extracting embeddings with ALBERT
Robustly Optimized BERT pre-training Approach
Understanding ELECTRA
Predicting span with SpanBERT
Summary
Questions
Further reading
7
BERT Variants II - Based on Knowledge Distillation
BERT Variants II - Based on Knowledge Distillation
Introducing knowledge distillation
DistilBERT – the distilled version of BERT
Introducing TinyBERT
Transferring knowledge from BERT to neural networks
Summary
Questions
Further reading
8
Section 3 - Applications of BERT
Section 3 - Applications of BERT
9
Exploring BERTSUM for Text Summarization
Exploring BERTSUM for Text Summarization
Text summarization
Fine-tuning BERT for text summarization
Understanding ROUGE evaluation metrics
The performance of the BERTSUM model
Training the BERTSUM model
Summary
Questions
Further reading
10
Applying BERT to Other Languages
Applying BERT to Other Languages
Understanding multilingual BERT
How multilingual is multilingual BERT?
The cross-lingual language model
Understanding XLM-R
Language-specific BERT
Summary
Questions
Further reading
11
Exploring Sentence and Domain-Specific BERT
Exploring Sentence and Domain-Specific BERT
Learning about sentence representation with Sentence-BERT
Exploring the sentence-transformers library
Learning multilingual embeddings through knowledge distillation
Domain-specific BERT
Summary
Questions
Further reading
12
Working with VideoBERT, BART, and More
Working with VideoBERT, BART, and More
Learning language and video representations with VideoBERT
Understanding BART
Exploring BERT libraries
Summary
Questions
Further reading
13
Assessments
Assessments
Chapter 1, A Primer on Transformers
Chapter 2, Understanding the BERT Model
Chapter 3, Getting Hands-On with BERT
Chapter 4, BERT Variants I – ALBERT, RoBERTa, ELECTRA, SpanBERT
Chapter 5, BERT Variants II – Based on Knowledge Distillation
Chapter 6, Exploring BERTSUM for Text Summarization
Chapter 7, Applying BERT to Other Languages
Chapter 8, Exploring Sentence- and Domain-Specific BERT
Chapter 9, Working with VideoBERT, BART, and More
14
Other Books You May Enjoy
Other Books You May Enjoy
Leave a review - let other readers know what you think

To get the most out of this book

To get the most out of the book, run all the code provided in the book using Google Colab.

Software/Hardware requirements

Operating System

Google Colab / Python 3.x

Windows/macOS/Linux

Download the example code files

You can download the example code files for this book from GitHub at https://github.com/PacktPublishing/Getting-Started-with-Google-BERT. In case there's an update to the code, it will be updated on the existing GitHub repository.

We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

Download the color images

We also provide a PDF file that has color images of the screenshots/diagrams used in this book. You can download it here: https://static.packt-cdn.com/downloads/9781838821593_ColorImages.pdf.

Conventions used

There are a number of text conventions used throughout this book.

CodeInText: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: "We will set maxlen to 100 and max_features to 100000."

A block of code is set as follows:

(x_train, y_train), (x_test, y_test), preproc = \
text.texts_from_df(train_df = df,  
                   text_column = 'reviewText',
                   label_columns=['sentiment'], 
                   maxlen=100, 
                   max_features=100000, 
                   preprocess_mode='bert', 
                   val_pct=0.1)

Bold: Indicates a new term, an important word, or words that you see onscreen. For example, words in menus or dialog boxes appear in the text like this. Here is an example: "Select System info from the Administration panel."

Warnings or important notes appear like this.
Tips and tricks appear like this.
Previous Section
Next Section