Book Image

Transformers for Natural Language Processing - Second Edition

By : Denis Rothman
5 (1)
Book Image

Transformers for Natural Language Processing - Second Edition

5 (1)
By: Denis Rothman

Overview of this book

Transformers are...well...transforming the world of AI. There are many platforms and models out there, but which ones best suit your needs? Transformers for Natural Language Processing, 2nd Edition, guides you through the world of transformers, highlighting the strengths of different models and platforms, while teaching you the problem-solving skills you need to tackle model weaknesses. You'll use Hugging Face to pretrain a RoBERTa model from scratch, from building the dataset to defining the data collator to training the model. If you're looking to fine-tune a pretrained model, including GPT-3, then Transformers for Natural Language Processing, 2nd Edition, shows you how with step-by-step guides. The book investigates machine translations, speech-to-text, text-to-speech, question-answering, and many more NLP tasks. It provides techniques to solve hard language problems and may even help with fake news anxiety (read chapter 13 for more details). You'll see how cutting-edge platforms, such as OpenAI, have taken transformers beyond language into computer vision tasks and code creation using DALL-E 2, ChatGPT, and GPT-4. By the end of this book, you'll know how transformers work and how to implement them and resolve issues like an AI detective.

Related Content you might be interested in

Current Title:

Small book image
Transformers for Natural Language Processing - Second Edition
Denis Rothman
Mar, 2022 | 602 pages
Small book image
Mastering Transformers
Meysam Asgari Chenaghlu | Savaş Yıldırım
Sep, 2021 | 374 pages
Small book image
Getting Started with Google BERT
Sudharsan Ravichandiran
Jan, 2021 | 352 pages
Small book image
Pretrain Vision and Large Language Models in Python
Emily Webber
May, 2023 | 258 pages
Table of Contents (25 chapters)
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Get in touch
Free Chapter
1
What are Transformers?
What are Transformers?
The ecosystem of transformers
Optimizing NLP models with transformers
What resources should we use?
Summary
Questions
References
2
Getting Started with the Architecture of the Transformer Model
Getting Started with the Architecture of the Transformer Model
The rise of the Transformer: Attention is All You Need
Training and performance
Tranformer models in Hugging Face
Summary
Questions
References
3
Fine-Tuning BERT Models
Fine-Tuning BERT Models
The architecture of BERT
Fine-tuning BERT
Summary
Questions
References
4
Pretraining a RoBERTa Model from Scratch
Pretraining a RoBERTa Model from Scratch
Training a tokenizer and pretraining a transformer
Building KantaiBERT from scratch
Next steps
Summary
Questions
References
5
Downstream NLP Tasks with Transformers
Downstream NLP Tasks with Transformers
Transduction and the inductive inheritance of transformers
Transformer performances versus Human Baselines
Running downstream tasks
Summary
Questions
References
6
Machine Translation with the Transformer
Machine Translation with the Transformer
Defining machine translation
Preprocessing a WMT dataset
Evaluating machine translation with BLEU
Translation with Google Translate
Translations with Trax
Summary
Questions
References
7
The Rise of Suprahuman Transformers with GPT-3 Engines
The Rise of Suprahuman Transformers with GPT-3 Engines
Suprahuman NLP with GPT-3 transformer models
The architecture of OpenAI GPT transformer models
Generic text completion with GPT-2
Training a custom GPT-2 language model
Running OpenAI GPT-3 tasks
Comparing the output of GPT-2 and GPT-3
Fine-tuning GPT-3
The role of an Industry 4.0 AI specialist
Summary
Questions
References
8
Applying Transformers to Legal and Financial Documents for AI Text Summarization
Applying Transformers to Legal and Financial Documents for AI Text Summarization
Designing a universal text-to-text model
Text summarization with T5
Summarization with GPT-3
Summary
Questions
References
9
Matching Tokenizers and Datasets
Matching Tokenizers and Datasets
Matching datasets and tokenizers
Standard NLP tasks with specific vocabulary
Exploring the scope of GPT-3
Summary
Questions
References
10
Semantic Role Labeling with BERT-Based Transformers
Semantic Role Labeling with BERT-Based Transformers
Getting started with SRL
SRL experiments with the BERT-based model
Basic samples
Difficult samples
Questioning the scope of SRL
Summary
Questions
References
11
Let Your Data Do the Talking: Story, Questions, and Answers
Let Your Data Do the Talking: Story, Questions, and Answers
Methodology
Method 0: Trial and error
Method 1: NER first
Method 2: SRL first
Next steps
Summary
Questions
References
12
Detecting Customer Emotions to Make Predictions
Detecting Customer Emotions to Make Predictions
Getting started: Sentiment analysis transformers
The Stanford Sentiment Treebank (SST)
Predicting customer behavior with sentiment analysis
Sentiment analysis with GPT-3
Some Pragmatic I4.0 thinking before we leave
Summary
Questions
References
13
Analyzing Fake News with Transformers
Analyzing Fake News with Transformers
Emotional reactions to fake news
A rational approach to fake news
Before we go
Summary
Questions
References
14
Interpreting Black Box Transformer Models
Interpreting Black Box Transformer Models
Transformer visualization with BertViz
LIT
Transformer visualization via dictionary learning
Exploring models we cannot access
Summary
Questions
References
15
From NLP to Task-Agnostic Transformer Models
From NLP to Task-Agnostic Transformer Models
Choosing a model and an ecosystem
The Reformer
DeBERTa
From Task-Agnostic Models to Vision Transformers
An expanding universe of models
Summary
Questions
References
16
The Emergence of Transformer-Driven Copilots
The Emergence of Transformer-Driven Copilots
Prompt engineering
Copilots
Domain-specific GPT-3 engines
Transformer-based recommender systems
Computer vision
Humans and AI copilots in metaverses
Summary
Questions
References
17
The Consolidation of Suprahuman Transformers with OpenAI’s ChatGPT and GPT-4
The Consolidation of Suprahuman Transformers with OpenAI’s ChatGPT and GPT-4
Consolidating suprahuman NLP with ChatGPT and GPT-4 transformer models
Jump-starting the ChatGPT API
ChatGPT Plus writes and comments on a program
Getting started with the GPT-4 API
Advanced prompt engineering
Explainable AI (XAI)
Getting started with the DALL-E 2 API
Putting it all together
Summary
Questions
References
18
Other Books You May Enjoy
Other Books You May Enjoy
19
Index
Index
Appendix I — Terminology of Transformer Models
Appendix I — Terminology of Transformer Models
Stack
Sublayer
Attention heads
Appendix II — Hardware Constraints for Transformer Models
Appendix II — Hardware Constraints for Transformer Models
The Architecture and Scale of Transformers
Why GPUs are so special
GPUs are designed for parallel computing
GPUs are also designed for matrix multiplication
Implementing GPUs in code
Testing GPUs with Google Colab
Google Colab Free with a CPU
Google Colab Pro with a GPU
Appendix III — Generic Text Completion with GPT-2
Appendix III — Generic Text Completion with GPT-2
Step 1: Activating the GPU
Step 2: Cloning the OpenAI GPT-2 repository
Step 3: Installing the requirements
Step 4: Checking the version of TensorFlow
Step 5: Downloading the 345M-parameter GPT-2 model
Steps 6-7: Intermediate instructions
Steps 7b-8: Importing and defining the model
Step 9: Interacting with GPT-2
References
Appendix IV — Custom Text Completion with GPT-2
Appendix IV — Custom Text Completion with GPT-2
Training a GPT-2 language model
References
Appendix V — Answers to the Questions
Chapter 1, What are Transformers?
Chapter 2, Getting Started with the Architecture of the Transformer Model
Chapter 3, Fine-Tuning BERT Models
Chapter 4, Pretraining a RoBERTa Model from Scratch
Chapter 5, Downstream NLP Tasks with Transformers
Chapter 6, Machine Translation with the Transformer
Chapter 7, The Rise of Suprahuman Transformers with GPT-3 Engines
Chapter 8, Applying Transformers to Legal and Financial Documents for AI Text Summarization
Chapter 9, Matching Tokenizers and Datasets
Chapter 10, Semantic Role Labeling with BERT-Based Transformers
Chapter 11, Let Your Data Do the Talking: Story, Questions, and Answers
Chapter 12, Detecting Customer Emotions to Make Predictions
Chapter 13, Analyzing Fake News with Transformers
Chapter 14, Interpreting Black Box Transformer Models
Chapter 15, From NLP to Task-Agnostic Transformer Models
Chapter 16, The Emergence of Transformer-Driven Copilots
Chapter 17, The Consolidation of Suprahuman Transformers with OpenAI’s ChatGPT and GPT-4

Training and performance

The original Transformer was trained on a 4.5 million sentence pair English-German dataset and a 36 million sentence pair English-French dataset.

The datasets come from Workshops on Machine Translation (WMT), which can be found at the following link if you wish to explore the WMT datasets: http://www.statmt.org/wmt14/

The training of the original Transformer base models took 12 hours to train for 100,000 steps on a machine with 8 NVIDIA P100 GPUs. The big models took 3.5 days for 300,000 steps.

The original Transformer outperformed all the previous machine translation models with a BLEU score of 41.8. The result was obtained on the WMT English-to-French dataset.

BLEU stands for Bilingual Evaluation Understudy. It is an algorithm that evaluates the quality of the results of machine translations.

The Google Research and Google Brain team applied optimization strategies to improve the performance of the Transformer. For example, the Adam optimizer...

Previous Section
End of Section 3
Next Section