Book Image

Mastering Transformers

By : Savaş Yıldırım, Meysam Asgari- Chenaghlu
Book Image

Mastering Transformers

By: Savaş Yıldırım, Meysam Asgari- Chenaghlu

Overview of this book

Transformer-based language models have dominated natural language processing (NLP) studies and have now become a new paradigm. With this book, you'll learn how to build various transformer-based NLP applications using the Python Transformers library. The book gives you an introduction to Transformers by showing you how to write your first hello-world program. You'll then learn how a tokenizer works and how to train your own tokenizer. As you advance, you'll explore the architecture of autoencoding models, such as BERT, and autoregressive models, such as GPT. You'll see how to train and fine-tune models for a variety of natural language understanding (NLU) and natural language generation (NLG) problems, including text classification, token classification, and text representation. This book also helps you to learn efficient models for challenging problems, such as long-context NLP tasks with limited computational capacity. You'll also work with multilingual and cross-lingual problems, optimize models by monitoring their performance, and discover how to deconstruct these models for interpretability and explainability. Finally, you'll be able to deploy your transformer models in a production environment. By the end of this NLP book, you'll have learned how to use Transformers to solve advanced NLP problems using advanced models.

Related Content you might be interested in

Current Title:

Small book image
Mastering Transformers
Savaş Yıldırım | Meysam Asgari- Chenaghlu
Sep, 2021 | 374 pages
Small book image
Getting Started with Google BERT
Sudharsan Ravichandiran
Jan, 2021 | 352 pages
Small book image
Transformers for Natural Language Processing
Denis Rothman
Jan, 2021 | 384 pages
Small book image
Transformers For Natural Language Processing
Denis Rothman
Mar, 2022 | 602 pages
Table of Contents (16 chapters)
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Download the example code files
Code in Action
Download the color images
Conventions used
Get in touch
Share Your Thoughts
1
Section 1: Introduction – Recent Developments in the Field, Installations, and Hello World Applications
Section 1: Introduction – Recent Developments in the Field, Installations, and Hello World Applications
Free Chapter
2
Chapter 1: From Bag-of-Words to the Transformer
Chapter 1: From Bag-of-Words to the Transformer
Technical requirements
Evolution of NLP toward Transformers
Understanding distributional semantics
Leveraging DL
Overview of the Transformer architecture
Using TL with Transformers
Summary
References
3
Chapter 2: A Hands-On Introduction to the Subject
Chapter 2: A Hands-On Introduction to the Subject
Technical requirements
Installing Transformer with Anaconda
Working with language models and tokenizers
Working with community-provided models
Working with benchmarks and datasets
Benchmarking for speed and memory
Summary
4
Section 2: Transformer Models – From Autoencoding to Autoregressive Models
Section 2: Transformer Models – From Autoencoding to Autoregressive Models
5
Chapter 3: Autoencoding Language Models
Chapter 3: Autoencoding Language Models
Technical requirements
BERT – one of the autoencoding language models
Autoencoding language model training for any language
Sharing models with the community
Understanding other autoencoding models
Working with tokenization algorithms
Summary
6
Chapter 4:Autoregressive and Other Language Models
Chapter 4:Autoregressive and Other Language Models
Technical requirements
Working with AR language models
Working with Seq2Seq models
AR language model training
NLG using AR models
Summarization and MT fine-tuning using simpletransformers
Summary
References
7
Chapter 5: Fine-Tuning Language Models for Text Classification
Chapter 5: Fine-Tuning Language Models for Text Classification
Technical requirements
Introduction to text classification
Fine-tuning a BERT model for single-sentence binary classification
Training a classification model with native PyTorch
Fine-tuning BERT for multi-class classification with custom datasets
Fine-tuning the BERT model for sentence-pair regression
Utilizing run_glue.py to fine-tune the models
Summary
8
Chapter 6: Fine-Tuning Language Models for Token Classification
Chapter 6: Fine-Tuning Language Models for Token Classification
Technical requirements
Introduction to token classification
Fine-tuning language models for NER
Question answering using token classification
Summary
9
Chapter 7: Text Representation
Chapter 7: Text Representation
Technical requirements
Introduction to sentence embeddings
Semantic similarity experiment with FLAIR
Text clustering with Sentence-BERT
Semantic search with Sentence-BERT
Summary
Further reading
10
Section 3: Advanced Topics
Section 3: Advanced Topics
11
Chapter 8: Working with Efficient Transformers
Chapter 8: Working with Efficient Transformers
Technical requirements
Introduction to efficient, light, and fast transformers
Implementation for model size reduction
Working with efficient self-attention
Summary
References
12
Chapter 9:Cross-Lingual and Multilingual Language Modeling
Chapter 9:Cross-Lingual and Multilingual Language Modeling
Technical requirements
Translation language modeling and cross-lingual knowledge sharing
XLM and mBERT
Cross-lingual similarity tasks
Cross-lingual classification
Cross-lingual zero-shot learning
Fundamental limitations of multilingual models
Summary
References
13
Chapter 10: Serving Transformer Models
Chapter 10: Serving Transformer Models
Technical requirements
fastAPI Transformer model serving
Dockerizing APIs
Faster Transformer model serving using TFX
Load testing using Locust
Summary
References
14
Chapter 11: Attention Visualization and Experiment Tracking
Chapter 11: Attention Visualization and Experiment Tracking
Technical requirements
Interpreting attention heads
Tracking model metrics
Summary
References
Why subscribe?
15
Other Books You May Enjoy
Other Books You May Enjoy
Leave a review - let other readers know what you think
Share Your Thoughts

Chapter 1: From Bag-of-Words to the Transformer

In this chapter, we will discuss what has changed in Natural Language Processing (NLP) over two decades. We experienced different paradigms and finally entered the era of Transformer architectures. All the paradigms help us to gain a better representation of words and documents for problem-solving. Distributional semantics describes the meaning of a word or a document with vectorial representation, looking at distributional evidence in a collection of articles. Vectors are used to solve many problems in both supervised and unsupervised pipelines. For language-generation problems, n-gram language models have been leveraged as a traditional approach for years. However, these traditional approaches have many weaknesses that we will discuss throughout the chapter.

We will further discuss classical Deep Learning (DL) architectures such as Recurrent Neural Networks (RNNs), Feed-Forward Neural Networks (FFNNs), and Convolutional Neural Networks (CNNs). These have improved the performance of the problems in the field and have overcome the limitation of traditional approaches. However, these models have had their own problems too. Recently, Transformer models have gained immense interest because of their effectiveness in all NLP tasks, from text classification to text generation. However, the main success has been that Transformers effectively improve the performance of multilingual and multi-task NLP problems, as well as monolingual and single tasks. These contributions have made Transfer Learning (TL) more possible in NLP, which aims to make models reusable for different tasks or different languages.

Starting with the attention mechanism, we will briefly discuss the Transformer architecture and the differences between previous NLP models. In parallel with theoretical discussions, we will show practical examples with the popular NLP framework. For the sake of simplicity, we will choose introductory code examples that are as short as possible.

In this chapter, we will cover the following topics:

  • Evolution of NLP toward Transformers
  • Understanding distributional semantics
  • Leveraging DL
  • Overview of the Transformer architecture
  • Using TL with Transformers
Previous Section
End of Section 1
Next Section