Mastering Transformers

By : Savaş Yıldırım, Meysam Asgari- Chenaghlu

Mastering Transformers

By: Savaş Yıldırım, Meysam Asgari- Chenaghlu

Overview of this book

Transformer-based language models have dominated natural language processing (NLP) studies and have now become a new paradigm. With this book, you'll learn how to build various transformer-based NLP applications using the Python Transformers library. The book gives you an introduction to Transformers by showing you how to write your first hello-world program. You'll then learn how a tokenizer works and how to train your own tokenizer. As you advance, you'll explore the architecture of autoencoding models, such as BERT, and autoregressive models, such as GPT. You'll see how to train and fine-tune models for a variety of natural language understanding (NLU) and natural language generation (NLG) problems, including text classification, token classification, and text representation. This book also helps you to learn efficient models for challenging problems, such as long-context NLP tasks with limited computational capacity. You'll also work with multilingual and cross-lingual problems, optimize models by monitoring their performance, and discover how to deconstruct these models for interpretability and explainability. Finally, you'll be able to deploy your transformer models in a production environment. By the end of this NLP book, you'll have learned how to use Transformers to solve advanced NLP problems using advanced models.

Preface

Who this book is for

What this book covers

To get the most out of this book

Download the example code files

Code in Action

Download the color images

Conventions used

Get in touch

Share Your Thoughts

Section 1: Introduction – Recent Developments in the Field, Installations, and Hello World Applications

Free Chapter

Chapter 1: From Bag-of-Words to the Transformer

Technical requirements

Evolution of NLP toward Transformers

Understanding distributional semantics

Leveraging DL

Overview of the Transformer architecture

Using TL with Transformers

Summary

References

Chapter 2: A Hands-On Introduction to the Subject

Technical requirements

Installing Transformer with Anaconda

Working with language models and tokenizers

Working with community-provided models

Working with benchmarks and datasets

Benchmarking for speed and memory

Summary

Section 2: Transformer Models – From Autoencoding to Autoregressive Models

Chapter 3: Autoencoding Language Models

Technical requirements

BERT – one of the autoencoding language models

Autoencoding language model training for any language

Sharing models with the community

Understanding other autoencoding models

Working with tokenization algorithms

Summary

Chapter 4:Autoregressive and Other Language Models

Technical requirements

Working with AR language models

Working with Seq2Seq models

AR language model training

NLG using AR models

Summarization and MT fine-tuning using simpletransformers

Summary

References

Chapter 5: Fine-Tuning Language Models for Text Classification

Technical requirements

Introduction to text classification

Fine-tuning a BERT model for single-sentence binary classification

Training a classification model with native PyTorch

Fine-tuning BERT for multi-class classification with custom datasets

Fine-tuning the BERT model for sentence-pair regression

Utilizing run_glue.py to fine-tune the models

Summary

Chapter 6: Fine-Tuning Language Models for Token Classification

Technical requirements

Introduction to token classification

Fine-tuning language models for NER

Question answering using token classification

Summary

Chapter 7: Text Representation

Technical requirements

Introduction to sentence embeddings

Text clustering with Sentence-BERT

Semantic search with Sentence-BERT

Summary

Conventions used

There are a number of text conventions used throughout this book.

Code in text: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: "Sequences that are shorter than max_sen_len (maximum sentence length) are padded with a PAD value until they are max_sen_len in length."

A block of code is set as follows:

max_sen_len=max([len(s.split()) for s in sentences])
words = ["PAD"]+ list(set([w for s in sentences for w in s.split()]))
word2idx= {w:i for i,w in enumerate(words)}
max_words=max(word2idx.values())+1
idx2word= {i:w for i,w in enumerate(words)}
train=[list(map(lambda x:word2idx[x], s.split())) for s in sentences]

When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:

[default]
exten => s,1,Dial(Zap/1|30)
exten => s,2,Voicemail(u100)
exten => s,102,Voicemail(b100)
exten => i,1,Voicemail(s0)

Any command-line input or output is written as follows:

$ conda activate transformers
$ conda install -c conda-forge tensorflow

Bold: Indicates a new term, an important word, or words that you see onscreen. For instance, words in menus or dialog boxes appear in bold. Here is an example: "We must now take care of the computational cost of a particular model for a given environment (Random Access Memory (RAM), CPU, and GPU) in terms of memory usage and speed."

Tips or important notes

Appear like this.

Mastering Transformers

By : Savaş Yıldırım, Meysam Asgari- Chenaghlu

Mastering Transformers

By: Savaş Yıldırım, Meysam Asgari- Chenaghlu

Overview of this book

Related Content you might be interested in

Current Title:

Mastering Transformers

Getting Started with Google BERT

Transformers for Natural Language Processing

Transformers For Natural Language Processing

Conventions used