-
Book Overview & Buying
-
Table Of Contents
Generative AI with Python and PyTorch - Second Edition
By :
OpenAI has been in the spotlight for quite some time because of its newsworthy works, such as GPT8, GPT-29, and 3 (and also instructGPT, 3.5, and 4, along with viral sensation ChatGPT, but these are a bit different and covered in subsequent chapters). In this section, we will briefly discuss GPT architectures up to GPT-3. We will then use a pretrained version of GPT-2 for our text generation task.
The first model in this series is called GPT, or Generative Pretraining. It was released in 2018, about the same time as BERT. The paper presents a task-agnostic architecture based on the ideas of transformers and unsupervised learning. The GPT model was shown to beat several benchmarks, such as GLUE and SST-216, although its performance was overtaken by BERT, which was released shortly after this.
GPT is essentially a language model based on the transformer-decoder we presented previously. Since a language model can be trained...