Why generative models?

Now that we have reviewed what generative models are and defined them more formally in the language of probability, why would we have a need for such models in the first place? What value do they provide in practical applications? To answer this question, let us take a brief tour of the topics that we will cover in more detail in the rest of this book.

The promise of deep learning

As noted previously, many of the models we will survey in the book are deep, multi-level neural networks. The last 15 years have seen a renaissance in the development of deep learning models for image classification, natural language processing (NLP) and understanding, and reinforcement learning. These advances were enabled by breakthroughs in traditional challenges in tuning and optimizing very complex models, combined with access to larger datasets, distributed computational power in the cloud, and frameworks such as PyTorch, which make it easier to prototype and reproduce research. We will also lay the theoretical groundwork for the components used in models in the rest of the book, by providing an overview of neural network architectures, optimizers, and regularization in Chapter 2.

Generating images

A challenge to generating images—such as the Théâtre D’opéra Spatial—is that, frequently, images have no labels (such as a digit); rather, we want to map the space of random numbers into a set of artificial images using a latent vector Z, as we described earlier in the chapter. A further constraint is that we want to promote the diversity of these images—if we input numbers within a certain range, we would like to know that they generate different outputs, and be able to tune the resulting image features. For this purpose, VAEs²³—a kind of deep neural network model that learns to encode images as a latent variable Z, which it decodes into the input image—were developed to generate diverse and photorealistic images (Figure 1.4), which we will cover in Chapter 3.

Figure 1.4: Sample images from a VAE^{24, 25}

In the context of image classification tasks, being able to generate new images can help us increase the number of examples in an existing dataset, or reduce the bias if our existing dataset is heavily skewed toward a particular kind of photograph. Applications could include generating alternative poses (angles, shades, and perspective shots) for product photographs on a fashion e-commerce website (Figure 1.5).

Figure 1.5: Simulating alternative poses with deep generative models²⁶

In a similar application, 2D images of automotive designs can be translated into 3D models using generative AI methods³⁹.

Data augmentation

Another powerful use case for generative models is to augment the limitations of small existing datasets with additional examples. These additional examples can help improve the quality of discriminate models trained from this expanded dataset by improving their generalization abilities. This augmented data can be used for semi-supervised learning; an initial discriminative model is trained using the real limited data. That model is then used to generate labels for the synthetic data, augmenting the dataset. Finally, a second discriminate model is trained using the combined real and synthetic datasets. Examples of these kinds of applications include increasing the number of diagnostic examples in medical image datasets for cancer and bone lesions^{37, 38}.

Style transfer and image transformation

In addition to mapping artificial images to a space of random numbers, we could also use generative models to learn a mapping between one kind of image and a second. This kind of model can, for example, be used to convert an image of a horse into that of a zebra (Figure 1.6²⁷), transform a photo into a painting, or create “deepfake videos,” in which one actor’s face has been replaced with another’s (Figure 1.2).

Figure 1.6: CycleGANs apply stripes to horses to generate zebras²⁷

Another fascinating example of applying generative modeling is a study in which a lost masterpiece of the artist Pablo Picasso was discovered to have been painted over with another image. After X-ray imaging of The Old Guitarist and The Crouching Beggar indicated that earlier images of a woman and a landscape lay underneath (Figure 1.7), researchers used the other paintings from Picasso’s “blue period” or other color photographs to train a “neural style transfer” model that transforms black and white images (the X-ray radiographs of the overlying paintings) to the coloration of the original artwork. Then, applying this transfer model to the “hidden” images allowed them to reconstruct “colored-in” versions of the lost paintings.

Figure 1.7: The Picasso paintings The Old Guitarist (top) and The Crouching Beggar (bottom) hid older paintings that were recovered using deep learning to color in the X-ray image of the painted-over scenes (middle) with color patterns learned from examples (column d), generating colorized versions of the lost art (far right)²⁸

All of these models use the previously mentioned GANs, a type of deep learning model proposed in 2014²⁹. In addition to changing the contents of an image (as in the preceding zebra example), these models can also be used to map one image into another, such as paired images (dogs and humans with similar facial features, shown in Figure 1.8), or generate textual descriptions from images (Figure 1.9).

Figure 1.8: Sim-GAN for mapping human to animal or anime faces³⁰

Figure 1.9: GAN for generating descriptions from images³¹

We could also condition the properties of the generated images on some auxiliary information such as labels, an approach used in the GANGogh algorithm, which synthesizes images in the style of different artists by supplying the desired artist as input to the generative model. We will describe these applications in Chapters 4 and 6. Generative AI is also enabling programmers to become artists through models such as Stable Diffusion, which translates natural language descriptions of an image into a visual rendering (Figure 1.10)—we’ll cover how it does this in Chapter 7 and try to reproduce Théâtre D’opéra Spatial.

Figure 1.10: Stable Diffusion examples³²

Fake news and chatbots

Humans have always wanted to talk to machines; the first chatbot, ELIZA³³, was written at MIT in the 1960s and used a simple program to transform a user’s input and generate a response, in the mode of a “therapist” who frequently responds in the form of a question. More sophisticated models can generate entirely novel text, such as Google’s BERT³⁴ and GPT-2¹¹, which use a unit called a “transformer” to generate new words based on past words in a body of text. A transformer module in a neural network allows a network to propose a new word in the context of preceding words in a piece of text, emphasizing those that are more relevant in order to generate plausible stretches of language. The BERT model then combines transformer units into a powerful multi-dimensional encoding of natural language patterns and contextual significance. This approach can be used in document creation for NLP tasks, or for chatbot dialogue systems (Figure 1.3), which we will cover in Chapters 8 and 9.

Increasingly powerful LLMs have demonstrated remarkable performance in language generation, creative writing, and authoring novel code. In Chapters 10 and 11, we’ll cover some of the most important general, or “foundational,” models that can be tuned for specific tasks after being trained on large sets of diverse language data. These include both closed-source (ChatGPT) and openly available (Llama) models (Figure 1.11).

To adapt these models to specific problems, we will apply methods such as prompt engineering (Chapter 12), fine-tuning, and RAG (Chapter 14). We’ll do so using common tools in this ecosystem such as LangChain and the Hugging Face Pipelines library, which are the topic of Chapter 13.

Figure 1.11: LLM examples—GPT-4 (top) and Llama2 (bottom)^{35, 36}