Since its launch, TensorFlow has been important for the data scientist community for being An Open Source Machine Learning Framework for Everyone. Magenta, which is based on TensorFlow, can be seen the same way: even if it's using state of the art machine learning techniques, it can still be used by anyone. Musicians and computer scientists alike can install it and generate new music in no time.

In this section, we'll look at the content of Magenta by introducing what it can and cannot do and refer to the chapters that explain the content in more depth.

Magenta is a framework for art generation, but also for attention, storytelling, and the evaluation of generative music. As the book advances, we'll come to see and understand how those elements are crucial for pleasing music generation.

We'll be using Magenta for two different purposes, assisting and autonomous music creation:

• Assisting music systems helps with the process of composing music. Examples of this would be the Magenta interface, magenta_midi.py, where the musician can enter a MIDI sequence and Magenta will answer with a generated sequence that's inspired by the provided one. These types of systems can be used alongside traditional systems to compose music and get new inspirations. We'll be talking about this in Chapter 9, Making Magenta Interact with Music Applications, where Magenta Studio can be integrated into a traditional music production tool.
• Autonomous music systems continuously produce music without the input of an operator. At the end of this book, you'll have all the tools you'll need to build an autonomous music generation system consisting of the various building blocks of Magenta.

Remembering what we saw in the previous section, there are many ways of representing music: symbolic data, spectrogram data, and raw audio data. Magenta works mainly with symbolic data, meaning we'll mainly work on the underlying score in music instead of working directly with audio. Let's look into Magenta's content, model by model.

In Magenta and in this book, the term model refers to a specific deep neural network that is specific for one task. For example, the Drums RNN model is an LSTM network with attention configuration, while the MusicVAE model is a variational autoencoder network. The Melody RNN model is also an LSTM network but is geared toward generating melodies instead of percussion patterns.

Each model has different configurations that will change how the data is encoded for the network, as well as how the network is configured. For example, the Drums RNN model has a one_drum configuration, which encodes the sequence to a single class, as well as a drum_kit configuration, which maps the sequence to nine drum instruments and also configures the attention length to 32.

Finally, each configuration comes with one or more pre-trained models. For example, Magenta provides a pre-trained Drums RNN drum_kit model, as well as multiple pre-trained MusicVAE cat-drums_2bar_small models.

We'll be using this terminology throughout this book. For the first few chapters, we'll be using the Magenta pre-trained models, since they are already quite powerful. After, we'll create our own configurations and train our own models.

Image generation and stylization can be achieved in Magenta with the Sketch RNN and Style Transfer models, respectively. Sketch-RNN is a Sequence-to-Sequence (Seq2Seq) variational autoencoder.

Seq2Seq models are used to convert sequences from one domain into another domain (for example, to translate a sentence in English to a sentence in French) that do not necessarily have the same length, which is not possible for a traditional model structure. The network will encode the input sequence into a vector, called a latent vector, from which a decoder will try to reproduce the input sequence as closely as possible.

Image processing is not part of this book, but we'll see the usage of latent space in Chapter 4, Latent Space Interpolation with MusicVAE, when we use the MusicVAE model. If you are interested in the SketchRNN model, see the Further reading section for more information.

Audio generation in Magenta is done with the NSynth, a WaveNet-based autoencoder, and GANSynth models. What's interesting about WaveNet is that it is a convolutional architecture, prevalent in image applications, but seldom used in music applications, in favor of recurrent networks. Convolutional neural networks (CNNs) are mainly defined by a convolution stage, in which a filter is slid through the image, computing a feature map of the image. Different filter matrices can be used to detect different features, such as edges or curves, which are useful for image classification.

We'll see the usage of these models in Chapter 5, Audio Generation with NSynth and GANSynth.

Score generation is the main part of Magenta and can be split into different categories representing the different parts of a musical score:

• Rhythms generation: This can be done with the "Drums RNN" model, an RNN network that applies language modeling using an LSTM. Drum tracks are polyphonic by definition because multiple drums can be hit simultaneously. This model will be presented in Chapter 2, Generating Drum Sequences with Drums RNN.
• Melody generation: Also known as monophonic generation, this can be done with the "Melody RNN" and "Improv RNN" models, which also implement the use of attention, allowing the models to learn longer dependencies. These models will be presented in Chapter 3, Generating Polyphonic Melodies.
• Polyphonic generation: This can be done with the Polyphony RNN and Performance RNN models, where the latter also implements expressive timing (sometimes called groove, where the notes don't start and stop exactly in the grid, giving it a human fell) and dynamics (or velocity). These models will be presented in Chapter 3, Generating Polyphonic Melodies.
• Interpolation: This can be done with the MusicVAE model, a variational autoencoder that learns the latent space of a musical sequence and can interpolate between existing sequences. This model will be presented in Chapter 4, Latent Space Interpolation with Music VAE.
• Transformation: This can be done with the GrooVAE model, a variant of the MusicVAE model that will add groove to an existing drum performance. This model will be presented in Chapter 4, Latent Space Interpolation with Music VAE.