Book Image

What's New in TensorFlow 2.0

By : Ajay Baranwal, Alizishaan Khatri, Tanish Baranwal
Book Image

What's New in TensorFlow 2.0

By: Ajay Baranwal, Alizishaan Khatri, Tanish Baranwal

Overview of this book

TensorFlow is an end-to-end machine learning platform for experts as well as beginners, and its new version, TensorFlow 2.0 (TF 2.0), improves its simplicity and ease of use. This book will help you understand and utilize the latest TensorFlow features. What's New in TensorFlow 2.0 starts by focusing on advanced concepts such as the new TensorFlow Keras APIs, eager execution, and efficient distribution strategies that help you to run your machine learning models on multiple GPUs and TPUs. The book then takes you through the process of building data ingestion and training pipelines, and it provides recommendations and best practices for feeding data to models created using the new tf.keras API. You'll explore the process of building an inference pipeline using TF Serving and other multi-platform deployments before moving on to explore the newly released AIY, which is essentially do-it-yourself AI. This book delves into the core APIs to help you build unified convolutional and recurrent layers and use TensorBoard to visualize deep learning models using what-if analysis. By the end of the book, you'll have learned about compatibility between TF 2.0 and TF 1.x and be able to migrate to TF 2.0 smoothly.
Table of Contents (13 chapters)
Title Page

Rich extensions

Rich extensions are a set of features that have been introduced in TensorFlow to boost user productivity and expand capabilities. In this section, we will cover Ragged Tensors and how to use them, and, we will also cover the new modules introduced in TF 2.0.

Ragged Tensors

Variable-sized data is a common occurrence when both training and serving machine learning models. This issue is constant across the different underlying media types and model architectures. The contemporary solution is to use the size of the largest record, and use padding for smaller records. This is inefficient, not only in terms of memory or storage, but also computational efficiency; for example, when dealing with inputs to a recurrent model.

Ragged Tensors help address this issue. At a very high level, Ragged Tensors can be thought of as the TensorFlow analogs of variable-length linked lists. An important fact to note here is that this variability can be present in nested dimensions as well. This means that it is possible to have a list of variable-sized elements. Generalizing this property to multiple dimensions opens doors to a variety of interesting use cases. One of the important restrictions to keep in mind, though, is that all values in a Ragged Tensor must be of the same type. Some commonly non-uniform shaped data types that Ragged Tensors can be used for includes the following:

  • Variable-length features:
    • Example—the number of characters in a word
  • Batches of variable-length sequential inputs:
    • Example—sentences, time-series data, and audio clips
  • Hierarchical inputs:
    • Example—text documents that are subdivided into sections, paragraphs, sentences, words and characters; organizational hierarchies 
  • Individual fields in structured inputs:
    • Example—HTTP Request payloads, protocol buffers, and JSON data

In the following subsections, we shall look at the main properties of Ragged Tensors and write some code to see them in action.

What are Ragged Tensors, really?

Ragged Tensors can also be defined as tensors with one or more ragged dimensions; in other words, dimensions with variable-length slices. As most common use-cases involve dealing with a finite number of records, Ragged Tensors require the outermost dimension to be uniform, in other words, that all slices of that dimension should have the same length. Dimensions preceding the outermost dimension can be both ragged and uniform. To summarize these points, we can state that the shape of a Ragged Tensor is currently restricted to the following form:

  • A single uniform dimension
  • Followed by one or more ragged dimensions
  • Followed by zero or more uniform dimensions

Constructing a Ragged Tensor

TF 2.0 provides a large number of methods that can be used to create or return Ragged Tensors. One of the most straightforward ones is tf.ragged.constant(). Let's use it to create a Ragged Tensor of dimension (num_sentences, (num_words)). Please note that we've used round brackets to indicate the dimension that is ragged:

sentences = tf.ragged.constant([   
["Hello", "World", "!"],
["We", "are", "testing", "tf.ragged.constant", "."]

You should see something like this:

<tf.RaggedTensor [[b'Hello', b'World', b'!'], [b'We', b'are', b'testing', b'tf.ragged.constant', b'.']]>

It is also possible to create a Ragged Tensor from an old-style tensor or Python list with padded elements. This can be very useful in building efficient TF 2.0 models that consume data from a lower-stage pipeline written for earlier versions of TensorFlow. The functionality is exposed by the tf.RaggedTensor.from_tensor() function. The padding value is provided by the padding keyword argument. If used correctly, this can provide users with significant amounts of memory, especially in cases of sparse arrays.

Consider the following example in which we define a Python list. Each element of this list has a further list containing a variable number of numerical values. Some of the numbers listed here are padded values and are indicated by the digit 0. This can also be looked at as a matrix of 4 records containing 5 attributes each; in other words, a 4 x 5 matrix:

x = [
[1, 7, 0, 0, 0],
[2, 0, 0, 0, 0],
[4, 5, 8, 9, 1],
[1, 0, 0, 0, 0]
print(tf.RaggedTensor.from_tensor(x, padding=0))

We can see that a majority of records in the preceding matrix contain padding values. These values occupy memory. As seen in the following output, converting the preceding matrix to a Ragged Tensor eliminates the lagging 0 (padding) values. This results in a memory-efficient representation of the data:

<tf.RaggedTensor [[1, 7], [2], [4, 5, 8, 9, 1], [1]]>

The preceding example is a small illustration of how using ragged representations saves memory. As the number of records and/or dimensions grow, the memory savings provided by this representation would become more pronounced. 

Basic operations on Ragged Tensors

Ragged Tensors can be used in a manner similar to regular tensors in many cases. TensorFlow provides over 100 operators that support Ragged Tensors. These operators can be broadly classified as fundamental mathematical operators, array operators, or string operators, among others.

The following code block shows the process of adding two Ragged Tensors:

x = tf.ragged.constant([
[1, 2, 3, 4],
[1, 2]
y = tf.ragged.constant([
[4, 3, 2, 1],
[5, 6]
print(tf.add(x, y))

This results in the following output:

<tf.RaggedTensor [[5, 5, 5, 5], [6, 8]]>

Another interesting feature is that operator overloading is defined for Ragged Tensors. This means that a programmer can intuitively use operators such as +, -, *, /, //, %, **, &, |, ^, <, <=, >, and >=, just like they would with other tensors.

The following code block shows the multiplication of a Ragged Tensor using an overloaded operator:

x = tf.ragged.constant([
[1, 2, 3, 4],
[1, 2]
print(x * 2) # Multiply a ragged tensor with a scalar
print(x * x) # Multiply a ragged tensor with another ragged tensor

The resultant output is as follows:

<tf.RaggedTensor [[2, 4, 6, 8], [2, 4]]>
<tf.RaggedTensor [[1, 4, 9, 16], [1, 4]]>

In addition, a variety of Ragged Tensor-specific operators are defined in the tf.ragged package. It could be worthwhile to check out the documentation of the package to learn more. Please see the following links for detailed documentation on this:

New and important packages

The arrival of TF 2.0 also comes with the arrival of many more interesting and useful packages under TensorFlow that can be installed separately. Some of these packages include TensorFlow Datasets, TensorFlow Addons, TensorFlow Text, and TensorFlow Probability.

TensorFlow Datasets is a Python module that provides easy access to over 100 datasets, ranging from audio to natural language to images. These datasets can be easily downloaded and used in models via the following code:

import tensorflow_datasets as tfds
dataset = tfds.load(name="mnist", split=tfds.Split.TRAIN)
dataset = dataset.shuffle(1024).batch(32).prefetch(
Datasets taken from this library are objects, which means that all methods can be used to modify the base dataset. More details on the TensorFlow datasets module are in Chapter 3, Designing and Constructing Input Data Pipelines.

TensorFlow Addons (TF Addonsis another TensorFlow module. This module contains most of the tf.contrib module from TF 1.x, other than the methods that were moved into the main tf module. TF Addons contains many experimental and state-of-the-art layers, loss functions, initializers, and optimizers, all in the form of TF 2.0 objects. This means that APIs taken from TF Addons can be seamlessly incorporated into a normal tf.keras model without any extra changes.

TensorFlow Text is a very recent module, which adds NLP APIs to TF 2.0. This module includes methods such as sentence and word tokenization, among other popular techniques in the NLP field. Something to note is that this module is very new and so is subject to multiple changes in the API structure.

TensorFlow Probability is a module that adds APIs for probability calculations in TensorFlow. This module allows researchers and developers to take advantage of TensorFlow's optimized operations and computations in order to perform a multitude of probability-related tasks.

All the aforementioned packages can be installed using pip and by installing in the tensorflow-module format.