Rich extensions are a set of features that have been introduced in TensorFlow to boost user productivity and expand capabilities. In this section, we will cover Ragged Tensors and how to use them, and, we will also cover the new modules introduced in TF 2.0.

Variable-sized data is a common occurrence when both training and serving machine learning models. This issue is constant across the different underlying media types and model architectures. The contemporary solution is to use the size of the largest record, and use padding for smaller records. This is inefficient, not only in terms of memory or storage, but also computational efficiency; for example, when dealing with inputs to a recurrent model.

Ragged Tensors help address this issue. At a very high level, Ragged Tensors can be thought of as the TensorFlow analogs of variable-length linked lists. An important fact to note here is that this variability can be present in nested dimensions as well. This means that it is possible to have a list of variable-sized elements. Generalizing this property to multiple dimensions opens doors to a variety of interesting use cases. One of the important restrictions to keep in mind, though, is that all values in a Ragged Tensor must be of the same type. Some commonly non-uniform shaped data types that Ragged Tensors can be used for includes the following:

• Variable-length features:
  • Example—the number of characters in a word
• Batches of variable-length sequential inputs:
  • Example—sentences, time-series data, and audio clips
• Hierarchical inputs:
  • Example—text documents that are subdivided into sections, paragraphs, sentences, words and characters; organizational hierarchies 
• Individual fields in structured inputs:
  • Example—HTTP Request payloads, protocol buffers, and JSON data

In the following subsections, we shall look at the main properties of Ragged Tensors and write some code to see them in action.

Ragged Tensors can also be defined as tensors with one or more ragged dimensions; in other words, dimensions with variable-length slices. As most common use-cases involve dealing with a finite number of records, Ragged Tensors require the outermost dimension to be uniform, in other words, that all slices of that dimension should have the same length. Dimensions preceding the outermost dimension can be both ragged and uniform. To summarize these points, we can state that the shape of a Ragged Tensor is currently restricted to the following form:

• A single uniform dimension
• Followed by one or more ragged dimensions
• Followed by zero or more uniform dimensions

TF 2.0 provides a large number of methods that can be used to create or return Ragged Tensors. One of the most straightforward ones is tf.ragged.constant(). Let's use it to create a Ragged Tensor of dimension (num_sentences, (num_words)). Please note that we've used round brackets to indicate the dimension that is ragged:

sentences = tf.ragged.constant([   
                       ["Hello", "World", "!"],
                       ["We", "are", "testing", "tf.ragged.constant", "."]
            ])
print(sentences)

You should see something like this:

<tf.RaggedTensor [[b'Hello', b'World', b'!'], [b'We', b'are', b'testing', b'tf.ragged.constant', b'.']]>

It is also possible to create a Ragged Tensor from an old-style tensor or Python list with padded elements. This can be very useful in building efficient TF 2.0 models that consume data from a lower-stage pipeline written for earlier versions of TensorFlow. The functionality is exposed by the tf.RaggedTensor.from_tensor() function. The padding value is provided by the padding keyword argument. If used correctly, this can provide users with significant amounts of memory, especially in cases of sparse arrays.

Consider the following example in which we define a Python list. Each element of this list has a further list containing a variable number of numerical values. Some of the numbers listed here are padded values and are indicated by the digit 0. This can also be looked at as a matrix of 4 records containing 5 attributes each; in other words, a 4 x 5 matrix:

x = [
        [1, 7, 0, 0, 0],
        [2, 0, 0, 0, 0],
        [4, 5, 8, 9, 1],
        [1, 0, 0, 0, 0]
]
print(tf.RaggedTensor.from_tensor(x, padding=0))

We can see that a majority of records in the preceding matrix contain padding values. These values occupy memory. As seen in the following output, converting the preceding matrix to a Ragged Tensor eliminates the lagging 0 (padding) values. This results in a memory-efficient representation of the data:

<tf.RaggedTensor [[1, 7], [2], [4, 5, 8, 9, 1], [1]]>

The preceding example is a small illustration of how using ragged representations saves memory. As the number of records and/or dimensions grow, the memory savings provided by this representation would become more pronounced. 

Ragged Tensors can be used in a manner similar to regular tensors in many cases. TensorFlow provides over 100 operators that support Ragged Tensors. These operators can be broadly classified as fundamental mathematical operators, array operators, or string operators, among others.

The following code block shows the process of adding two Ragged Tensors:

x = tf.ragged.constant([
                         [1, 2, 3, 4],
                         [1, 2]
                       ])
y = tf.ragged.constant([
                         [4, 3, 2, 1],
                         [5, 6]
                       ])
print(tf.add(x, y))

This results in the following output:

<tf.RaggedTensor [[5, 5, 5, 5], [6, 8]]>

Another interesting feature is that operator overloading is defined for Ragged Tensors. This means that a programmer can intuitively use operators such as +, -, *, /, //, %, **, &, |, ^, <, <=, >, and >=, just like they would with other tensors.

The following code block shows the multiplication of a Ragged Tensor using an overloaded operator:

x = tf.ragged.constant([
                          [1, 2, 3, 4],
                          [1, 2]
                       ])
print(x * 2)  # Multiply a ragged tensor with a scalar
print(x * x)  # Multiply a ragged tensor with another ragged tensor

The resultant output is as follows:

<tf.RaggedTensor [[2, 4, 6, 8], [2, 4]]>
<tf.RaggedTensor [[1, 4, 9, 16], [1, 4]]>

In addition, a variety of Ragged Tensor-specific operators are defined in the tf.ragged package. It could be worthwhile to check out the documentation of the package to learn more. Please see the following links for detailed documentation on this:

• https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/RaggedTensor
• https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/ragged

The arrival of TF 2.0 also comes with the arrival of many more interesting and useful packages under TensorFlow that can be installed separately. Some of these packages include TensorFlow Datasets, TensorFlow Addons, TensorFlow Text, and TensorFlow Probability.

TensorFlow Datasets is a Python module that provides easy access to over 100 datasets, ranging from audio to natural language to images. These datasets can be easily downloaded and used in models via the following code:

import tensorflow_datasets as tfds
dataset = tfds.load(name="mnist", split=tfds.Split.TRAIN)
dataset = dataset.shuffle(1024).batch(32).prefetch(tf.data.experimental.AUTOTUNE)

TensorFlow Addons (TF Addons) is another TensorFlow module. This module contains most of the tf.contrib module from TF 1.x, other than the methods that were moved into the main tf module. TF Addons contains many experimental and state-of-the-art layers, loss functions, initializers, and optimizers, all in the form of TF 2.0 objects. This means that APIs taken from TF Addons can be seamlessly incorporated into a normal tf.keras model without any extra changes.

TensorFlow Text is a very recent module, which adds NLP APIs to TF 2.0. This module includes methods such as sentence and word tokenization, among other popular techniques in the NLP field. Something to note is that this module is very new and so is subject to multiple changes in the API structure.

TensorFlow Probability is a module that adds APIs for probability calculations in TensorFlow. This module allows researchers and developers to take advantage of TensorFlow's optimized operations and computations in order to perform a multitude of probability-related tasks.

All the aforementioned packages can be installed using pip and by installing in the tensorflow-module format.