Visualizing Attention patterns
Remember that we specifically defined a model called attention_visualizer to generate attention matrices? With the model trained, we can now look at these attention patterns by feeding data to the model. Here’s how the model was defined:
attention_visualizer = tf.keras.models.Model(inputs=[encoder.inputs, decoder_input], outputs=[attn_weights, decoder_out])
We’ll also define a function to get the processed attention matrix along with label data that we can use directly for visualization purposes:
def get_attention_matrix_for_sampled_data(attention_model, target_lookup_layer, test_xy, n_samples=5):
test_x, test_y = test_xy
rand_ids = np.random.randint(0, len(test_xy[0]),
size=(n_samples,))
results = []
for rid in rand_ids:
en_input = test_x[rid:rid+1]
de_input = test_y[rid:rid+1,:-1]
attn_weights, predictions = attention_model.predict([en_input...