Building the model

In this section, the core model-building exercise is illustrated. We first define a embedding layer for words in the vocabulary of the text captions followed by the two LSTMs. The weights self.encode_W and self.encode_b are used to reduce the dimension of the features f_t from the convolutional neural network. For the second LSTM (LSTM 2), one of the other inputs at any time step t > N is the previous word w_t-1 along with the output h_t from the first LSTM (LSTM 1). The word embedding for w_t-1 is fed to the LSTM 2 instead of the raw one hot encoded vector. For the first N (self.video_lstm_step), the LSTM 1 processes the input features f_t from the CNN, and the output hidden state ^h_t(output1) is fed to the LSTM 2. During this encoding phase, the LSTM 2 doesn't receive any word w_t-1 as an input.

From the (N+1) time step, we enter the decoding phase, where...