DBNs are trained using a greedy algorithm where one layer is trained at a time; the RBMs are learned sequentially. A key concept surrounding this greedy approach is that it allows each model in the sequence to receive a different representation of the input data.
There are two phases to be considered during the training of a DBN, a positive phase and a negative phase:
- Positive phase: The first layer is trained with the data from the training dataset whilst all of the other layers are frozen. All of the individual activation probabilities for the first hidden layer are derived. This is referred to as the positive phase:
- Negative phase: During the negative phase the visible units are reconstructed in a similar fashion to the positive phase. From here, all of the associated weights are updated:
From here, the activations of the previously trained features are...