As the name implies, RBMs originated from Boltzmann machines. Invented by Geoffrey Hinton and Paul Smolensky in 1983, Boltzmann machines are a type of network where all units (visible and hidden) are in a binary state and are connected together. Despite their theoretical capability of learning intriguing representations, there are many practical issues for them, including training time, which grows exponentially with the model size (as all units are connected). A general diagram of Boltzmann machines is depicted as follows:
To make it easier to learn a Boltzmann machine model, a connectivity restricted version called Harmonium was initially invented in 1986 by Paul Smolensky. In mid-2000, Geoffrey Hinton and other researchers invented a much more efficient architecture, which contains only one hidden layer and does not allow any internal connections...