The Continuous Bag-of-Words algorithm
The CBOW model works in a similar way to the skip-gram algorithm, with one significant change in the problem formulation. In the skip-gram model, we predict the context words from the target word. However, in the CBOW model, we predict the target word from contextual words. Let’s compare what data looks like for the skip-gram algorithm and the CBOW model by taking the previous example sentence:
The dog barked at the mailman.
For the skip-gram algorithm, the data tuples—(input word, output word)—might look like this:
(dog, the), (dog, barked), (barked, dog), and so on
For CBOW, the data tuples would look like the following:
([the, barked], dog), ([dog, at], barked), and so on
Consequently, the input of the CBOW has a dimensionality of 2 × m × D, where m is the context window size and D is the dimensionality of the embeddings. The conceptual model of CBOW is shown in Figure 3.13:
Figure...