Classical approaches to learning word representation
In this section, we will discuss some of the classical approaches used for numerically representing words. It is important to have an understanding of the alternatives to word vectors, as these methods are still used in the real world, especially when limited data is available.
More specifically, we will discuss common representations, such as one-hot encoding and Term Frequency-Inverse Document Frequency (TF-IDF).
One-hot encoded representation
One of the simpler ways of representing words is to use the one-hot encoded representation. This means that if we have a vocabulary of size V, for each ith word wi, we will represent the word wi with a V-length vector [0, 0, 0, …, 0, 1, 0, …, 0, 0, 0] where the ith element is 1 and other elements are 0. As an example, consider this sentence:
Bob and Mary are good friends.
The one-hot encoded representation of each word might look like this:
Bob: [1...