Let's continue with the IMDb data and put into practice the ideas from the previous sections. In this section, we will use a few familiar packages, like tidytext, plyr and dplyr, as well as the excellent text2vec by Dimitriy Selivanov, which was released in 2017, and the well-known caret package by Max Kuhn.
We need to prepare our data for the algorithm.
First, a few imports that will be necessary:
library(plyr) library(dplyr) library(text2vec) library(tidytext) library(caret)
We will use the IMDb data as before:
imdb <- read.csv("./data/labeledTrainData.tsv", encoding = "utf-8", quote = "", sep="\t", stringsAsFactors = F)And create an iterator over the tokens:
tokens <- space_tokenizer(imdb$review) token_iterator <- itoken(tokens)
The tokens are simple words, also known as unigrams. This constitutes our vocabulary:
vocab <- create_vocabulary(token_iterator)
It's important for the co-occurrence matrix to include only words that appear...