We are now going to implement the Naive Bayes algorithm using mrjob, allowing it to process our dataset. Technically our version will be a reduced version of most Naive Bayes' implementations, without many of the features that you would expect like smoothing small values.
The mrjob package allows us to create MapReduce jobs that can easily be computed on Amazon's infrastructure. While mrjob sounds like a sedulous addition to the Mr. Men series of children's books, it stands for Map Reduce Job.
Note
You can install mrjob using the following: pip install
mrjob
I had to install the filechunkio package separately using
conda install -c conda-forge filechunkio
, but this will depend on your system setup. There are other Anaconda channels for installing mrjob, check them with:anaconda search -t conda mrjob
In essence, mrjob
provides the standard functionality that most MapReduce jobs need. Its most amazing feature is that you can write the same code, test on...