As usual, now we need to clean up the data and put it into a shape that we can work with. The news article dataset particularly will require some attention, so let's turn our attention to it first.
The OANC is published in an XML format that includes a lot of information and annotations about the data. Specifically, this marks off:
Sections and chapters
Sentences
Words with part-of-speech lemma
Noun chunks
Verb chunks
Named entities
However, we want the option to use raw text later when the system is actually being used. Because of that, we will ignore the annotations and just extract the raw tokens. In fact, all we're really interested in is each document's text—either as a raw string or a feature vector—and the date it was published. Let's create a record type for this.
We'll put this into the types.clj file in src/financial/. Put this simple namespace header into the file:
(ns financial.types)
This data record will be similarly simple. It can...