The Analyzer class is an abstract base class containing two methods of interest. The first one is createComponents (String fieldname, Reader reader). This is where the analyzer is put together by chaining the tokenizer and filters. The second method is tokenStream (String fieldname, Reader reader). This is the method we will review in this section. We will use the tokenStream method to return a processed TokenStream so we can examine its content after the analysis process.

We need two arguments to call the tokenStream method. The first is a field name and the second is a reader:

Reader reader = new StringReader("Text to be passed");
Analyzer analyzer ...