We will train a model using OpenNLP, which can be used to perform lemmatization. The actual process of performing lemmatization is illustrated in the following recipe, Determining the lexical meaning of a word using OpenNLP.

The most straightforward technique to train a model is to use the OpenNLP command-line tools. Download these tools from the OpenNLP page at https://opennlp.apache.org/download.html. We will not need the source code for these tools, so download the file named apache-opennlp-1.9.0-bin.tar.gz. Selecting that file will take you to a page that lists mirror sites for the file. Choose one that will work best for your location.

Once the file has been saved, expand the file. This will extract a .tar file. Next, expand this file, which will create a directory called apache-opennlp-1.9.0. In its bin subdirectory, you will find the tools that we need.

We will need training data for the training process. We will use the en-lemmatizer.dict file found at https://raw.githubusercontent.com/richardwilly98/elasticsearch-opennlp-auto-tagging/master/src/main/resources/models/en-lemmatizer.dict. Use a browser to open this page and then save this page using the file name en-lemmatizer.dict.

Let's go through the following steps:

1. Open a command-line window. We used the Window's cmd program in this example
2. Set up a path for the OpenNLP tool's bin directory and then navigate to the directory containing the en-lemmatizer.dict file.
3. Execute the following command:

opennlp LemmatizerTrainerME -model en-lemmatizer.bin -lang en -data en-lemmatizer.dict -encoding UTF-8

You will get the following output. It has been shortened here to save space:

Indexing events with TwoPass using cutoff of 5
        Computing event counts...  done. 301403 events        Indexing...  done.

Sorting and merging events... done. Reduced 301403 events to 297777.
Done indexing in 9.09 s.

Incorporating indexed data for training...
done.
   Number of Event Tokens: 297777
   Number of Outcomes: 432
   Number of Predicates: 69122
...done.

Computing model parameters ...
Performing 100 iterations.
  1:  ... loglikelihood=-1829041.6775780176     3.317817009120679E-6
  2:  ... loglikelihood=-452333.43760414346     0.876829361353404
  3:  ... loglikelihood=-211099.05280473927     0.9506806501594212
  4:  ... loglikelihood=-132195.3981804198      0.9667554735686108
...
 98:  ... loglikelihood=-6702.5821153954375     0.9988420818638168
 99:  ... loglikelihood=-6652.6134177562335     0.998845399680826
100:  ... loglikelihood=-6603.518040975329      0.9988553531318534

Writing lemmatizer model
... done (1.274s)
Wrote lemmatizer model to
path: C:\Downloads\OpenNLP\en-lemmatizer.bin

Execution time: 275.369 seconds

To understand the output, we need to explain the following command:

opennlp LemmatizerTrainerME -model en-lemmatizer.bin -lang en -data en-lemmatizer.dict -encoding UTF-8

The opennlp command is used with a number of OpenNLP tools. The tool to be used is specified by the command's first argument. In this example, we used the LemmatizerTrainerME tool. The arguments that follow control how the training process works. The LemmatizerTrainerME arguments are documented at https://opennlp.apache.org/docs/1.9.0/manual/opennlp.html#tools.cli.lemmatizer.LemmatizerTrainerME.

We use the -model, -lang, -data, and -encoding arguments, as detailed in the following list:

• The -model argument specifies the name of the model output file. This is the file that holds the trained model that we will use in the next recipe.
• The -lang argument specifies the natural language used. In this case, we use en, which indicates the training data is English.
• The -data argument specifies the file containing the training data. We used the en-lemmatizer.dict file.
• The -encoding parameter specifies the character set used by the training data. We used UTF-8, which indicates the data is Unicode data.

The output shows the training process. It displays various statistics, such as the number of passes and iterations performed. During each iteration, the probability increases, as shown in the following code. With the 100^th iteration, the probability approaches 100.

Performing 100 iterations:

  1:  ... loglikelihood=-1829041.6775780176     3.317817009120679E-6
  2:  ... loglikelihood=-452333.43760414346     0.876829361353404
  3:  ... loglikelihood=-211099.05280473927     0.9506806501594212
  4:  ... loglikelihood=-132195.3981804198      0.9667554735686108
...
 98:  ... loglikelihood=-6702.5821153954375     0.9988420818638168
 99:  ... loglikelihood=-6652.6134177562335     0.998845399680826
100:  ... loglikelihood=-6603.518040975329      0.9988553531318534
Writing lemmatizer model ... done (1.274s)

The final part of the output shows where the file is written. We wrote the lemmatizer model to the path :\Downloads\OpenNLP\en-lemmatizer.bin.

If you have specialized lemmatization needs, then you will need to create a training file. The training data file consists of a series of lines. Each line consists of three entries separated by spaces. The first entry contains a word. The second entry is the POS tag for the word. The third entry is the lemma for the word.

For example, in en-lemmatizer.dict, there are several lines for variations of the word bump, as shown in the following code:

bump    NN         bump
bump    VB          bump
bump    VBP        bump
bumped               VBD       bump
bumped               VBN       bump
bumper JJ             bumper
bumper NN         bumper

As you can see, a word may be used in different contexts and with different suffixes. Other datasets can be used for training. These include the Penn Treebank (https://web.archive.org/web/19970614160127/http://www.cis.upenn.edu/~treebank/) and the CoNLL 2009 datasets (https://www.ldc.upenn.edu/).

Training parameters other than the default parameters can be specified depending on the needs of the problem.

In the next recipe, Determining the lexical meaning of a word using OpenNLP, we will use the model to develop and determine the lexical meaning of a word.

• The OpenNLP API can be found at https://opennlp.apache.org/docs/1.9.0/apidocs/opennlp-tools/index.html