Book Image

Natural Language Processing with Java and LingPipe Cookbook

Book Image

Natural Language Processing with Java and LingPipe Cookbook

Overview of this book

Related Content you might be interested in

Current Title:

Small book image
Natural Language Processing with Java and LingPipe Cookbook
Nov, 2014 | 312 pages
No titles found
Table of Contents (14 chapters)
Natural Language Processing with Java and LingPipe Cookbook
Natural Language Processing with Java and LingPipe Cookbook
Credits
Credits
About the Authors
About the Authors
About the Reviewers
About the Reviewers
www.PacktPub.com
www.PacktPub.com
Preface
Preface
Free Chapter
1
Simple Classifiers
Simple Classifiers
Introduction
Deserializing and running a classifier
Getting confidence estimates from a classifier
Getting data from the Twitter API
Applying a classifier to a .csv file
Evaluation of classifiers – the confusion matrix
Training your own language model classifier
How to train and evaluate with cross validation
Viewing error categories – false positives
Understanding precision and recall
How to serialize a LingPipe object – classifier example
Eliminate near duplicates with the Jaccard distance
How to classify sentiment – simple version
2
Finding and Working with Words
Finding and Working with Words
Introduction
Introduction to tokenizer factories – finding words in a character stream
Combining tokenizers – lowercase tokenizer
Combining tokenizers – stop word tokenizers
Using Lucene/Solr tokenizers
Using Lucene/Solr tokenizers with LingPipe
Evaluating tokenizers with unit tests
Modifying tokenizer factories
Finding words for languages without white spaces
3
Advanced Classifiers
Advanced Classifiers
Introduction
A simple classifier
Language model classifier with tokens
Naïve Bayes
Feature extractors
Logistic regression
Multithreaded cross validation
Tuning parameters in logistic regression
Customizing feature extraction
Combining feature extractors
Classifier-building life cycle
Linguistic tuning
Thresholding classifiers
Train a little, learn a little – active learning
Annotation
4
Tagging Words and Tokens
Tagging Words and Tokens
Introduction
Interesting phrase detection
Foreground- or background-driven interesting phrase detection
Hidden Markov Models (HMM) – part-of-speech
N-best word tagging
Confidence-based tagging
Training word tagging
Word-tagging evaluation
Conditional random fields (CRF) for word/token tagging
Modifying CRFs
5
Finding Spans in Text – Chunking
Finding Spans in Text – Chunking
Introduction
Sentence detection
Evaluation of sentence detection
Tuning sentence detection
Marking embedded chunks in a string – sentence chunk example
Paragraph detection
Simple noun phrases and verb phrases
Regular expression-based chunking for NER
Dictionary-based chunking for NER
Translating between word tagging and chunks – BIO codec
HMM-based NER
Mixing the NER sources
CRFs for chunking
NER using CRFs with better features
6
String Comparison and Clustering
String Comparison and Clustering
Introduction
Distance and proximity – simple edit distance
Weighted edit distance
The Jaccard distance
The Tf-Idf distance
Using edit distance and language models for spelling correction
The case restoring corrector
Automatic phrase completion
Single-link and complete-link clustering using edit distance
Latent Dirichlet allocation (LDA) for multitopic clustering
7
Finding Coreference Between Concepts/People
Finding Coreference Between Concepts/People
Introduction
Named entity coreference with a document
Adding pronouns to coreference
Cross-document coreference
The John Smith problem
Index
Index

Simple noun phrases and verb phrases

This recipe will show you how to find simple noun phrases (NP) and verb phrases (VP). By "simple", we mean that there is no complex structure within the phrases. For example, the complex NP "The rain in Spain" will be broken into two simple NP chunks "The rain" and "Spain". These phrases are also called "basal".

This recipe will not go into the details of how the basal NPs/VPs are calculated but rather how to use the class—it can come in handy, and the source can be included if you want to sort out how it works.

How to do it…

Like many of the recipes, we will provide a command-line-interactive interface here:

  1. Haul up the command line and type:

    java -cp lingpipe-cookbook.1.0.jar:lib/lingpipe-4.1.0.jar: com.lingpipe.cookbook.chapter5.PhraseChunker
INPUT> The rain in Spain falls mainly on the plain.
The/at rain/nn in/in Spain/np falls/vbz mainly/rb on/in the/at plain/jj ./. 
  noun(0,8) The rain
  noun(12,17) Spain
  verb(18,30) falls mainly
  noun(34,43...
Previous Section
End of Section 8
Next Section