3. Advanced Classifiers | Natural Language Processing with Java and LingPipe Cookbook

Sign In Start Free Trial

Book Overview & Buying
Table Of Contents

Natural Language Processing with Java and LingPipe Cookbook

By : Krishna Dayanidhi

4.8 (4)

Natural Language Processing with Java and LingPipe Cookbook

4.8 (4)

By: Krishna Dayanidhi

Overview of this book

This book is for experienced Java developers with NLP needs, whether academics, industrialists, or hobbyists. A basic knowledge of NLP terminology will be beneficial.

Preface

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Free Chapter

1. Simple Classifiers

1. Simple Classifiers

Introduction

Deserializing and running a classifier

Getting confidence estimates from a classifier

Getting data from the Twitter API

Applying a classifier to a .csv file

Evaluation of classifiers – the confusion matrix

Training your own language model classifier

How to train and evaluate with cross validation

Viewing error categories – false positives

Understanding precision and recall

How to serialize a LingPipe object – classifier example

Eliminate near duplicates with the Jaccard distance

How to classify sentiment – simple version

2. Finding and Working with Words

2. Finding and Working with Words

Introduction

Introduction to tokenizer factories – finding words in a character stream

Combining tokenizers – lowercase tokenizer

Combining tokenizers – stop word tokenizers

Using Lucene/Solr tokenizers

Using Lucene/Solr tokenizers with LingPipe

Evaluating tokenizers with unit tests

Modifying tokenizer factories

Finding words for languages without white spaces

3. Advanced Classifiers

3. Advanced Classifiers

Introduction

A simple classifier

Language model classifier with tokens

Naïve Bayes

Feature extractors

Logistic regression

Multithreaded cross validation

Tuning parameters in logistic regression

Customizing feature extraction

Combining feature extractors

Classifier-building life cycle

Linguistic tuning

Thresholding classifiers

Train a little, learn a little – active learning

Annotation

4. Tagging Words and Tokens

4. Tagging Words and Tokens

Introduction

Interesting phrase detection

Foreground- or background-driven interesting phrase detection

Hidden Markov Models (HMM) – part-of-speech

N-best word tagging

Confidence-based tagging

Training word tagging

Word-tagging evaluation

Conditional random fields (CRF) for word/token tagging

Modifying CRFs

5. Finding Spans in Text – Chunking

5. Finding Spans in Text – Chunking

Introduction

Sentence detection

Evaluation of sentence detection

Tuning sentence detection

Marking embedded chunks in a string – sentence chunk example

Paragraph detection

Simple noun phrases and verb phrases

Regular expression-based chunking for NER

Dictionary-based chunking for NER

Translating between word tagging and chunks – BIO codec

HMM-based NER

Mixing the NER sources

CRFs for chunking

NER using CRFs with better features

6. String Comparison and Clustering

6. String Comparison and Clustering

Introduction

Distance and proximity – simple edit distance

Weighted edit distance

The Jaccard distance

The Tf-Idf distance

Using edit distance and language models for spelling correction

The case restoring corrector

Automatic phrase completion

Single-link and complete-link clustering using edit distance

Latent Dirichlet allocation (LDA) for multitopic clustering

7. Finding Coreference Between Concepts/People

7. Finding Coreference Between Concepts/People

Introduction

Named entity coreference with a document

Adding pronouns to coreference

Cross-document coreference

The John Smith problem

Index

Index

Customizing feature extraction

Logistic regression allows for arbitrary features to be used. Features are any observations that can be made about data being classified. Some examples are as follows:

Words/tokens from the text.
We found that character ngrams work very well in lieu of words or stemmed words. For small data sets of less than 10,000 words of training, we will use 2-4 grams. Bigger training data can merit a longer gram, but we have never had good results above 8-gram characters.
Output from another component can be a feature, for example, a part-of-speech tagger.
Metadata known about the text, for example, the location of a tweet or time of the day it was created.
Recognition of dates and numbers abstracted from the actual value.

How to do it…

The source for this recipe is in src/com/lingpipe/cookbook/chapter3/ContainsNumberFeatureExtractor.java.

Feature extractors are straightforward to build. The following is a feature extractor that returns a CONTAINS_NUMBER feature with weight 1...

CONTINUE READING

83

Tech Concepts

36

Programming languages

73

Tech Tools

Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.

Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.

50+ new titles added per month and exclusive early access to books as they are being written.

Natural Language Processing with Java and LingPipe Cookbook

Search

Your notes and bookmarks