7. Finding Coreference Between Concepts/People | Natural Language Processing with Java and LingPipe Cookbook

Sign In Start Free Trial

Book Overview & Buying
Table Of Contents

Natural Language Processing with Java and LingPipe Cookbook

By : Krishna Dayanidhi

4.8 (4)

Natural Language Processing with Java and LingPipe Cookbook

4.8 (4)

By: Krishna Dayanidhi

Overview of this book

This book is for experienced Java developers with NLP needs, whether academics, industrialists, or hobbyists. A basic knowledge of NLP terminology will be beneficial.

Preface

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Free Chapter

1. Simple Classifiers

1. Simple Classifiers

Introduction

Deserializing and running a classifier

Getting confidence estimates from a classifier

Getting data from the Twitter API

Applying a classifier to a .csv file

Evaluation of classifiers – the confusion matrix

Training your own language model classifier

How to train and evaluate with cross validation

Viewing error categories – false positives

Understanding precision and recall

How to serialize a LingPipe object – classifier example

Eliminate near duplicates with the Jaccard distance

How to classify sentiment – simple version

2. Finding and Working with Words

2. Finding and Working with Words

Introduction

Introduction to tokenizer factories – finding words in a character stream

Combining tokenizers – lowercase tokenizer

Combining tokenizers – stop word tokenizers

Using Lucene/Solr tokenizers

Using Lucene/Solr tokenizers with LingPipe

Evaluating tokenizers with unit tests

Modifying tokenizer factories

Finding words for languages without white spaces

3. Advanced Classifiers

3. Advanced Classifiers

Introduction

A simple classifier

Language model classifier with tokens

Naïve Bayes

Feature extractors

Logistic regression

Multithreaded cross validation

Tuning parameters in logistic regression

Customizing feature extraction

Combining feature extractors

Classifier-building life cycle

Linguistic tuning

Thresholding classifiers

Train a little, learn a little – active learning

Annotation

4. Tagging Words and Tokens

4. Tagging Words and Tokens

Introduction

Interesting phrase detection

Foreground- or background-driven interesting phrase detection

Hidden Markov Models (HMM) – part-of-speech

N-best word tagging

Confidence-based tagging

Training word tagging

Word-tagging evaluation

Conditional random fields (CRF) for word/token tagging

Modifying CRFs

5. Finding Spans in Text – Chunking

5. Finding Spans in Text – Chunking

Introduction

Sentence detection

Evaluation of sentence detection

Tuning sentence detection

Marking embedded chunks in a string – sentence chunk example

Paragraph detection

Simple noun phrases and verb phrases

Regular expression-based chunking for NER

Dictionary-based chunking for NER

Translating between word tagging and chunks – BIO codec

HMM-based NER

Mixing the NER sources

CRFs for chunking

NER using CRFs with better features

6. String Comparison and Clustering

6. String Comparison and Clustering

Introduction

Distance and proximity – simple edit distance

Weighted edit distance

The Jaccard distance

The Tf-Idf distance

Using edit distance and language models for spelling correction

The case restoring corrector

Automatic phrase completion

Single-link and complete-link clustering using edit distance

Latent Dirichlet allocation (LDA) for multitopic clustering

7. Finding Coreference Between Concepts/People

7. Finding Coreference Between Concepts/People

Introduction

Named entity coreference with a document

Adding pronouns to coreference

Cross-document coreference

The John Smith problem

Index

Index

Adding pronouns to coreference

The preceding recipe handled coreference between named entities. This recipe will add pronouns to the mix.

How to do it…

This recipe will use an interactive version to help you explore the properties of the coreference algorithm. The system is very dependent on the quality of the named-entity detection, so use examples that the HMM is likely to get right. This was trained on Wall Street Journal articles from the '90s.

Saddle up your console and type the following command:

java -cp lingpipe-cookbook.1.0.jar:lib/lingpipe-4.1.0.jar: com.lingpipe.cookbook.chapter7.Coreference

In the resulting command prompt, type this:

Enter text followed by new line
>John Smith went to Washington. He was a senator.
Sentence Text=John Smith went to Washington.
mention text=John Smith type=PERSON id=0
mention text=Washington type=LOCATION id=1
Sentence Text= He was a senator.
mention text=He type=MALE_PRONOUN id=0

The shared ID between He and John Smith indicates the coreference between...

CONTINUE READING

83

Tech Concepts

36

Programming languages

73

Tech Tools

Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.

Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.

50+ new titles added per month and exclusive early access to books as they are being written.

Natural Language Processing with Java and LingPipe Cookbook

Search

Your notes and bookmarks