4. Tagging Words and Tokens | Natural Language Processing with Java and LingPipe Cookbook

Book Overview & Buying
Table Of Contents

Natural Language Processing with Java and LingPipe Cookbook

By : Dayanidhi

4.8 (4)

Buy this Book

Natural Language Processing with Java and LingPipe Cookbook

4.8 (4)

By: Dayanidhi

Buy this Book

Overview of this book

This book is for experienced Java developers with NLP needs, whether academics, industrialists, or hobbyists. A basic knowledge of NLP terminology will be beneficial.

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Free Chapter

1. Simple Classifiers

Introduction

Deserializing and running a classifier

Getting confidence estimates from a classifier

Getting data from the Twitter API

Applying a classifier to a .csv file

Evaluation of classifiers – the confusion matrix

Training your own language model classifier

How to train and evaluate with cross validation

Viewing error categories – false positives

Understanding precision and recall

How to serialize a LingPipe object – classifier example

Eliminate near duplicates with the Jaccard distance

How to classify sentiment – simple version

2. Finding and Working with Words

Introduction

Introduction to tokenizer factories – finding words in a character stream

Combining tokenizers – lowercase tokenizer

Combining tokenizers – stop word tokenizers

Using Lucene/Solr tokenizers

Using Lucene/Solr tokenizers with LingPipe

Evaluating tokenizers with unit tests

Modifying tokenizer factories

Finding words for languages without white spaces

3. Advanced Classifiers

Introduction

A simple classifier

Language model classifier with tokens

Naïve Bayes

Feature extractors

Logistic regression

Multithreaded cross validation

Tuning parameters in logistic regression

Customizing feature extraction

Combining feature extractors

Classifier-building life cycle

Linguistic tuning

Thresholding classifiers

Train a little, learn a little – active learning

Annotation

4. Tagging Words and Tokens

Introduction

Interesting phrase detection

Foreground- or background-driven interesting phrase detection

Hidden Markov Models (HMM) – part-of-speech

N-best word tagging

Confidence-based tagging

Training word tagging

Word-tagging evaluation

Conditional random fields (CRF) for word/token tagging

Modifying CRFs

5. Finding Spans in Text – Chunking

Introduction

Sentence detection

Evaluation of sentence detection

Tuning sentence detection

Marking embedded chunks in a string – sentence chunk example

Paragraph detection

Simple noun phrases and verb phrases

Regular expression-based chunking for NER

Dictionary-based chunking for NER

Translating between word tagging and chunks – BIO codec

HMM-based NER

Mixing the NER sources

CRFs for chunking

NER using CRFs with better features

6. String Comparison and Clustering

Introduction

Distance and proximity – simple edit distance

Weighted edit distance

The Jaccard distance

The Tf-Idf distance

Using edit distance and language models for spelling correction

The case restoring corrector

Automatic phrase completion

Single-link and complete-link clustering using edit distance

Latent Dirichlet allocation (LDA) for multitopic clustering

7. Finding Coreference Between Concepts/People

Introduction

Named entity coreference with a document

Adding pronouns to coreference

Cross-document coreference

The John Smith problem

Index

Natural Language Processing with Java and LingPipe Cookbook

By : Dayanidhi

Natural Language Processing with Java and LingPipe Cookbook

By: Dayanidhi

Overview of this book

Chapter 4. Tagging Words and Tokens

Confirmation

Buy this book with your credits?

Submit Your Feedback

Create a Free Account To Continue Reading

Sign in to activate your 7-day free access