Book Image

Lucene 4 Cookbook

By : Edwood Ng, Vineeth Mohan
Book Image

Lucene 4 Cookbook

By: Edwood Ng, Vineeth Mohan

Overview of this book

Related Content you might be interested in

Current Title:

Small book image
Lucene 4 Cookbook
Edwood Ng | Vineeth Mohan
Jun, 2015 | 220 pages
No titles found
Table of Contents (16 chapters)
Lucene 4 Cookbook
Lucene 4 Cookbook
Credits
Credits
About the Authors
About the Authors
About the Reviewers
About the Reviewers
www.PacktPub.com
www.PacktPub.com
Preface
Preface
Free Chapter
1
Introducing Lucene
Introducing Lucene
Introduction
Installing Lucene
Setting up a simple Java Lucene project
Obtaining an IndexWriter
Creating an analyzer
Creating fields
Creating and writing documents to an index
Deleting documents
Obtaining an IndexSearcher
Creating queries with the Lucene QueryParser
Performing a search
Enumerating results
2
Analyzing Your Text
Analyzing Your Text
Introduction
Obtaining a common analyzer
Obtaining a TokenStream
Obtaining TokenAttribute values
Using PositionIncrementAttribute
Using PerFieldAnalyzerWrapper
Defining custom TokenFilters
Defining custom analyzers
Defining custom tokenizers
Defining custom attributes
3
Indexing Your Data
Indexing Your Data
Introduction
Obtaining an IndexWriter
Creating a StringField
Creating a TextField
Creating a numeric field
Creating a DocValue Field
Transactional commits and index versioning
Reusing field and document objects per thread
Delving into field norms
Changing similarity implementation used during indexing
4
Searching Your Indexes
Searching Your Indexes
Introduction
Obtaining IndexReaders
Un-inverting single-valued fields in memory with FieldCache
TermVectors
IndexSearcher
Constructing queries
Specifying sort logic
Forming a search result
Pagination
Using Collectors
Sorting with custom FieldComparator
5
Near Real-time Searching
Near Real-time Searching
Introduction
Using the DirectoryReader to open index in Near Real-Time
Using the SearcherManager to refresh IndexSearcher
Generational indexing with TrackingIndexWriter
Maintaining search sessions with SearcherLifetimeManager
Performance tuning: latency and throughput
6
Querying and Filtering Data
Querying and Filtering Data
Introduction
Performing advanced filtering
Creating a custom filter
Searching with QueryParser
TermQuery and TermRangeQuery
BooleanQuery
PrefixQuery and WildcardQuery
PhraseQuery and MultiPhraseQuery
FuzzyQuery
NumericRangeQuery
DisjunctionMaxQuery
RegexpQuery
SpanQuery
CustomScoreQuery
7
Flexible Scoring
Flexible Scoring
Introduction
Overriding similarity
Implementing the BM25 model
Implementing the language model
Implementing the divergence from randomness model
Implementing the information-based model
8
Introducing Elasticsearch
Introducing Elasticsearch
Introduction
Getting Elasticsearch
Creating a new index
Predefine field mappings
Adding a document
Deleting a document
Updating a document
Performing bulk indexing
Searching the index
Scaling Elasticsearch
9
Extending Lucene with Modules
Extending Lucene with Modules
Introduction
Exploring spatial search
Implementing joins
Performing faceting
Implementing grouping
Employing autosuggest
Implementing highlighting
Index
Index

Implementing the divergence from randomness model

In Lucene, divergence from randomness model is implemented as DFRSimilarity. It's made up of three components: BasicModel, AfterEffect, and Normalization. BasicModel is a model of information content, AfterEffect is the first normalization, and Normalization is second (length) normalization. Here is an excerpt from Lucene's Javadoc on DFRSimilarity's components:

  1. BasicModel: This is a basic model of information content:

    • BasicModelBE: This is the limiting form of Bose-Einstein

    • BasicModelG: This is the geometric approximation of Bose-Einstein

    • BasicModelP: This is the Poisson approximation of the Binomial

    • BasicModelD: This is the divergence approximation of the Binomial

    • BasicModelIn: This is the inverse document frequency

    • BasicModelIne: This is the inverse expected document frequency (mixture of Poisson and IDF)

    • BasicModelIF: This is the inverse term frequency (approximation of I(ne))

  2. AfterEffect: This is the first normalization of information...

Previous Section
End of Section 6
Next Section