10. Text Tagging with Lucene FST | Apache Solr Search Patterns

Sign In Start Free Trial

Book Overview & Buying
Table Of Contents

Apache Solr Search Patterns

By : Jayant Kumar

4.5 (2)

Apache Solr Search Patterns

4.5 (2)

By: Jayant Kumar

Overview of this book

This book is for developers who already know how to use Solr and are looking at procuring advanced strategies for improving their search using Solr. This book is also for people who work with analytics to generate graphs and reports using Solr. Moreover, if you are a search architect who is looking forward to scale your search using Solr, this is a must have book for you. It would be helpful if you are familiar with the Java programming language.

Preface

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Free Chapter

1. Solr Indexing Internals

1. Solr Indexing Internals

The job site problem statement – Solr indexing fundamentals

Working of analyzers, tokenizers, and filters

Handling a multilingual search

Measuring the quality of search results

The e-commerce problem statement

The job site problem statement

Challenges of large-scale indexing

The SolrCloud solution

Summary

2. Customizing the Solr Scoring Algorithm

2. Customizing the Solr Scoring Algorithm

Relevance calculation

Building a custom scorer

Drawbacks of the TF-IDF model

The information gain model

Implementing the information gain model

Options to TF-IDF similarity

Summary

3. Solr Internals and Custom Queries

3. Solr Internals and Custom Queries

Working of a scorer on an inverted index

Working of OR and AND clauses

The eDisMax query parser

Using BRS queries instead of DisMax

Building a custom query parser

Summary

4. Solr for Big Data

4. Solr for Big Data

Introduction to big data

Getting data points using facets

Radius faceting for location-based data

Data analysis using pivot faceting

Graphs for analytics

Summary

5. Solr in E-commerce

5. Solr in E-commerce

Designing an e-commerce search

Handling unclean data

Handling variations in the product

Sorting

Problems and solutions of flash sale searches

Faceting with the option of multi-select

Faceting with hierarchical taxonomy

Faceting with size

Implementing semantic search

Optimizations

Summary

6. Solr for Spatial Search

6. Solr for Spatial Search

Features of spatial search

Lucene 4 spatial module

Indexing for spatial search

Searching and filtering on a spatial index

Distance sort and relevancy boost

Advanced concepts

Summary

7. Using Solr in an Advertising System

7. Using Solr in an Advertising System

Ad system functionalities

Architecture of an ad distribution system

Requirements of an ad distribution system

Performance improvements

Merging Solr with Redis

Summary

8. AJAX Solr

8. AJAX Solr

The purpose of AJAX Solr

The AJAX Solr architecture

Working with AJAX Solr

Performance tuning

Summary

9. SolrCloud

9. SolrCloud

The SolrCloud architecture

Centralized configuration

Setting up SolrCloud

Distributed indexing and search

Routing documents to a particular shard

Adding more nodes to the SolrCloud

Fault tolerance and high availability in SolrCloud

Advanced sharding with SolrCloud

Asynchronous calls

Migrating documents to another collection

Sizing and monitoring of SolrCloud

Using SolrCloud as a NoSQL database

Summary

10. Text Tagging with Lucene FST

10. Text Tagging with Lucene FST

An overview of FST and text tagging

Implementation of FST in Lucene

Text tagging algorithms

Using Solr for text tagging

Implementing a text tagger using Solr

Summary

Index

Index

Implementation of FST in Lucene

The algorithm used for implementing an FST in Lucene is based on the paper Direct Construction of Minimal Acyclic Subsequential Transducers published by Stoyan Mihov and Denis Maurel. This algorithm can be used to build a minimal acyclic sub-sequential transducer (a type of FST) that represents a finite relation, given a sorted list of input words and their outputs. As this algorithm constructs the minimal transducer directly, it has better efficiency than other algorithms. It is the perfect fit for a Lucene FST, as all the terms in the Lucene index are stored in a sorted order.

Note

The following paper was referred to for building the algorithm: http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.24.3698.

An FST is implemented in Lucene under the following package:

org.apache.lucene.util.fst

Let us see a few of the classes inside the package that can be used to work with an FST:

Builder<T>: Can be used to build a minimal FST from pre-sorted terms with...

CONTINUE READING

83

Tech Concepts

36

Programming languages

73

Tech Tools

Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.

Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.

50+ new titles added per month and exclusive early access to books as they are being written.

Apache Solr Search Patterns

Search

Your notes and bookmarks