Machine learning for search

As a cross-disciplinary task, neural search has gone beyond the boundaries of information retrieval. It requires a general understanding of the concepts of machine learning, deep learning, and how we can apply these techniques to improve a search task. In this section, we will give a brief introduction to machine learning and how it can be applied to search systems.

Understanding machine learning and artificial intelligence

Machine learning refers to a technique that teaches computers to make decisions in a way that comes naturally to humans by enabling computers to learn the inherent laws of data and acquire new experience and knowledge, thus improving their intelligence.

Because various industries require an increased level of efficiency during data processing and analysis due to their growing demand for data, a large number of machine learning algorithms have emerged. The concept of statistical machine learning algorithms primarily refers to the steps and processes of solving optimization problems through mathematical and statistical methods.

With respect to different data and model requirements, appropriate machine learning algorithms are selected and employed to tackle practical issues in a more efficient manner. Machine learning has achieved great success in many fields, such as natural language understanding, computer vision, machine translation, and expert systems. It is fair to say that whether a system has a learning function or not has become a hallmark of it possessing intelligence.

Hinton et al. (2006) proposed the concept of deep learning (deep learning/deep neural networks). In 2009, Hinton introduced deep neural networks to scholars specialized in voices. Hence, in 2010, this field of research witnessed a remarkable breakthrough in speech recognition. In the next 11 years, convolutional neural networks (CNNs) were applied in the field of image recognition, leading to significant achievements.

Three founders of neural networks, LeCun, Bengio, and Hinton (2015), published a review titled Deep Learning in Nature. This shows that deep neural networks have not only been accepted by academia, but also the industrial field. Furthermore, in 2016 and 2017, the world witnessed a general expansion in deep learning. AlphaGo and AlphaZero were invented by Google after a short learning period and won a landslide victory over the top three Go players in the world. The intelligent voice system launched by iFLYTEK boasts a recognition accuracy rate of over 97% and stands at the forefront of AI worldwide; the autonomous driving systems developed by companies such as Google and Tesla have passed a milestone of testing on the road. These achievements have unveiled the value and charm of neural networks to humans again.

Machine learning has been applied to various industries, so maybe we can ask ourselves: can we apply machine learning to search applications? The answer here is “Yes.” In the next section, we’ll give a brief overview of different types of machine learning and how search can benefit from it.

Machine learning and learning-to-rank

Imagine a scenario where you intend to train a model capable of evaluating the price of a new apartment or house based on the collected data related to local real estate information and prices. This is one of the most important tasks of machine learning: regression.

Before the popularization of the deep learning technique, data analysts would have had to clean this data, use business logic to perform feature engineering, and design features of a real estate price predictor, such as the floor area, construction time, and type of apartments or houses, as well as the average prices of surrounding apartments or houses, and so on.

After feature engineering has been completed, raw data will be used to form a two-dimensional data table similar to Excel. The horizontal axis represents each house record, and the vertical axis represents each feature. The data is usually divided into two to three parts again: the majority of the data is used for model training, while a small amount of the data is used for model evaluation.

Next, machine learning engineers will select one or more appropriate algorithms from the machine learning toolkit to train the model and evaluate the performance of the model in the test data. Finally, the model with the best performance will be deployed in the production environment to serve customers.

Imagine another scenario where many landmark pictures have been collected from social networks. When a user uploads a new landmark picture, you expect your system to automatically recognize the name of the site. This is another important task of machine learning: classification.

In the field of traditional machine learning and computer vision, some features, such as SIFT, SURF, and HOG, are employed to develop a Bag-of-Visual-Words (BoW) through which a vector representation of this photo is established. Moreover, models are used to predict the classification. Nowadays, deep learning serves as the model to extract visual features from images without the requirement of feature engineering.

Let’s take a moment to look at our two examples. During the training process of predicting prices of houses (apartments), models are trained using feature engineering. All the training data is ground-truth, i.e., the house (apartment) prices and landmark names are documented. Such tasks are collectively referred to as supervised machine learning in the field of machine learning.

Since we can perform regression analysis and classification of data through supervised machine learning, is it possible to apply supervised machine learning in the search? The answer is yes, of course.

Assuming that our task is to optimize the search system, the goal is to predict the user’s click rate for the document and return documents with a higher predicted click rate to the user first. This is called learning-to-rank (first stage) and neural information retrieval (second stage). The concept of learning-to-rank (based on statistical machine learning), as proposed by academia in the early 1990s, evolved for nearly 20 years before experiencing a downturn since the emergence of deep learning in 2010, when neural information retrieval was at its peak.

Just like the prediction of an apartment (or house) price, or landmark recognition, engineers first performed data engineering after collecting such data. Common features include the number of query keywords in the document title/body, the percentage of the document title body that contains the query keywords, the tf-idf score, and the bm-25 score, among others. It follows that the final score of a traditional search system is used as a numerical feature during the training of models.

In a real-world scenario, Microsoft’s Bing search platform was designed with its own Microsoft Learning to Rank Datasets, which contain 136 features. Besides, they published a contest for learning-to-rank, calling for the use of these datasets as a basic training model for predicting the matching degree of web pages. After that, a trained model is applied in the production environment of a Bing search, which has improved the search effect to a certain extent.

At the same time, search companies such as Google, Yahoo, and Baidu have also conducted a large amount of research and have partially deployed their research results into the production environment.

In the fields of enterprise and personal search, Elastic has developed its learning-to-rank plugin named ElasticSearch LTR, which can be plug-in into your ES-powered search system. As a user, you still need to use a familiar machine learning framework to design features, train the learning-to-rank model, evaluate the model performance, and select models. Elasticsearch’s support for learning-to-rank can be plugged into the existence search system and get a new predicted ranking score based on model output. Although machine learning can be used to design models for multi-modal data, Elasticsearch places more emphasis on text-to-text search. Figure 1.4 demonstrates how learning-to-rank works in a search system.

Figure 1.4 – Learning-to-rank

This book will focus on search powered by deep neural networks, namely neural information retrieval.

The advantage of neural information retrieval is that users do not have to design features by themselves. Normally, we leverage two independent deep learning models (neural nets) as feature extractors to extract vectors from queries and documents respectively. Then, we measure the similarity between two vectors using metrics, such as cosine similarity. At this stage, neural-network-powered search has become very promising for industrial use cases. In the next section, we will introduce some of the potential applications for neural-network-powered search.