Let's imagine that we have an index with several million documents. We already know how to build our query and so on. However, when trying to fetch a large number of documents, you see that when getting further and further with pages of the results, the queries slow down and finally timeout or result in memory issues.
The reason for this is that full-text search engines, especially those that are distributed, don't handle paging very well. Of course, getting a few hundred pages of results is not a problem for Elasticsearch, but for going through all the indexed documents or through large result set, a specialized API has been introduced.
When Elasticsearch generates a response, it must determine the order of the documents that form the result. If we are on the first page, this is not a big problem. Elasticsearch just finds the set of documents and collects the first ones; let's say, 20 documents. But if we are on the tenth page, Elasticsearch has to take all...