-
Book Overview & Buying
-
Table Of Contents
RAG from First Principles
By :
Data of different modalities such as text, images, audio, etc. can all be represented as vectors to support more generalized similarity search. This enables vector databases to support multimodal retrieval, where results can be found by comparing text, image, or combined image-text embeddings.
In this example, the system simultaneously retrieves text and image vectors, and uses result fusion strategies such as RRF (reciprocal rank fusion) to assign weights to the retrieval results from both paths, ultimately producing ranked results.

Figure 4.19: Flowchart showing query process for text and image leading to reranking and response output
This section introduces two methods for multimodal and image-based retrieval. The first method uses the Visualized BGE model for image-text embedding and matching. The second method uses ResNet-34 to extract image features for visual similarity retrieval.