Skip to main content
Version: User Guides (Cloud)

Multi-Vector Hybrid Search

In many applications, data is represented using various fields like title and description, or through different modalities such as text, images, and audio. Hybrid search enhances accuracy by combining searches across these diverse fields. Zilliz Cloud supports this feature with multiple vector fields, conducting several approximate nearest neighbor (ANN) searches simultaneously. Multi-vector hybrid search typically yields better results than a single search. This approach is especially valuable in domains that demand a complex understanding of content, such as e-commerce search, academic paper search, and recommendation systems.

Qx7UwgI6jhrku8bAxZqcYxZMnSe

The multi-vector hybrid search integrates different search methods or spans embeddings from various modalities:

  • Sparse-Dense Vector Search: Dense Vector are excellent for capturing semantic relationships, while Sparse Vector are highly effective for precise keyword matching. Hybrid search combines these approaches to provide both a broad conceptual understanding and exact term relevance, thus improving search results. By leveraging the strengths of each method, hybrid search overcomes the limitations of indiviual approaches, offering better performance for complex queries.

  • Multimodal Vector Search: Multimodal vector search is a powerful technique that allows you to search across various data types, including text, images, audio, and others. The main advantage of this approach is its ability to unify different modalities into a seamless and cohesive search experience. For instance, in product search, a user might input a text query to find products described with both text and images. By combining these modalities through a hybrid search method, you can enhance search accuracy or enrich the search results.

Example​

Let's consider a product search scenario where each product includes a text description and an image. Based on the available data, we can conduct three types of searches:

  • Semantic Text Search: This involves querying the text description of the product using dense vectors. Text embeddings can be generated using models such as BERT and Transformers or services like OpenAI.

  • Full-Text Search: Here, we query the text description of the product using a keyword match with sparse vectors. Algorithms like BM25 or sparse embedding models such as BGE-M3 or SPLADE can be utilized for this purpose.

  • Multimodal Image Search: This method queries over the image using a text query with dense vectors. Image embeddings can be generated with models like CLIP.

This guide will walk you through an example of a multimodal hybrid search combining the above search methods, given the raw text description and image embeddings of products. We will demonstrate how to store multi-vector data and perform hybrid searches with a reranking strategy.

Create a collection with multiple vector fields​

The process of creating a collection involves three key steps: defining the collection schema, configuring the index parameters, and creating the collection.

Define schema​

For multi-vector hybrid search, we should define multiple vector fields within a collection schema. By default, each collection can accommodate up to 4 vector fields. However, if necessary, you can contact us to include up to 10 vector fields in your collections.

This example incorporates the following fields into the schema:

  • id: Serves as the primary key for storing text IDs. This field is of data type INT64.

  • text: Used for storing textual content. This field is of the data type VARCHAR with a maximum length of 1000 bytes. The enable_analyzer option is set to True to facilitate full-text search.

  • text_dense: Used to store dense vectors of the texts. This field is of the data type FLOAT_VECTOR with a vector dimension of 768.

  • text_sparse: Used to store sparse vectors of the texts. This field is of the data type SPARSE_FLOAT_VECTOR.

  • image_dense: Used to store dense vectors of the product images. This field is of the data type FLOAT_VETOR with a vector dimension of 512.

Since we will use the built-in BM25 algorithm to perform a full-text search on the text field, it is necessary to add the Milvus Function to the schema. For further details, please refer to Full Text Search.

from pymilvus import (
    MilvusClient, DataType, Function, FunctionType
)

client = MilvusClient(
    uri="YOUR_CLUSTER_ENDPOINT",
    token="YOUR_CLUSTER_TOKEN"
)

# Init schema with auto_id disabled
schema = MilvusClient.create_schema(auto_id=False)

# Add fields to schema
schema.add_field(field_name="id", datatype=DataType.INT64, is_primary=True, description="product id")
schema.add_field(field_name="text", datatype=DataType.VARCHAR, max_length=1000, enable_analyzer=True, description="raw text of product description")
schema.add_field(field_name="text_dense", datatype=DataType.FLOAT_VECTOR, dim=768, description="text dense embedding")
schema.add_field(field_name="text_sparse", datatype=DataType.SPARSE_FLOAT_VECTOR, description="text sparse embedding auto-generated by the built-in BM25 function")
schema.add_field(field_name="image_dense", datatype=DataType.FLOAT_VECTOR, dim=512, description="image dense embedding")

# Add function to schema
bm25_function = Function(
    name="text_bm25_emb",
    input_field_names=["text"],
    output_field_names=["text_sparse"],
    function_type=FunctionType.BM25,
)
schema.add_function(bm25_function)

Create index​

from pymilvus import MilvusClient

# Prepare index parameters
index_params = client.prepare_index_params()

# Add indexes
index_params.add_index(
    field_name="text_dense",
    index_name="text_dense_index",
    index_type="AUTOINDEX",
    metric_type="IP"
)

index_params.add_index(
    field_name="text_sparse",
    index_name="text_sparse_index",
    index_type="AUTOINDEX",
    metric_type="BM25"
)

index_params.add_index(
    field_name="image_dense",
    index_name="image_dense_index",
    index_type="AUTOINDEX",
    metric_type="IP"
)

Create collection​

Create a collection named demo with the collection schema and indexes configured in the previous two steps.

from pymilvus import MilvusClient

client.create_collection(
    collection_name="my_collection",
    schema=schema,
    index_params=index_params
)

Insert data​

This section inserts data into the my_collection collection based on the schema defined earlier. During insert, ensure all fields, except those with auto-generated values, are provided with data in the correct format. In this example:

  • id: an integer representing the product ID

  • text: a string containing the product description

  • text_dense: a list of 768 floating-point values representing the dense embedding of the text description

  • image_dense: a list of 512 floating-point values representing the dense embedding of the product image

You may use the same or different models to generate dense embeddings for each field. In this example, the two dense embeddings have different dimensions, suggesting they were generated by different models. When defining each search later, be sure to use the corresponding model to generate the appropriate query embedding.

Since this example uses the built-in BM25 function to generate sparse embeddings from the text field, you do not need to supply sparse vectors manually. However, if you opt not to use BM25, you must precompute and provide the sparse embeddings yourself.

from pymilvus import MilvusClient

data=[
    {
        "id": 0,
        "text": "Red cotton t-shirt with round neck",
        "text_dense": [0.3580376395471989, -0.6023495712049978, 0.18414012509913835, ...],
        "image_dense": [0.6366019600530924, -0.09323198122475052, ...]
    }，
    {
        "id": 1,
        "text": "Wireless noise-cancelling over-ear headphones",
        "text_dense": [0.19886812562848388, 0.06023560599112088, 0.6976963061752597, ...],
        "image_dense": [0.6414180010301553, 0.8976979978567611, ...]
    },
    {
        "id": 2,
        "text": "Stainless steel water bottle, 500ml",
        "dense": [0.43742130801983836, -0.5597502546264526, 0.6457887650909682, ...],
        "image_dense": [-0.6901259768402174, 0.6100500332193755, ...]
    }
]

res = client.insert(
    collection_name="my_collection",
    data=data
)

Create multiple AnnSearchRequest instances​

Hybrid Search is implemented by creating multiple AnnSearchRequest in the hybrid_search() function, where each AnnSearchRequest represents a basic ANN search request for a specific vector field. Therefore, before conducting a Hybrid Search, it is necessary to create an AnnSearchRequest for each vector field.

In addition, by configuring the expr parameter in an AnnSearchRequest, you can set the filtering conditions for your hybrid search. Please refer to Filtered Search and Filtering.

📘Notes

In Hybrid Search, each AnnSearchRequest supports only one query data.

To demonstrate the capabilities of various search vector fields, we will construct three AnnSearchRequest search requests using a sample query. We will also use its pre-computed dense vectors for this process. The search requests will target the following vector fields:

  • text_dense for semantic text search, allowing for contextual understanding and retrieval based on meaning rather than direct keyword matching.

  • text_sparsefor full-text search or keyword matching, focusing on exact word or phrase matches within the text.

  • image_densefor multimodal text-to-image search, to retrieve relevant product images based on the semantic content of the query.

from pymilvus import AnnSearchRequest

query_text = "white headphones, quiet and comfortable"
query_dense_vector = [0.3580376395471989, -0.6023495712049978, 0.5142999509918703, ...]
query_multimodal_vector = [0.015829865178701663, 0.5264158340734488, ...]

# text semantic search (dense)
search_param_1 = {
    "data": [query_dense_vector],
    "anns_field": "text_dense",
    "param": {"nprobe": 10},
    "limit": 2
}
request_1 = AnnSearchRequest(**search_param_1)

# full-text search (sparse)
search_param_2 = {
    "data": [query_text],
    "anns_field": "text_sparse",
    "param": {"drop_ratio_search": 0.2},
    "limit": 2
}
request_2 = AnnSearchRequest(**search_param_2)

# text-to-image search (multimodal)
search_param_3 = {
    "data": [query_multimodal_vector],
    "anns_field": "image_dense",
    "param": {"nprobe": 10},
    "limit": 2
}
request_3 = AnnSearchRequest(**search_param_3)

reqs = [request_1, request_2, request_3]

Given that the parameter limit is set to 2, each AnnSearchRequest returns 2 search results. In this example, 3 AnnSearchRequest instances are created, resulting in a total of 6 search results.

Configure a reranking strategy​

To merge and rerank the sets of ANN search results, selecting an appropriate reranking strategy is essential. Zilliz Cloud offers two types of reranking strategies:

  • WeightedRanker: Use this strategy if the results need to emphasize a particular vector field. WeightedRanker allows you to assign greater weight to certain vector fields, highlighting them more prominently.

  • RRFRanker (Reciprocal Rank Fusion Ranker): Choose this strategy when no specific emphasis is required. RRFRanker effectively balances the importance of each vector field.

For more details on these reranking mechanisms, please refer to Reranking.

In this example, since there is no particular emphasis on specific search queries, we will proceed with the RRFRanker strategy.

from pymilvus import RRFRanker

ranker = RRFRanker(100)

Before initiating a Hybrid Search, ensure that the collection is loaded. If any vector fields within the collection lack an index or are not loaded into memory, an error will occur upon executing the Hybrid Search method.

from pymilvus import MilvusClient

res = client.hybrid_search(
    collection_name="my_collection",
    reqs=reqs,
    ranker=ranker,
    limit=2
)
for hits in res:
    print("TopK results:")
    for hit in hits:
        print(hit)

The following is the output:

["['id: 1, distance: 0.006047376897186041, entity: {}', 'id: 2, distance: 0.006422005593776703, entity: {}']"]

With the limit=2 parameter specified for the Hybrid Search, Zilliz Cloud will rerank the six results obtained from the three searches. Ultimately, they will return only the top two most similar results.