Skip to main content
Version: User Guides (BYOC)

Single-Vector Search

After your data is inserted, the next step is to send a search request to search for vectors that are similar to your query vector. A single-vector search compares your query vector against the existing vectors in your collection to find the most similar entities, returning their IDs and the distances between them. This process can optionally return the vector values and metadata of the results.

Overview​

There are a variety of search types to meet different requirements:

  • Basic search: Includes single-vector search, bulk-vector search, partition search, and search with specified output fields.

  • Filtered search: Applies filtering criteria based on scalar fields to refine search results.

  • Range search: Finds vectors within a specific distance range from the query vector.

  • Grouping search: Groups search results based on a specific field to ensure diversity in the results.

Preparations​

The code snippets below repurpose the existing code to establish connections to a Zilliz Cloud cluster, quickly set up a collection and two partitions, and insert data into them.

# 1. Set up a Milvus client
client = MilvusClient(
    uri=CLUSTER_ENDPOINT,
    token=TOKEN 
)

# 2. Create a collection
client.create_collection(
    collection_name="quick_setup",
    dimension=5, # The dimension value should be greater than 1
    metric_type="IP"
)

# 3. Insert randomly generated vectors 
colors = ["green", "blue", "yellow", "red", "black", "white", "purple", "pink", "orange", "brown", "grey"]
data = []

for i in range(1000):
    current_color = random.choice(colors)
    data.append({
        "id": i,
        "vector": [ random.uniform(-1, 1) for _ in range(5) ],
        "color": current_color,
        "color_tag": f"{current_color}_{str(random.randint(1000, 9999))}"
    })

res = client.insert(
    collection_name="quick_setup",
    data=data
)

print(res)

# Output
#
# {
#     "insert_count": 1000,
#     "ids": [
#         0,
#         1,
#         2,
#         3,
#         4,
#         5,
#         6,
#         7,
#         8,
#         9,
#         "(990 more items hidden)"
#     ]
# }

# 6.1 Create partitions 
client.create_partition(
    collection_name="quick_setup",
    partition_name="red"
)

client.create_partition(
    collection_name="quick_setup",
    partition_name="blue"
)

# 6.1 Insert data into partitions
red_data = [ {"id": i, "vector": [ random.uniform(-1, 1) for _ in range(5) ], "color": "red", "color_tag": f"red_{str(random.randint(1000, 9999))}" } for i in range(500) ]
blue_data = [ {"id": i, "vector": [ random.uniform(-1, 1) for _ in range(5) ], "color": "blue", "color_tag": f"blue_{str(random.randint(1000, 9999))}" } for i in range(500) ]

res = client.insert(
    collection_name="quick_setup",
    data=red_data,
    partition_name="red"
)

print(res)

# Output
#
# {
#     "insert_count": 500,
#     "ids": [
#         0,
#         1,
#         2,
#         3,
#         4,
#         5,
#         6,
#         7,
#         8,
#         9,
#         "(490 more items hidden)"
#     ]
# }

res = client.insert(
    collection_name="quick_setup",
    data=blue_data,
    partition_name="blue"
)

print(res)

# Output
#
# {
#     "insert_count": 500,
#     "ids": [
#         0,
#         1,
#         2,
#         3,
#         4,
#         5,
#         6,
#         7,
#         8,
#         9,
#         "(490 more items hidden)"
#     ]
# }

When sending a search request, you can provide one or more vector values representing your query embeddings and a limit value indicating the number of results to return.

Depending on your data and your query vector, you may get fewer than limit results. This happens when limit is larger than the number of possible matching vectors for your query.

Single-vector search is the simplest form of search operations in Zilliz Cloud, designed to find the most similar vectors to a given query vector.

To perform a single-vector search, specify the target collection name, the query vector, and the desired number of results (limit). This operation returns a result set comprising the most similar vectors, their IDs, and distances from the query vector.

Here is an example of searching for the top 3 entities that are most similar to the query vector:

# 4. Single vector search
query_vector = [0.3580376395471989, -0.6023495712049978, 0.18414012509913835, -0.26286205330961354, 0.9029438446296592],

res = client.search(
    collection_name="quick_setup",
    data=[query_vector],
    limit=3, # The number of results to return
    search_params={"metric_type": "IP", "params": {"level": 1}}
)

print(res)

The output is similar to the following:

[
    [
        {
            "id": 206,
            "distance": 2.50616455078125,
            "entity": {}
        },
        {
            "id": 138,
            "distance": 2.500145435333252,
            "entity": {}
        },
        {
            "id": 224,
            "distance": 2.484044313430786,
            "entity": {}
        }
    ]
]

The output showcases the top 3 neighbors nearest to your query vector, including their unique IDs and the calculated distances.

A bulk-vector search extends the single-vector search concept by allowing multiple query vectors to be searched in a single request. This type of search is ideal for scenarios where you need to find similar vectors for a set of query vectors, significantly reducing the time and computational resources required.

In a bulk-vector search, you can include several query vectors in the data field. The system processes these vectors in parallel, returning a separate result set for each query vector, each set containing the closest matches found within the collection.

Here is an example of searching for two distinct sets of the most similar entities from two query vectors:

# 5. Batch-vector search
query_vectors = [
    [0.3580376395471989, -0.6023495712049978, 0.18414012509913835, -0.26286205330961354, 0.9029438446296592],
    [0.19886812562848388, 0.06023560599112088, 0.6976963061752597, 0.2614474506242501, 0.838729485096104]
] # A list of two vectors

res = client.search(
    collection_name="quick_setup",
    data=query_vectors, 
    limit=2,
    search_params={"metric_type": "IP", "params": {"level": 1}}
)

print(res) # Two sets of results are to return

The output is similar to the following:

# Two sets of vectors are returned as expected

[
    [
        {
            "id": 81,
            "distance": 1.633650779724121,
            "entity": {}
        },
        {
            "id": 428,
            "distance": 1.6174099445343018,
            "entity": {}
        }
    ],
    [
        {
            "id": 972,
            "distance": 1.7308459281921387,
            "entity": {}
        },
        {
            "id": 545,
            "distance": 1.670518398284912,
            "entity": {}
        }
    ]
]

The results include two sets of nearest neighbors, one for each query vector, showcasing the efficiency of bulk-vector searches in handling multiple query vectors at once.

Partition search narrows the scope of your search to a specific subset or partition of your collection. This is particularly useful for organized datasets where data is segmented into logical or categorical divisions, allowing for faster search operations by reducing the volume of data to scan.

To conduct a partition search, simply include the name of the target partition in partition_names of your search request. This specifies that the search operation only considers vectors within the specified partition.

Here is an example of searching for entities in the partition named red:

# 6.2 Search within a partition
query_vector = [0.3580376395471989, -0.6023495712049978, 0.18414012509913835, -0.26286205330961354, 0.9029438446296592]

res = client.search(
    collection_name="quick_setup",
    data=[query_vector],
    limit=5,
    search_params={"metric_type": "IP", "params": {"level": 1}},
    partition_names=["red"]
)

print(res)

The output is similar to the following:

[
    [
        {
            "id": 320,
            "distance": 1.243729591369629,
            "entity": {}
        },
        {
            "id": 200,
            "distance": 1.2299367189407349,
            "entity": {}
        },
        {
            "id": 154,
            "distance": 1.1562182903289795,
            "entity": {}
        },
        {
            "id": 29,
            "distance": 1.1135238409042358,
            "entity": {}
        },
        {
            "id": 109,
            "distance": 1.0907914638519287,
            "entity": {}
        }
    ]
]

Then, search for entities in the partition named blue:

res = client.search(
    collection_name="quick_setup",
    data=[query_vector],
    limit=5,
    search_params={"metric_type": "IP", "params": {"level": 1}},
    partition_names=["blue"]
)

print(res)

The output is similar to the following:

[
    [
        {
            "id": 59,
            "distance": 1.3296087980270386,
            "entity": {}
        },
        {
            "id": 139,
            "distance": 1.1872179508209229,
            "entity": {}
        },
        {
            "id": 201,
            "distance": 1.1474100351333618,
            "entity": {}
        },
        {
            "id": 298,
            "distance": 1.117565631866455,
            "entity": {}
        },
        {
            "id": 435,
            "distance": 1.0910152196884155,
            "entity": {}
        }
    ]
]

The data in red differs from that in blue. Therefore, the search results will be constrained to the specified partition, reflecting the unique characteristics and data distribution of that subset.

Search with output fields​

Search with output fields allows you to specify which attributes or fields of the matched vectors should be included in the search results.

You can specify output_fields in a request to return results with specific fields.

Here is an example of returning results with color attribute values:

# 7. Search with output fields
query_vector = [0.3580376395471989, -0.6023495712049978, 0.18414012509913835, -0.26286205330961354, 0.9029438446296592]

res = client.search(
    collection_name="quick_setup",
    data=[query_vector],
    limit=5,
    search_params={"metric_type": "IP", "params": {"level": 1}},
    output_fields=["color"]
)

print(res)

The output is similar to the following:

[
    [
        {
            "id": 29,
            "distance": 2.6317718029022217,
            "entity": {
                "color": "red"
            }
        },
        {
            "id": 405,
            "distance": 2.6302318572998047,
            "entity": {
                "color": "blue"
            }
        },
        {
            "id": 458,
            "distance": 2.3892529010772705,
            "entity": {
                "color": "green"
            }
        },
        {
            "id": 555,
            "distance": 2.350921154022217,
            "entity": {
                "color": "orange"
            }
        },
        {
            "id": 435,
            "distance": 2.29063081741333,
            "entity": {
                "color": "blue"
            }
        }
    ]
]

Alongside the nearest neighbors, the search results will include the specified field color, providing a richer set of information for each matching vector.

Filtered search applies scalar filters to vector searches, allowing you to refine the search results based on specific criteria. You can find more about filter expressions in Boolean Expression Rules and examples in Get & Scalar Query.

Use the like operator​

The like operator enhances string searches by evaluating patterns including prefixes, infixes, and suffixes:

  • Prefix matching: To find values starting with a specific prefix, use the syntax 'like "prefix%"'.

  • Infix matching: To find values containing a specific sequence of characters anywhere within the string, use the syntax 'like "%infix%"'.

  • Suffix matching: To find values ending with a specific suffix, use the syntax 'like "%suffix"'.

For single-character matching, underscore (_) acts as a wildcard for one character, e.g., 'like "y_llow"'.

Special characters in search strings​

If you want to search for a string that contains special characters like underscores (_) or percent signs (%), which are normally used as wildcards in search patterns (_ for any single character and % for any sequence of characters), you must escape these characters to treat them as literal characters. Use a backslash (\) to escape special characters, and remember to escape the backslash itself. For instance:

  • To search for a literal underscore, use \_.

  • To search for a literal percent sign, use \%.

So, if you need to search for the text "_version_", your query should be formatted as 'like "\_version\_"' to ensure the underscores are treated as part of the search term and not as wildcards.

Filter results whose color is prefixed with red:

# 8. Filtered search
# 8.1 Filter with "like" operator and prefix wildcard
query_vector = [0.3580376395471989, -0.6023495712049978, 0.18414012509913835, -0.26286205330961354, 0.9029438446296592]

res = client.search(
    collection_name="quick_setup",
    data=[query_vector],
    limit=5,
    search_params={"metric_type": "IP", "params": {"level": 1}},
    filter='color_tag like "red%"',
    output_fields=["color_tag"]
)

print(res)

The output is similar to the following:

[
    [
        {
            "id": 58,
            "distance": 1.4645483493804932,
            "entity": {
                "color_tag": "red_8218"
            }
        },
        {
            "id": 307,
            "distance": 1.4149816036224365,
            "entity": {
                "color_tag": "red_3923"
            }
        },
        {
            "id": 16,
            "distance": 1.3404488563537598,
            "entity": {
                "color_tag": "red_9524"
            }
        },
        {
            "id": 142,
            "distance": 1.31600022315979,
            "entity": {
                "color_tag": "red_4160"
            }
        },
        {
            "id": 438,
            "distance": 1.315270185470581,
            "entity": {
                "color_tag": "red_8131"
            }
        }
    ]
]

Filter results whose color contains the letters ll anywhere within the string:

# Infix match on color field
query_vector = [0.3580376395471989, -0.6023495712049978, 0.18414012509913835, -0.26286205330961354, 0.9029438446296592]

res = client.search(
    collection_name="quick_setup", # Replace with the actual name of your collection
    data=[query_vector],
    limit=5, # Max. number of search results to return
    search_params={"metric_type": "IP", "params": {"level": 1}}, # Search parameters
    output_fields=["color_tag"], # Output fields to return
    filter='color like "%ll%"' # Filter on color field, infix match on "ll"
)

result = json.dumps(res, indent=4)
print(result)

The output is similar to the following:

[
    [
        {
            "id": 5,
            "distance": 0.7972343564033508,
            "entity": {
                "color": "yellow_4222"
            }
        }
    ]
]

Range search allows you to find vectors that lie within a specified distance range from your query vector.

By setting radius and optionally range_filter, you can adjust the breadth of your search to include vectors that are somewhat similar to the query vector, providing a more comprehensive view of potential matches.

  • radius: Defines the outer boundary of your search space. Only vectors that are within this distance from the query vector are considered potential matches.

  • range_filter: While radius sets the outer limit of the search, range_filter can be optionally used to define an inner boundary, creating a distance range within which vectors must fall to be considered matches.

# Conduct a range search
search_params = {
    "metric_type": "IP",
    "params": {
        "radius": 0.8, # Radius of the search circle
        "range_filter": 1.0 # Range filter to filter out vectors that are not within the search circle
    }
}

query_vector = [0.3580376395471989, -0.6023495712049978, 0.18414012509913835, -0.26286205330961354, 0.9029438446296592]

res = client.search(
    collection_name="quick_setup", # Replace with the actual name of your collection
    data=[query_vector],
    limit=3, # Max. number of search results to return
    search_params=search_params, # Search parameters
    output_fields=["color_tag"], # Output fields to return
)

result = json.dumps(res, indent=4)
print(result)

The output is similar to the following:

[
    [
        {
            "id": 136,
            "distance": 2.4410910606384277,
            "entity": {}
        },
        {
            "id": 897,
            "distance": 2.2852015495300293,
            "entity": {}
        },
        {
            "id": 336,
            "distance": 2.2819623947143555,
            "entity": {}
        },
        {
            "id": 50,
            "distance": 2.2552754878997803,
            "entity": {}
        },
        {
            "id": 462,
            "distance": 2.2343976497650146,
            "entity": {}
        }
    ]
]

The parameter settings for radius and range_filter vary with the metric type in use.

Metric Type

Charactericstics

Range Search Settings

L2

Smaller L2 distances indicate higher similarity.

To exclude the closest vectors from results, ensure that:

0 <= range_filter <= distance < radius < ∞

IP

Larger IP distances indicate higher similarity.

To exclude the closest vectors from results, ensure that:

-1 <= radius < distance <= range_filter <= 1

COSINE

Larger cosine value indicates higher similarity.

To exclude the closest vectors from results, ensure that:

-1 <= radius < distance <= range_filter <= 1

JACCARD

Smaller Jaccard distances indicate higher similarity.

To exclude the closest vectors from results, ensure that:

0 <= radius < distance <= range_filter <= 1

HAMMING

Smaller Hamming distances indicate higher similarity.

To exclude the closest vectors from results, ensure that:

0 <= radius < distance <= range_filter <= dim(vector)

To learn more about distance metric types, refer to Similarity Metrics Explained.

In Zilliz Cloud, grouping search by a specific field can avoid redundancy of the same field item in the results. You can get a varied set of results for the specific field.

Consider a collection of documents, each document splits into various passages. Each passage is represented by one vector embedding and belongs to one document. To find relevant documents instead of similar passages, you can include the group_by_field argument in the search() opeartion to group results by the document ID. This helps return the most relevant and unique documents, rather than separate passages from the same document.

Here is the example code to group search results by field:

# Load data into collection
client.load_collection("group_search") # Collection name

# Group search results
res = client.search(
    collection_name="group_search", # Collection name
    data=[[0.14529211512077012, 0.9147257273453546, 0.7965055218724449, 0.7009258593102812, 0.5605206522382088]], # Query vector
    search_params={
    "metric_type": "L2",
    "params": {"level": 1},
    }, # Search parameters
    limit=10, # Max. number of search results to return
    group_by_field="doc_id", # Group results by document ID
    output_fields=["doc_id", "passage_id"]
)

# Retrieve the values in the `doc_id` column
doc_ids = [result['entity']['doc_id'] for result in res[0]]

print(doc_ids)

The output is similar to the following:

[5, 10, 1, 7, 9, 6, 3, 4, 8, 2]

In the given output, it can be observed that the returned entities do not contain any duplicate doc_id values.

For comparison, let's comment out the group_by_field and conduct a regular search:

# Load data into collection
client.load_collection("group_search") # Collection name

# Search without `group_by_field`
res = client.search(
    collection_name="group_search", # Collection name
    data=query_passage_vector, # Replace with your query vector
    search_params={
    "metric_type": "L2",
    "params": {"level": 1},
    }, # Search parameters
    limit=10, # Max. number of search results to return
    # group_by_field="doc_id", # Group results by document ID
    output_fields=["doc_id", "passage_id"]
)

# Retrieve the values in the `doc_id` column
doc_ids = [result['entity']['doc_id'] for result in res[0]]

print(doc_ids)

The output is similar to the following:

[1, 10, 3, 10, 1, 9, 4, 4, 8, 6]

In the given output, it can be observed that the returned entities contain duplicate doc_id values.

Limitations

  • Vector: Currently, grouping search does not support a vector field of the BINARY_VECTOR type. For more information on data types, refer to Schema Explained.

  • Field: Currently, grouping search allows only for a single column. You cannot specify multiple field names in the group_by_field config. Additionally, grouping search is incompatible with data types of JSON, FLOAT, DOUBLE, ARRAY, or vector fields.

  • Performance Impact: Be mindful that performance degrades with increasing query vector counts. Using a cluster with 2 CPU cores and 8 GB of memory as an example, the execution time for grouping search increases proportionally with the number of input query vectors.

  • Functionality: Grouping search is not supported by range search, search iterators, or multi-vector search.

Search parameters​

In the above searches except the range search and grouping search, the default search parameters apply. In normal cases, you do not need to manually set search parameters.

# In normal cases, you do not need to set search parameters manually
# Except for range searches.
search_parameters = {
    'metric_type': 'L2',
    'params': {
        'level': 1，
        'radius': 1.0
        'range_filter': 0.8
    }
}

The following table lists all possible settings in the search parameters.

Parameter Name

Parameter Description

metric_type

How to measure similarity between vector embeddings.

Possible values are IP, L2, COSINE, JACCARD, and HAMMING, and defaults to that of the loaded index file.

params.level

Search precision level.

Possible values are 1, 2, 3, 4, and 5, and defaults to 1. Higher values yield more accurate results but slower performance.

params.radius

Defines the outer boundary of your search space. Only vectors that are within this distance from the query vector are considered potential matches.

The value range is determined by the metric_type parameter. For instance, if metric_type is set to L2, the valid value range is [0, ∞]. If metric_type is set to COSINE, the valid value range is [-1, 1]. For more information, refer to Similarity Metrics Explained.

params.range_filter

While radius sets the outer limit of the search, range_filter can be optionally used to define an inner boundary, creating a distance range within which vectors must fall to be considered matches.

The value range is determined by the metric_type parameter. For instance, if metric_type is set to L2, the valid value range is [0, ∞]. If metric_type is set to COSINE, the valid value range is [-1, 1]. For more information, refer to Similarity Metrics Explained.

📘Notes

[1] Number of cluster units after indexing. When indexing a collection, Zilliz Cloud sub-divides the vector data into multiple cluster units, the number of which varies with the actual index settings.

[2] Number of entities to return in a search.