Skip to main content

Full Text Search

Full text search retrieves documents that contain the terms or phrases in your query and ranks the results by relevance. It complements semantic search, which can overlook precise terms, and is especially useful in retrieval-augmented generation (RAG) scenarios, where documents that closely match specific search terms matter most.

With full text search, you can also skip embedding models: you provide raw text, and Zilliz Cloud converts it into sparse embeddings automatically.

📘Notes

By integrating full text search with semantic-based dense vector search, you can enhance the accuracy and relevance of search results. For more information, refer to Hybrid Search.

Zilliz Cloud supports enabling full text search programmatically or via the web console. This page focuses on how to enable full text search programmatically. For details about operations on the web console, refer to Manage Collections (Console).

BM25 implementation​

Zilliz Cloud implements full text search with the BM25 relevance algorithm, a widely adopted scoring function in information retrieval systems, and integrates it into the search workflow to deliver accurate, relevance-ranked text results.

Full text search in Zilliz Cloud follows the workflow below:

  1. Raw text input: You insert text documents or provide a query using plain text, no embedding models required.

  2. Text analysis: Zilliz Cloud uses an analyzer to process your text into meaningful terms that can be indexed and searched.

  3. BM25 function processing: A built-in function transforms these terms into sparse vector representations optimized for BM25 scoring.

  4. Collection store: Zilliz Cloud stores the resulting sparse embeddings in a collection for fast retrieval and ranking.

  5. BM25 relevance scoring: At search time, Zilliz Cloud applies the BM25 scoring function to compute document relevance and return ranked results that best match the query terms.

Full Text Search

To use full text search, follow these main steps:

  1. Create a collection: Set up the required fields and define a BM25 function that converts raw text into sparse embeddings.

  2. Insert data: Ingest your raw text documents to the collection.

  3. Perform searches: Use natural-language query text to retrieve ranked results based on BM25 relevance.

To enable BM25-powered full text search, you must prepare a collection with the required fields, define a BM25 function to generate sparse vectors, configure an index, and then create the collection.

Define schema fields​

Your collection schema must include at least three required fields:

  • Primary field: Uniquely identifies each entity in the collection.

  • String field (VARCHAR or TEXT): Stores raw text documents. You must set enable_analyzer=True so that Zilliz Cloud can process the text for BM25 relevance ranking. By default, Zilliz Cloud uses the standard analyzer; to configure a different analyzer, refer to Analyzer Overview. The examples on this page use VARCHAR; for long text, define the field as TEXT and omit max_length. For a complete example, refer to Text Field.

  • Sparse vector field (SPARSE_FLOAT_VECTOR): Stores sparse embeddings automatically generated by the BM25 function.

python
from pymilvus import MilvusClient, DataType, Function, FunctionType

client = MilvusClient(
uri="YOUR_CLUSTER_ENDPOINT",
token="YOUR_CLUSTER_TOKEN"
)

schema = client.create_schema()

schema.add_field(field_name="id", datatype=DataType.INT64, is_primary=True, auto_id=True) # Primary field
schema.add_field(field_name="text", datatype=DataType.VARCHAR, max_length=1000, enable_analyzer=True) # Text field
schema.add_field(field_name="sparse", datatype=DataType.SPARSE_FLOAT_VECTOR) # Sparse vector field; no dim required for sparse vectors

In the preceding config,

  • id: serves as the primary key and is automatically generated with auto_id=True.

  • text: stores your raw text data for full text search operations. The field can use VARCHAR for bounded text or TEXT for long source content.

  • sparse: a vector field reserved to store internally generated sparse embeddings for full text search operations. The data type must be SPARSE_FLOAT_VECTOR.

Define the BM25 function​

The BM25 function converts tokenized text into sparse vectors that support BM25 scoring.

Define the function and add it to your schema:

python
bm25_function = Function(
name="text_bm25_emb", # Function name
input_field_names=["text"], # Name of the VARCHAR or TEXT field containing raw text data
output_field_names=["sparse"], # Name of the SPARSE_FLOAT_VECTOR field reserved to store generated embeddings
function_type=FunctionType.BM25, # Set to `BM25`
)

schema.add_function(bm25_function)

Parameter

Description

name

The name of the function. This function converts your raw text from the text field into BM25-compatible sparse vectors that will be stored in the sparse field.

input_field_names

The name of the VARCHAR or TEXT field requiring text-to-sparse-vector conversion. For FunctionType.BM25, this parameter accepts only one field name.

output_field_names

The name of the field where the internally generated sparse vectors will be stored. For FunctionType.BM25, this parameter accepts only one field name.

function_type

The type of the function to use. Must be FunctionType.BM25.

📘Notes

If multiple text fields require BM25 processing, define one BM25 function per field, each with a unique name and output field.

Configure the index​

After defining the schema with necessary fields and the built-in function, set up the index for your collection. To simplify this process, use AUTOINDEX as the index_type, an option that allows Zilliz Cloud to choose and configure the most suitable index type based on the structure of your data.

python
index_params = client.prepare_index_params()

index_params.add_index(
field_name="sparse",

index_type="SPARSE_INVERTED_INDEX",
metric_type="BM25",
params={
"inverted_index_algo": "DAAT_MAXSCORE",
"bm25_k1": 1.2,
"bm25_b": 0.75
}

)

Parameter

Description

field_name

The name of the vector field to index. For full text search, this should be the field that stores the generated sparse vectors. In this example, set the value to sparse.

index_type

The type of the index to create. AUTOINDEX allows Zilliz Cloud to automatically optimize index settings. If you need more control over your index settings, you can choose from various index types available for sparse vectors in Zilliz Cloud. .

metric_type

The value for this parameter must be set to BM25 specifically for full text search functionality.

params

A dictionary of additional parameters specific to the index.

params.inverted_index_algo

The algorithm used for building and querying the index. Valid values:

  • "DAAT_MAXSCORE" (default): Optimized Document-at-a-Time (DAAT) query processing using the MaxScore algorithm. MaxScore provides better performance for high k values or queries with many terms by skipping terms and documents likely to have minimal impact. It achieves this by partitioning terms into essential and non-essential groups based on their maximum impact scores, focusing on terms that can contribute to the top-k results.

  • "DAAT_WAND": Optimized DAAT query processing using the WAND algorithm. WAND evaluates fewer hit documents by leveraging maximum impact scores to skip non-competitive documents, but it has a higher per-hit overhead. This makes WAND more efficient for queries with small k values or short queries, where skipping is more feasible.

  • "TAAT_NAIVE": Basic Term-at-a-Time (TAAT) query processing. While it is slower compared to DAAT_MAXSCORE and DAAT_WAND, TAAT_NAIVE offers a unique advantage. Unlike DAAT algorithms, which use cached maximum impact scores that remain static regardless of changes to the global collection parameter (avgdl), TAAT_NAIVE dynamically adapts to such changes.

params.bm25_k1

Controls the term frequency saturation. Higher values increase the importance of term frequencies in document ranking. Value range: [1.2, 2.0].

params.bm25_b

Controls the extent to which document length is normalized. Values between 0 and 1 are typically used, with a default value of 0.75. A value of 0 means no length normalization, while a value of 1 means full length normalization.

Create the collection​

Now create the collection using the schema and index parameters defined.

python
client.create_collection(
collection_name='my_collection',
schema=schema,
index_params=index_params
)

client.load_collection('my_collection')

Insert text data​

After setting up your collection and index, you can insert text data. You only need to provide the raw text — the BM25 function defined earlier automatically generates the corresponding sparse vector for each text entry.

python
client.insert('my_collection', [
{'text': 'information retrieval is a field of study.'},
{'text': 'information retrieval focuses on finding relevant information in large datasets.'},
{'text': 'data mining and information retrieval overlap in research.'},
])

Once you've inserted data into your collection, you can search it with raw text queries. Zilliz Cloud automatically converts your query into a sparse vector, ranks the matched results using the BM25 algorithm, and returns the top limit results.

📘Notes

You can highlight the matched terms in search results by configuring a text highlighter. See Text Highlighter for details.

python
res = client.search(
collection_name='my_collection',
data=['whats the focus of information retrieval?'],
anns_field='sparse',
output_fields=['text'], # Fields to return in search results; sparse field cannot be output
limit=3,
search_params={"metric_type": "BM25", "params": {"level": 10}},
)

print(res)
ParameterDescription
search_paramsA dictionary containing search parameters.
params.levelControls the search precision with simplified search optimization. For details, refer to Tune Recall Rate.
dataRaw query text in natural language. Zilliz Cloud automatically converts your text query into sparse vectors using the BM25 function - do not provide pre-computed vectors.
anns_fieldThe name of the field that contains internally generated sparse vectors.
output_fieldsList of field names to return in search results. Supports all fields except the sparse vector field containing BM25-generated embeddings. Common output fields include the primary key field (e.g., id) and the original text field (e.g., text). For more information, refer to FAQ.
limitMaximum number of top matches to return.

FAQ​

No, the sparse vectors generated by the BM25 function are not directly accessible or returnable in full text search. Here are the details:

  • The BM25 function generates sparse vectors internally for ranking and retrieval.

  • These vectors are stored in the sparse field but cannot be included in output_fields.

  • You can only output the original text fields and metadata (like id, text).

Example:

python
search_params = {"metric_type": "BM25"}

# ❌ This throws an error - you cannot output the sparse field
client.search(
collection_name='my_collection',
data=['query text'],
anns_field='sparse',
output_fields=['text', 'sparse'], # 'sparse' causes an error
limit=3,
search_params=search_params
)

# ✅ This works - output text fields only
client.search(
collection_name='my_collection',
data=['query text'],
anns_field='sparse',
output_fields=['text'],
limit=3,
search_params=search_params
)

Why do I need to define a sparse vector field if I can't access it?​

The sparse vector field serves as an internal search index, similar to database indexes that users don't directly interact with.

Design rationale:

  • Separation of concerns: You work with text for input and output, and Zilliz Cloud handles the vectors internally.

  • Performance: Pre-computed sparse vectors enable fast BM25 ranking during queries.

  • User experience: Complex vector operations are abstracted away behind a simple text interface.

If you need vector access:

  • Use manual sparse vector operations instead of full text search.

  • Create separate collections for custom sparse vector workflows.

For details, refer to Sparse Vector.