Skip to main content

VarChar Field

Zilliz Cloud supports textual scalar data with VARCHAR and TEXT fields. This page describes VARCHAR, which is designed for short, bounded string metadata such as names, tags, categories, and external IDs.

For longer source text, document passages, article bodies, tickets, or logs that should be stored and returned with entities, use a TEXT field instead. Use TEXT when a value may exceed 65,535 bytes or when you do not want to set a fixed max_length in the collection schema. For details, refer to Text Field.

When you define a VARCHAR field, two parameters are mandatory:

  • Set the datatype to DataType.VARCHAR.

  • Specify the max_length, which defines the maximum number of UTF-8 bytes allowed for each field value. The valid range for max_length is from 1 to 65,535.

The number of bytes can differ from the number of characters. For example, with max_length=10, "abcdefghij" fits within the limit because it uses 10 UTF-8 bytes. The string "一二三四五六七八九十" also has 10 characters, but uses 30 UTF-8 bytes and exceeds the limit.

📘Notes

Zilliz Cloud supports NULL values and default values for VARCHAR fields. To enable these features, enable nullable and set default_value to a string value. For details, refer to Nullable & Default.

Add VARCHAR field​

To store short, bounded string metadata in Zilliz Cloud, define a VARCHAR field in your collection schema. Below is an example of defining a collection schema with two VARCHAR fields:

  • varchar_field1: stores up to 100 bytes, allows NULL values, and has a default value of "Unknown".

  • varchar_field2: stores up to 200 bytes, allows NULL values, but does not have a default value.

📘Notes

If you enable enable_dynamic_field when defining the schema, Zilliz Cloud allows you to insert scalar fields that were not defined in advance. However, this may increase the complexity of queries and management, potentially impacting performance. For more information, refer to Dynamic Field.

python
# Import necessary libraries
from pymilvus import MilvusClient, DataType

# Define server address
SERVER_ADDR = "YOUR_CLUSTER_ENDPOINT"

# Create a MilvusClient instance
client = MilvusClient(uri=SERVER_ADDR, token="YOUR_CLUSTER_TOKEN")

# Define the collection schema
schema = client.create_schema(
    auto_id=False,
    enable_dynamic_field=True,
)

# Add `varchar_field1` that supports NULL values with default value "Unknown"
schema.add_field(field_name="varchar_field1", datatype=DataType.VARCHAR, max_length=100, nullable=True, default_value="Unknown")
# Add `varchar_field2` that supports NULL values without default value
schema.add_field(field_name="varchar_field2", datatype=DataType.VARCHAR, max_length=200, nullable=True)
schema.add_field(field_name="pk", datatype=DataType.INT64, is_primary=True)
schema.add_field(field_name="embedding", datatype=DataType.FLOAT_VECTOR, dim=3)

Set index params​

Indexing helps improve search and query performance. In Zilliz Cloud, indexing is mandatory for vector fields but optional for scalar fields.

The following example creates indexes on the vector field embedding and the scalar field varchar_field1, both using the AUTOINDEX index type. With this type, Zilliz Cloud automatically selects the most suitable index based on the data type. You can also customize the index type and params for each field. For details, refer to Index Explained.

📘Notes

You can also build an NGRAM index to accelerate LIKE filtering on VARCHAR fields. For details, refer to NGRAM.

python
# Set index params

index_params = client.prepare_index_params()

# Index `varchar_field1` with AUTOINDEX
index_params.add_index(
    field_name="varchar_field1",
    index_type="AUTOINDEX",
    index_name="varchar_index"
)

# Index `embedding` with AUTOINDEX and specify metric_type
index_params.add_index(
    field_name="embedding",
    index_type="AUTOINDEX",  # Use automatic indexing to simplify complex index settings
    metric_type="COSINE"  # Specify similarity metric type, options include L2, COSINE, or IP
)

Create collection​

Once the schema and index are defined, create a collection that includes string fields.

python
# Create Collection
client.create_collection(
    collection_name="my_collection",
    schema=schema,
    index_params=index_params
)

Insert data​

After creating the collection, insert entities that match the schema.

python
# Sample data
data = [
    {"varchar_field1": "Product A", "varchar_field2": "High quality product", "pk": 1, "embedding": [0.1, 0.2, 0.3]},
    {"varchar_field1": "Product B", "pk": 2, "embedding": [0.4, 0.5, 0.6]}, # varchar_field2 field is missing, which should be NULL
    {"varchar_field1": None, "varchar_field2": None, "pk": 3, "embedding": [0.2, 0.3, 0.1]},  # `varchar_field1` should default to `Unknown`, `varchar_field2` is NULL
    {"varchar_field1": "Product C", "varchar_field2": None, "pk": 4, "embedding": [0.5, 0.7, 0.2]},  # `varchar_field2` is NULL
    {"varchar_field1": None, "varchar_field2": "Exclusive deal", "pk": 5, "embedding": [0.6, 0.4, 0.8]},  # `varchar_field1` should default to `Unknown`
    {"varchar_field1": "Unknown", "varchar_field2": None, "pk": 6, "embedding": [0.8, 0.5, 0.3]},  # `varchar_field2` is NULL
    {"varchar_field1": "", "varchar_field2": "Best seller", "pk": 7, "embedding": [0.8, 0.5, 0.3]}, # Empty string is not treated as NULL
]

# Insert data
client.insert(
    collection_name="my_collection",
    data=data
)

Query with filter expressions​

After inserting entities, use the query method to retrieve entities that match the specified filter expressions.

To retrieve entities where the varchar_field1 matches the string "Product A":

python
# Filter `varchar_field1` with value "Product A"
filter = 'varchar_field1 == "Product A"'

res = client.query(
    collection_name="my_collection",
    filter=filter,
    output_fields=["varchar_field1", "varchar_field2"]
)

print(res)

# Example output:
# data: [
#     "{'varchar_field1': 'Product A', 'varchar_field2': 'High quality product', 'pk': 1}"
# ]

To retrieve entities where the varchar_field2 is NULL:

python
# Filter entities where `varchar_field2` is null
filter = 'varchar_field2 is null'

res = client.query(
    collection_name="my_collection",
    filter=filter,
    output_fields=["varchar_field1", "varchar_field2"]
)

print(res)

# Example output:
# data: [
#     "{'varchar_field1': 'Product B', 'varchar_field2': None, 'pk': 2}",
#     "{'varchar_field1': 'Unknown', 'varchar_field2': None, 'pk': 3}",
#     "{'varchar_field1': 'Product C', 'varchar_field2': None, 'pk': 4}",
#     "{'varchar_field1': 'Unknown', 'varchar_field2': None, 'pk': 6}"
# ]

To retrieve entities where varchar_field1 has the value "Unknown", use the following expression below. As the default value of varchar_field1 is "Unknown", the expected result should include entities with varchar_field1 explicitly set to "Unknown" or with varchar_field1 set to NULL.

python
# Filter entities with `varchar_field1` with value `Unknown`
filter = 'varchar_field1 == "Unknown"'

res = client.query(
    collection_name="my_collection",
    filter=filter,
    output_fields=["varchar_field1", "varchar_field2"]
)

print(res)

# Example output:
# data: [
#     "{'varchar_field1': 'Unknown', 'varchar_field2': None, 'pk': 3}",
#     "{'varchar_field1': 'Unknown', 'varchar_field2': 'Exclusive deal', 'pk': 5}",
#     "{'varchar_field1': 'Unknown', 'varchar_field2': None, 'pk': 6}"
# ]

Vector search with filter expressions​

In addition to basic scalar field filtering, you can combine vector similarity searches with scalar field filters. For example, the following code shows how to add a scalar field filter to a vector search:

python
# Search with string filtering

# Filter `varchar_field2` with value "Best seller"
filter = 'varchar_field2 == "Best seller"'

res = client.search(
    collection_name="my_collection",
    data=[[0.3, -0.6, 0.1]],
    limit=5,
    search_params={"params": {"nprobe": 10}},
    output_fields=["varchar_field1", "varchar_field2"],
    filter=filter
)

print(res)

# Example output:
# data: [
#     "[{'id': 7, 'distance': -0.04468163847923279, 'entity': {'varchar_field1': '', 'varchar_field2': 'Best seller'}}]"
# ]
Was this page helpful?
Ctrl I