Skip to main content

Alter Collection Schema

As a collection moves from development to production, its schema often changes. You might add scalar fields such as source_uri or review_status for filtering and application logic, add a new vector field for embeddings generated by your application, add a BM25 Function and its generated sparse vector field for lexical search over existing text, or remove fields and Functions that are no longer used. Alter Collection Schema lets you make supported field and Function changes in place instead of recreating the collection.

📘Notes

This guide covers schema changes for user-defined fields and for Functions with their generated vector fields in managed collections. To add a field to an external collection, refer to Alter External Collection Schema. For field property changes, such as changing max_length on a VARCHAR field or max_capacity on an ARRAY field, refer to Alter Collection Field. For dynamic field behavior, refer to Dynamic Field and Modify Collection.

Limits​

Add user-defined fields

  • Added user-defined fields must be nullable. Set nullable=True when calling add_collection_field(). For existing entities, the added field is NULL unless you add a scalar field with a default_value.

  • Adding user-defined scalar fields is supported in Milvus 2.6.x and later. Adding user-defined vector fields is supported in Milvus 2.6.18 and later.

  • Nullable StructArray fields and dynamic StructArray field addition are supported on Dedicated and On-Demand Clusters. A StructArray field added dynamically must be nullable.

  • Adding a sub-field to an existing StructArray field is supported on Dedicated Clusters compatible with Milvus v3.0.x; it is not yet available on On-Demand Clusters. The added sub-field must allow element-level null (nullable=True on a scalar or vector sub-field, element_nullable=True on an array sub-field), cannot have a default value, and cannot be bound to an index in the same request.

  • Field names must be unique among fields in the collection.

Add a Function and its generated vector field

  • Each schema update can add only one Function and one generated vector field.

  • The supported Function determines the generated vector field type: BM25 generates a SPARSE_FLOAT_VECTOR field, and MINHASH generates a BINARY_VECTOR field.

  • The generated vector field must be a new field. It cannot point to a field that already exists in the collection schema.

  • The generated vector field cannot be nullable.

  • The input fields used by the Function must already exist in the collection.

  • When adding a BM25 or MinHash Function to an existing collection, the Function input must be a VARCHAR field. A TEXT input is not supported in this workflow because Zilliz Cloud cannot backfill the generated output for existing entities from that input type.

Drop user-defined fields

  • You cannot drop the primary key field, partition key field, clustering key field, or the last vector field in a collection.

  • You can drop a whole ARRAY<STRUCT> field. On Dedicated Clusters compatible with Milvus v3.0.x, you can also drop an individual sub-field by its path, such as chunks[lang], except the last sub-field of the ARRAY<STRUCT> field; this capability is not yet available on On-Demand Clusters.

  • You cannot directly drop a field that is used as a Function input field or generated as a Function output field. To remove a Function output field, drop the Function that generates it.

Drop a Function and its generated vector field

  • In this schema-change workflow, dropping a Function removes the Function, its generated vector field, and the associated index. The Function input fields remain in the collection schema.

  • Dropping a Function is rejected if removing its generated vector field would leave the collection without any vector field.

📘Notes

For schema changes outside supported add and drop operations, recreate or migrate the collection.

Add fields and Functions to an existing collection​

Choose the workflow based on whether you are adding a user-defined field or a Function that generates a vector field:

In all cases, the new field name must not already exist in the collection, and the total number of fields cannot exceed the Zilliz Cloud field-count limit. For details, refer to Zilliz Cloud Limits.

Add user-defined scalar fields​

Use add_collection_field() to add a user-defined scalar field to an existing collection.

This differs from storing arbitrary keys in the dynamic field: after the schema update is available, the new scalar field becomes a regular part of the collection schema. You can insert or upsert values into it, create indexes on it where supported, use it in queries and search filters, and return it in query or search output.

Because existing entities were inserted before the new field existed, every added user-defined scalar field must be nullable:

  • If you add a scalar field with nullable=True and no default_value, existing entities return NULL for the new field.

  • If you add a scalar field with nullable=True and default_value, existing entities return the default value instead of NULL.

Scalar filter expressions do not match NULL scalar values. For details, refer to Nullable Fields.

Example: Add a nullable scalar field

The following example adds a nullable source field to an existing collection named product_catalog.

python
from pymilvus import DataType, MilvusClient, Function, FunctionType

client = MilvusClient(uri="YOUR_CLUSTER_ENDPOINT", token="YOUR_CLUSTER_TOKEN")

client.add_collection_field(
    collection_name="product_catalog",
    field_name="source",
    data_type=DataType.VARCHAR,
    max_length=128,
    nullable=True,
)

After the field is added, entities that already existed in the collection return NULL for source. New entities can set source during insert or upsert.

Example: Add a scalar field with a default value

If existing entities should return a concrete value instead of NULL, specify default_value when adding a scalar field. The following example adds a review_status field and uses "unreviewed" as the default value.

python
client.add_collection_field(
    collection_name="product_catalog",
    field_name="review_status",
    data_type=DataType.VARCHAR,
    max_length=32,
    nullable=True,
    default_value="unreviewed",
)

After the field is added, entities that already existed in the collection return "unreviewed" for review_status. New entities can set a different value or use the default value when no value is provided.

Add StructArray fields​

Public Preview

Use add_collection_struct_field() to add a StructArray field that accepts arrays of Struct elements. To add a StructArray field, do as follows:

  1. Create a Struct schema that contains the necessary sub-fields of supported data types. For applicable data types, refer to StructArray Limits.

  2. Reference the Struct schema created above and set the maximum capacity of the field in add_collection_struct_field().

  3. Set nullable=True in the request.

Example: Add a nullable StructArray field

python
# Create a struct schema
struct_schema = client.create_struct_field_schema()

# add a scalar field to the struct
struct_schema.add_field("text", DataType.VARCHAR, max_length=65535)
struct_schema.add_field("chapter", DataType.VARCHAR, max_length=512)

# add a vector field to the struct with mmap enabled
struct_schema.add_field("text_vector", DataType.FLOAT_VECTOR, mmap_enabled=True, dim=5)
struct_schema.add_field("chapter_vector", DataType.FLOAT_VECTOR, mmap_enabled=True, dim=5)

client.add_collection_struct_field(
    collection_name="books",
    field_name="chunks",
    struct_schema=struct_schema,
    max_capacity=1024,
    nullable=True
)

After the StructArray field is added, entities that already exist in the collection return NULL for chunks across all its sub-fields. When you insert a new entity, every Struct element must provide a value for every sub-field, and a Struct element itself cannot be NULL. On Dedicated Clusters compatible with Milvus v3.0.x, element-nullable sub-fields may hold NULL in individual Struct elements; for details, refer to Create a StructArray Field.

Add a sub-field to an existing StructArray field​

Public Preview

Use add_collection_field() with the sub-field path as the field name to add a sub-field to a StructArray field that already holds data. This workflow is supported on Dedicated Clusters compatible with Milvus v3.0.x; it is not yet available on On-Demand Clusters. To add a sub-field, do as follows:

  1. Name the field with the path structArrayField[subField], such as chunks[lang].

  2. Choose a scalar, vector, or scalar array data type supported by StructArray.

  3. Set nullable=True on the added sub-field. This is required for every added sub-field and is what allows element-level null. On an array sub-field, additionally set element_nullable=True to allow null values inside the inner array. The sub-field inherits the row-level nullable setting and the max_capacity of the StructArray field.

Example: Add an element-nullable sub-field

python
from pymilvus import DataType, MilvusClient

client = MilvusClient(uri="YOUR_CLUSTER_ENDPOINT")

client.add_collection_field(
    collection_name="books",
    field_name="chunks[lang]",
    data_type=DataType.VARCHAR,
    max_length=16,
    nullable=True,
)

After the sub-field is added, entities that already exist return NULL for chunks[lang] in each of their Struct elements, and entities whose chunks field is NULL stay NULL. New entities can omit the sub-field or set it to NULL in any Struct element.

The following rules apply:

  • The added sub-field must be created with nullable=True, and an added array sub-field additionally takes element_nullable=True to allow null values inside the inner array. It cannot have a default value.

  • You cannot bind an index while adding the sub-field. Create the index with a separate create_index() call. An added vector sub-field supports element-level metrics only, not MAX_SIM*, and the collection must have an index on it before you load the collection with the new sub-field included.

  • History segments receive the added sub-field and its index through schema-version compaction. Set dataCoord.compaction.bumpSchemaVersion.enabled to true (it is false by default); otherwise the added sub-field's index does not cover history segments.

  • Adding a sub-field to a loaded collection does not release the collection. Queries can return the new sub-field before its index exists.

  • Clients that pin a schema timestamp receive a schema-mismatch error and must refresh, as for any added field.

Add user-defined vector fields​

Use add_collection_field() to add a user-defined vector field when your application generates embeddings and writes vector values to Zilliz Cloud.

Every added user-defined vector field must be nullable. Existing entities have NULL for the new vector field until you write vector values through upsert or a backfill workflow. New entities can include the vector field during insert. Vector search skips entities whose vector value is NULL. For details, refer to Nullable Fields.

Example: Add a nullable vector field

The following example adds a nullable dense vector field named embedding_v2 to an existing collection. Set dim to the dimensionality of the embeddings generated by your application.

python
client.add_collection_field(
    collection_name="product_catalog",
    field_name="embedding_v2",
    data_type=DataType.FLOAT_VECTOR,
    dim=768,
    nullable=True,
)

After the field is added, create an index on the new vector field before searching it:

python
index_params = client.prepare_index_params()

index_params.add_index(
    field_name="embedding_v2",
    index_type="AUTOINDEX",
    metric_type="COSINE",
)

client.create_index(
    collection_name="product_catalog",
    index_params=index_params,
)

Existing entities have NULL for embedding_v2 and are skipped when you search on this field. To make existing entities searchable through embedding_v2, write non-NULL vector values through upsert or a backfill workflow. New entities can include embedding_v2 during insert.

Add a Function and its generated vector field​

Public Preview

Use this workflow when Zilliz Cloud should generate a new vector field from data already stored in an existing collection. The operation adds three related schema elements:

  • A Function definition that reads from one or more existing input fields.

  • A new vector field that stores the Function output.

  • An index definition bound to the new vector field.

For example, a BM25 Function reads an existing VARCHAR field and generates a SPARSE_FLOAT_VECTOR field for lexical search. A MinHash Function generates a BINARY_VECTOR field for near-duplicate detection. This workflow does not add or replace the Function input field.

This schema-change workflow is available on Zilliz Cloud On-Demand Clusters.

The supported Function determines the generated vector field type:

FunctionGenerated vector field typeTypical input fieldTypical use case
BM25SPARSE_FLOAT_VECTORA VARCHAR field with analyzer enabledLexical search and keyword relevance
MINHASHBINARY_VECTORA VARCHAR fieldNear-duplicate detection

For details about how each Function works, refer to BM25 Function and MinHash Function.

The generated vector field must not already exist in the collection, and it cannot be nullable. The Function input field must already exist.

Example: Add a BM25 Function and its generated sparse vector field

The following example adds a BM25 Function named text_bm25 and its generated sparse vector field named text_sparse to an existing collection. The collection must already have a VARCHAR field named text with analyzer enabled.

python
sparse_field = client.create_field_schema(
    name="text_sparse",
    data_type=DataType.SPARSE_FLOAT_VECTOR,
    desc="BM25-generated sparse vector field",
)

bm25_function = Function(
    name="text_bm25",
    input_field_names=["text"],
    output_field_names=["text_sparse"],
    function_type=FunctionType.BM25,
)

index_params = client.prepare_index_params()

index_params.add_index(
    field_name="text_sparse",
    index_type="SPARSE_INVERTED_INDEX",
    metric_type="BM25",
    params={
        "inverted_index_algo": "DAAT_MAXSCORE",
        "bm25_k1": 1.2,
        "bm25_b": 0.75,
    },
)

client.add_function_field(
    collection_name="product_catalog",
    field_schema=sparse_field,
    func=bm25_function,
    index_params=index_params,
)

The index_params object must contain exactly one index definition for the new Function output field. Zilliz Cloud adds the Function, its generated vector field, and the bound index definition in the same schema change. Do not call create_index() separately after add_function_field().

Conceptually, this operation adds the following Function, generated output field, and bound index definitions:

plaintext
New Function:
  name: "text_bm25"
  type: BM25
  input_field_names: ["text"]
  output_field_names: ["text_sparse"]

New generated output field:
  name: "text_sparse"
  data_type: SPARSE_FLOAT_VECTOR
  nullable: false

Bound index:
  field_name: "text_sparse"
  index_type: SPARSE_INVERTED_INDEX
  metric_type: BM25

After the request succeeds, describe_collection() returns both the new text_bm25 Function and its generated text_sparse vector field in the collection schema. Zilliz Cloud generates the Function output for new entities as they are written. For existing entities, Zilliz Cloud populates the generated vector field asynchronously through background compaction. Schema visibility confirms that the schema update succeeded, but does not indicate that backfill has completed for every existing entity. For the complete BM25 search workflow, refer to Full Text Search.

Zilliz Cloud also supports MinHash Functions and their generated binary vector fields for near-duplicate detection. A MinHash Function uses FunctionType.MINHASH and writes to a new BINARY_VECTOR output field. For configuration details, refer to MinHash Function.

Drop fields and Functions from an existing collection​

You can remove user-defined fields directly when they are no longer part of your collection model. To remove a Function and its generated vector field, drop the Function; Zilliz Cloud removes the generated field and its index in the same schema change.

Drop user-defined fields​

Public Preview

Use drop_collection_field() to remove a user-defined scalar, vector, or StructArray field that is no longer part of your collection model.

Dropping a field first changes the collection schema and field visibility:

  • After drop_collection_field() succeeds, the collection schema is updated: describe_collection() no longer returns the dropped field, and queries or searches can no longer return the field in output_fields or use it in expressions.

  • Indexes built on the dropped field are cleaned up as part of the schema update.

Storage cleanup is handled separately from schema cleanup. For details, refer to When is storage space reclaimed after dropping a field?.

Example: Drop a user-defined scalar field

The following example assumes that experiment_tag is a user-defined scalar field in product_catalog, and drops it from the collection.

python
client.drop_collection_field(
    collection_name="product_catalog",
    field_name="experiment_tag",
)

After dropping a field, you can call describe_collection() to verify that the field is no longer part of the schema.

Example: Drop a StructArray field

The following example assumes that chunks is a StructArray field in my_collection, and drops it from the collection.

python
client.drop_collection_field(
    collection_name="my_collection",
    field_name="chunks",
)

Example: Drop a user-defined vector field

You can drop a vector field with the same drop_collection_field() method, but the collection must still contain at least one vector field after the drop. This is useful for collections that temporarily carry multiple vector representations and later standardize on one of them.

The following example assumes that image_vector is a user-defined vector field in hybrid_catalog, and that the collection still retains another vector field, such as text_vector.

python
client.drop_collection_field(
    collection_name="hybrid_catalog",
    field_name="image_vector",
)

If image_vector is the last vector field in the collection, the drop operation is rejected.

Drop a StructArray sub-field​

Public Preview

On Dedicated Clusters compatible with Milvus v3.0.x, you can drop an individual sub-field from a StructArray field with drop_collection_field() by passing the sub-field path; this capability is not yet available on On-Demand Clusters. Zilliz Cloud removes the sub-field from the schema together with its index; its data is reclaimed as described in When is storage space reclaimed after dropping a field?.

The following example assumes that chunks is a StructArray field in my_collection with a lang sub-field, and drops only that sub-field.

python
from pymilvus import MilvusClient

client = MilvusClient(uri="YOUR_CLUSTER_ENDPOINT")

client.drop_collection_field(
    collection_name="my_collection",
    field_name="chunks[lang]",
)

You cannot drop the last sub-field of a StructArray field, the last vector field of the collection, or a sub-field used by a Function. To remove the whole StructArray field, drop it by name, as shown in the previous example. After a sub-field is dropped, insert and upsert requests that still carry the sub-field are rejected.

Dropping a sub-field while queries are still running against it can fail transiently: query plans that were built on the older schema keep executing for a short time and fail with an internal error. Finish queries against a sub-field before dropping it, or retry the drop after the running queries finish.

Drop a Function and its generated vector field​

Public Preview

Use this operation when you no longer need a Function or its generated vector field, such as a BM25 Function and its generated sparse vector field.

Call drop_function_field() with the Function name. Zilliz Cloud removes the Function, its generated vector field, and the associated index while preserving the Function input fields.

Example: Drop a BM25 Function and its generated sparse vector field

The following example assumes that text_bm25 is a BM25 Function in product_catalog and generates a sparse vector output field named text_sparse.

python
client.drop_function_field(
    collection_name="product_catalog",
    function_name="text_bm25",
)

After the operation succeeds, describe_collection() no longer returns the dropped Function or its generated vector field. The Function input fields remain in the schema.

If removing the Function output field would leave the collection without any vector field, the operation is rejected.

FAQ​

Which method should I use to add a field or Function?​

Use add_collection_field() to add a user-defined scalar or vector field.

Use add_collection_struct_field() to add a StructArray field when you need an array field whose elements share the same Struct schema.

Use add_function_field() to add a Function, its generated vector field, and the bound index definition in the same schema change.

Why must added user-defined fields be nullable?​

Existing entities were inserted before the new field existed, so they do not have values for that field. Setting nullable=True lets Zilliz Cloud represent the missing value as NULL until your application writes a value or, for scalar fields, until a default value applies.

This rule applies to user-defined scalar fields and user-defined vector fields added with add_collection_field(), and to StructArray fields added with add_collection_struct_field(). It does not apply to a Function's generated vector field, which cannot be nullable.

What happens to existing entities after I add a user-defined field?​

For a user-defined scalar field, existing entities return NULL unless you set a default_value. If you set a default_value, existing entities return that default value.

For a user-defined vector field, existing entities have NULL for the new vector field. Vector search on the added field skips entities whose vector value is NULL. To make existing entities searchable through the new vector field, write non-NULL vector values through upsert or a backfill workflow. New entities can include the new vector field during insert.

For a StructArray field, existing entities return NULL for the new StructArray field across all its sub-fields. New entities must provide either NULL for the whole field or a valid value for every sub-field in every Struct element, except element-nullable sub-fields on Dedicated Clusters compatible with Milvus v3.0.x (not yet available on On-Demand Clusters). For a sub-field added to an existing StructArray field, existing entities return NULL for that sub-field in each Struct element.

Can I add BM25 lexical search to an existing collection?​

Yes. If the collection already has a VARCHAR field with analyzer enabled, you can add a BM25 Function and its generated sparse vector field for lexical search. In this workflow, Zilliz Cloud adds the Function, the new SPARSE_FLOAT_VECTOR output field, and the bound index definition in the same schema change. You cannot use an existing TEXT field as the BM25 input in this schema-change workflow. To use a TEXT input, define the field and BM25 Function when you create the collection.

When calling add_function_field(), provide an index_params object that contains one SPARSE_INVERTED_INDEX index with metric_type="BM25" for the new output field. Zilliz Cloud binds the index definition to the generated field as part of the same schema change.

How do I drop a Function and its generated vector field?​

Call drop_function_field() with the Function name. In this schema-change workflow, Zilliz Cloud removes the Function, its generated vector field, and the associated index together while preserving the Function input fields.

Do I need to wait after altering a collection schema?​

Usually, no manual wait is required. If your next operation depends on the updated schema, you can call describe_collection() first to confirm the schema that Zilliz Cloud currently returns.

In a distributed deployment, there can be a short propagation window while Zilliz Cloud components refresh collection metadata. If an operation immediately after the schema change fails with a schema-related error, refresh the schema and retry the operation.

Why does an added vector sub-field fail with a FieldNotLoaded error?​

After you add a vector sub-field and create its index, queries against the new sub-field can still fail with error 2027 (FieldNotLoaded) even though the collection reports ready. Release the collection and load it again; queries then cover the new sub-field.

When is storage space reclaimed after dropping a field?​

Dropping a field removes it from the current schema and normal query/search visibility, but historical data for that field is not physically deleted from object storage immediately.

Storage space can be reclaimed later during compaction. Compaction is a background process that reorganizes existing data files into new, more compact files. After a field is dropped, newly compacted files follow the current schema and omit the dropped field. Zilliz Cloud does not guarantee an immediate or fixed-time storage-space reduction after dropping a field.

What happens if I add a scalar field with the same name as a dynamic field key?​

If dynamic field is enabled, you can add a scalar field with the same name as an existing dynamic field key. The new scalar field masks the dynamic field key in normal query output, but the original dynamic data is preserved in $meta.

For example, if existing entities store a dynamic key named source, and you later add a scalar field named source, normal output for source refers to the scalar field. To access the original dynamic value, use the $meta path syntax, such as $meta["source"].

Contact Sales
Was this page helpful?
Ctrl I