Skip to main content
Version: User Guides (Cloud)

Create an External Collection
Public Preview

An external collection is a type of data collection in Zilliz Cloud that accesses data from external storage systems or database tables such as AWS S3 and Iceberg without copying it into Zilliz Cloud. It acts as a query layer over data lakes while maintaining compatibility with Zilliz Cloud query interfaces.

Overview​

In a typical AI data pipeline, users may already have stored their data in Parquet or other formats on their storage system, such as AWS S3. To make Zilliz Cloud consume this externally stored data, users usually need to import it into Zilliz Cloud's own storage using Extract-Transform-Load (ETL) pipelines.

This bring-your-data-to-Zilliz Cloud workflow creates redundant data that is hard to synchronize and adds to the engineering maintenance burden to ensure data consistency.

YQXWwPQ3vheYa4b8398cWoPNnyN

To resolve these issues, Zilliz Cloud delivers external collections that let you access your externally stored data from Zilliz Cloud without worrying about data synchronization and ETL pipelines.

Q6F4wtcd2h3PnKbnMxncw3urn3f

Once created, an external collection can access your data directly and keep it in the same place where you store it. In the background, Zilliz Cloud creates manifest files to record the mappings between the Zilliz Cloud metadata and the rows in external data files. After the manifest files are ready, you can create indexes in the external collection as you would in any managed collection.

When your data changes, manually triggering a sub-second refresh updates the metadata, keeping Zilliz Cloud always up to date.

External collections are available in the databases for on-demand computing.

Step 1: Create schema​

As with creating a managed collection, you also need to create a schema before creating an external collection. However, the schema is slightly different from that of a managed collection.

Preparation​

  • You have obtained an API key with sufficient permissions to create an external collection in a database for on-demand computing.

    For details, refer to API Keys.

  • You have integrated your object storage bucket with Zilliz Cloud.

    For details, refer to AWS, GCP, and Azure docs in Integrate with Third Parties.

  • You have created an external volume out of the bucket integration. Ensure that the volume contains the target data files.

    For details, refer to External Volumes.

Support data sources​

Zilliz Cloud supports the following data sources, and you should provide the corresponding external sources in the format you choose.

  • parquet

    Set external_source to the folder containing the target Parquet files.

  • vortex,

    Set external_source to the folder containing the Vortex columnar files for version 0.56.

  • lance-table

    Set external_source to a folder path that contains sub-folders, such as _transactions, _versions, and data.

  • iceberg-table

    Set external_source to the metadata.json file of the Iceberg table, and pass in the snapshot ID, as in

    external_spec={
        "format": "iceberg-table",
        "snapshot_id": "473984310232959286"
    }

Set up schema​

Once you have an external volume containing the target data files, create the schema to map collection columns to Parquet files (parquet), a lance table (lance-table), an Iceberg table (iceberg-table), or Vortex files of the 0.56.0 format (vortex).

📘Notes

The external source should end with a forward slash (/) to indicate this is a folder.

from pymilvus import MilvusClient, DataType

schema = MilvusClient.create_schema(
    external_source='volume://my_volume/path/to/a/folder/',
    external_spec='{"format": "parquet"}'
)

Step 2: Add fields​

Once the schema is ready, you can add fields as follows:

schema.add_field(
    field_name="product_id",
    datatype=DataType.INT64,
    # highlight-next
    external_field="id" # field name in the external data file
)
schema.add_field(
    field_name="product_name",
    datatype=DataType.VARCHAR,
    max_length=512,
    # highlight-next
    external_field="name"
)
schema.add_field(
    field_name="embedding",
    datatype=DataType.FLOAT_VECTOR,
    dim=768,
    # highlight-next
    external_field="vector"
)

Step 3: Create a collection​

After adding all the fields to the schema, you can create the external collection.

📘Notes

You can create external collections in a database at the project level, which is usually associated with an on-demand cluster.

# connect the database
client = MilvusClient(
    uri="https://{project-id}.{region}.vectordb.zillizcloud.com",
    token="YOUR_API_KEY"
)

client.use_database(
    db_name="my_database"
)
# create the collection
client.create_collection(
    collection_name="test_collection",
    schema=schema
)

Step 4: Create indexes​

You can create indexes for external collection columns as you do in managed collections.

index_params = client.prepare_index_params()
# Add indexes
index_params.add_index(
    field_name="embedding",
    index_type="AUTOINDEX",
    metric_type="COSINE"
)
index_params.add_index(
    field_name="product_name",
    index_type="AUTOINDEX"
)
client.create_index(
    db_name="my_database",
    collection_name="test_collection",
    index_params=index_params
)

Step 5: Refresh data​

Once the collection is ready, refresh it to create the metadata and indexes for your data.

job_id = client.refresh_external_collection(
    db_name="my_database",
    collection_name="test_collection"
)
while True:
    progress = client.get_refresh_external_collection_progress(job_id=job_id)
    print(f"  {progress.state}: {progress.progress}%")
    if progress.state == "RefreshCompleted":
        elapsed = progress.end_time - progress.start_time
        print(f"  Completed in {elapsed}ms")
        break
    elif progress.state == "RefreshFailed":
        print(f"  Failed: {progress.reason}")
        break
    time.sleep(2)

The refresh operation is asynchronous, so you need to set up an iteration to monitor its progress.

📘Notes

  • The refresh operation scans the metadata of the data files and generates the manifest files accordingly. It usually takes 150-250 ms.

  • The manifest files record the mapping between the metadata in Milvus and the rows in external files.

  • If there is an update to your source data, you need to manually call refresh again to keep Zilliz Cloud up to date.

  • A refresh that requires removing all active metadata without any insertions results in a denial.

  • For external collections in a database for on-demand computing, you do not need to load and release them manually.

Follow-ups​

Once you have refreshed the external collection, you can perform similarity searches and queries in the external collection as you would in any managed collection, except that collections in a database for on-demand computing must be attached to an on-demand cluster for searches and queries. For details, refer to On-Demand Compute.

Before conducting DQL operations, such as search, query, get, and hybrid search, you need to create a session to attach the compute resources of an on-demand cluster. For details, refer to DQL Operations in External Collections.