Skip to main content
Version: User Guides (Cloud)

External Volumes

An external volume is a read-only reference to a bucket or path in your own cloud object storage (such as AWS S3 or Google Cloud Storage), allowing Zilliz Cloud to access your data in place without copying or moving it.

This page explains how to create and delete external volumes via the web console and SDKs.

Considerations​

  • Volumes are available on AWS and Google Cloud only. For Azure, contact support.

  • A volume is restricted to your project’s cloud provider and region. For example, if your project is in AWS us-west-2, you can create volumes only in AWS us-west-2.

  • To use a volume with a cluster, the cluster must be in the same cloud provider and region as the volume.

  • To create and manage volumes, you need to be a Project Admin.

  • You cannot edit the configurations of a volume once it is created. If you want to change the volume settings, please create a new volume with the desired settings instead.

  • For external volumes, data stays in your bucket. Therefore, you need to manage data files in your cloud object storage rather than on the external volume.

  • Each organization can create up to 100 external volumes.

Before you start​

Before creating an external volume, you need to integrate your AWS S3 bucket or Google GCS bucket. Note that the storage integration should be in the same cloud provider and region as the external volume you wish to create.

Create an external volume​

  • Via SDKs

    # Initiate a volume manager
    from pymilvus.bulk_writer.volume_manager import VolumeManager

    volume_manager = VolumeManager(
        cloud_endpoint="https://api.cloud.zilliz.com",
        api_key="YOUR_API_KEY"
    )

    # Create a volume
    volume_manager.create_volume(
        project_id="proj-xxxxxxxxxxxxxxxxxxxxxxx", 
        region_id="aws-us-west-2", 
        volume_name="external_volume",
        volume_type="EXTERNAL",
        storage_integration_id="integ-xxxx",
        path="data/",
    )

    print(f"\nVolume external_volume created")

    # Volume external_volume created

    The following table describes the parameters.

    Parameter

    Description

    projectId

    The ID of the project in which you want to create the volume.

    regionId

    The region of the volume to create must match the cloud provider and region of the target cluster you plan to import or migrate data into.

    volumeName

    The name of the volume to create must be unique across the organization, no longer than 64 characters, start with a letter or underscore, and contain only letters, digits, hyphens, and underscores.

    type

    Set the parameter to EXTERNAL to create an external volume. Defaults to MANAGED.

    storageIntegrationId

    The ID of the storage integration to reference. Required when type=EXTERNAL. The storage integration you select must belong to the same org and region as the external volume you want to create.

    path

    The storage path. Required when type=EXTERNAL.

  • Via web console

    1

    In the left navigation, click on Volumes.

    2

    On the volumes page, click on + Volume.

    3

    Set the volume configurations.

    The following table describes each parameter used when creating an external volume.

    Parameter

    Description

    Name

    The volume name must be unique across the organization, no longer than 64 characters, start with a letter or underscore, and contain only letters, digits, hyphens, and underscores.

    Description

    This parameter is optional.

    Volume Type

    Select "External" as the volume type.

    Cloud Provider & Region

    The volume cloud provider and region must match the cloud provider and region of the target cluster you plan to import or migrate data into.

    Storage Integration & Path

    Storage integration (AWS S3 bucket or Google GCS bucket) is the credential object that encapsulates the access configuration for your cloud storage.

    Path is a pointer to where your data is placed. (Eg. folder/)

    4

    Click on Create.

List external volumes​

You can view all existing volumes in a project.

  • Via SDKs

    # Initiate a volume manager
    from pymilvus.bulk_writer.volume_manager import VolumeManager

    volume_manager = VolumeManager(
        cloud_endpoint="https://api.cloud.zilliz.com",
        api_key="YOUR_API_KEY"
    )

    # View volumes
    volume_list = volume_manager.list_volumes(
        project_id="proj-xxxxxxxxxxxxxxxxxxxxxxx",
        current_page=1, 
        page_size=10
    )

    print(f"\nlistVolumes results: \n", volume_list.json()['data'])

    # listVolumes results: 
    # 
    # {
    #     "count": 1,
    #     "currentPage": 1,
    #     "pageSize": 10,
    #     "volumes": [
    #         {
    #             "volumeName": "external_volume"
    #             "type":"EXTERNAL"
    #         }        
    #     ]
    # }

  • Via web console

    PeL0wrKF1hTHvwbNAZBctTQonZf

Describe external volume​

You can check the details of a specific volume.

  • Via SDKs

    # Initiate a volume manager
    from pymilvus.bulk_writer.volume_manager import VolumeManager

    volume_manager = VolumeManager(
        cloud_endpoint="https://api.cloud.zilliz.com",
        api_key="YOUR_API_KEY"
    )

    # View volumes
    volume_list = volume_manager.describe_volume(
            volume_name="external_volume"
    )

    print(f"\ndescVolume results: \n", volume_list.json()['data'])

    # descVolume results: 
    # 
    # {
    #    "volumeName": "external_volume",
    #    "type": "EXTERNAL",
    #    "regionId": "aws-us-west-2",
    #    "storageIntegrationId": "integ-xxxx",
    #    "path": "data/",
    #    "status": "RUNNING",
    #    "createTime": "2024-04-15T12:00:00Z",
    # }

  • Via web console

    NrgXwPhxGhq78NbBfDYcWc6Ened

Delete an external volume​

You can delete an external volume at any time if it is no longer needed.

Deleting an external volume removes only the volume metadata from Zilliz Cloud; your data remains intact in your cloud object storage.

  • Via SDKs

    # Initiate a volume manager
    from pymilvus.bulk_writer.volume_manager import VolumeManager

    volume_manager = VolumeManager(
        cloud_endpoint="https://api.cloud.zilliz.com",
        api_key="YOUR_API_KEY"
    )

    # Delete a volume
    volume_manager.delete_volume(
        volume_name="external_volume"
    )

    print(f"\nVolume external_volume deleted")

    # Volume external_volume deleted

  • Via web console

    1

    In the left navigation, click on Volumes.

    2

    Click on ... in the Actions column, and then select Delete.

    3

    Enter the volume name and click on Delete.

Billing​

Creating and using an external volume incurs no Zilliz Cloud charges. No payment method is required.

However, your cloud provider may charge data request fees when Zilliz Cloud reads from your bucket during import or migration. For details, see Amazon S3 Pricing or Google Cloud Storage Pricing.

FAQs​

What happens to my volumes if my organization is frozen due to overdue invoices?

If an organization is frozen, all managed Volumes — both free trial and pay-as-you-go — and all files stored in them are deleted and cannot be restored. External volumes are also frozen and cannot be used for new operations, but your data in your own bucket is not affected.

To continue using volumes, first settle all outstanding invoices.

What is the difference between an external volume and importing directly from external storage?

Both allow you to import data from your own S3 or GCS bucket. The key differences are:

  • External volume requires you to integrate an AWS S3 bucket, a Google Cloud Storage bucket, or a Microsoft Azure blob storage container with Zilliz Cloud for credential management. Credentials are set up once and reused across multiple volumes and operations. Data engineers do not need direct access to cloud storage keys.

  • Direct external storage import requires you to provide credentials (access key, secret key) inline with each import request. This is simpler for one-time imports but does not offer credential separation or reusability.

Can I modify the storage integration or path of an external volume after creation?

No. The storage integration and path cannot be changed after an external volume is created. To use a different storage integration or path, create a new external volume.

Can I delete an external volume that is referenced by an active job or external collection?

No. Deletion is blocked if downstream external collections or active jobs reference the volume.

Will I be charged for data transfer fees when I use an external volume?

No. External volumes must be in the same cloud provider and region as your cluster. Since all data access occurs within the same region, no cross-region data transfer fees are incurred on Zilliz Cloud.

What do the volume statuses mean?

The following table lists the possible volume statuses.

Status

Description

Available

The volume is active and usable.

Frozen

The organization is frozen due to overdue invoices. The volume cannot be used for new operations. Please pay your bill to continue using volumes.

Error

The storage integration validation failed. Check the configuration and retry.