VolumeBulkWriter

VolumeBulkWriter handles writing local bulk files to a remote volume.

Request Syntax

python

VolumeBulkWriter(
    schema: CollectionSchema,
    remote_path: str,
    cloud_endpoint: str,
    api_key: str,
    volume_name: str,
    chunk_size: int = 1024 * MB,
    file_type: BulkFileType = BulkFileType.PARQUET,
    config: Optional[dict] = None,
    connect_type: ConnectType = ConnectType.AUTO,
    **kwargs,
)

PARAMETERS:

• schema (CollectionSchema) -
  [REQUIRED]
  The collection schema used to validate rows and generate bulk files.

• remote_path (str) -
  [REQUIRED]
  The directory in the target volume where committed files are uploaded.

• cloud_endpoint (str) -
  [REQUIRED]
  The Zilliz Cloud API server endpoint, which is https://api.cloud.zilliz.com.

• api_key (str) -
  [REQUIRED]
  The API key used to authenticate with Zilliz Cloud.

• volume_name (str) -
  [REQUIRED]
  The name of the target Zilliz Cloud volume.

• chunk_size (int) -
  Default: 1024 * MB
  The maximum local chunk size, in bytes, before the writer starts a new file.

• file_type (BulkFileType) -
  Default: BulkFileType.PARQUET
  The bulk-file format generated by the writer.

• config (Optional[dict]) -
  Default: None
  The optional writer configuration.

• connect_type (ConnectType) -
  Default: ConnectType.AUTO
  The connection mode used for volume operations.

• kwargs (Any) -
  The additional options forwarded to LocalBulkWriter.

RETURN TYPE:

VolumeBulkWriter

RETURNS:

A writer that stages bulk files locally and uploads committed files to the configured Zilliz Cloud volume.

EXCEPTIONS:

• MilvusException
  Raised when the server rejects the request or the RPC fails. Inspect the server error message for exact failure details.

Examples

The example demonstrates VolumeBulkWriter usage.

python

from pymilvus.bulk_writer import VolumeFileManager, VolumeManager

manager = VolumeManager(cloud_endpoint="https://api.cloud.zilliz.com", api_key="YOUR_API_KEY")
manager.create_volume(project_id="proj-xxxx", region_id="aws-us-west-2", volume_name="book-volume", volume_type="EXTERNAL")
manager.describe_volume("book-volume")
manager.list_volumes(project_id="proj-xxxx", volume_type="EXTERNAL")

file_manager = VolumeFileManager(cloud_endpoint="https://api.cloud.zilliz.com", api_key="YOUR_API_KEY", volume_name="book-volume")
file_manager.upload_file_to_volume(source_file_path="./data/books.parquet", target_volume_path="datasets/books/books.parquet", upload_concurrency=4)