Migrate from Milvus to Zilliz Cloud Via Stage

Private Preview

Zilliz Cloud provides an internal stage feature that serves as a storage spot for backup data during data migration from Milvus. This feature allows users to perform data migration more easily and efficiently, without needing to handle complex details manually, thus greatly enhancing usability and success rates.

This feature eliminates the operational complexity across various migration scenarios, such as:

• File size limitations when migrating with local backup files.

• Understanding cloud storage bucket configurations from different cloud providers when using bucket-based migration.

• Ensuring network accessibility of the Milvus instance endpoint when performing endpoint-based migrations.

📘Notes

Migration via stage is in Private Preview. If you are interested, encounter any issues, or would like to learn about associated costs, contact Zilliz Cloud support.

Before you start

• You have been granted the Organization Owner or Project Admin role. If you do not have the necessary permissions, contact your Zilliz Cloud administrator.

• Make sure the CU size of the target cluster can accommodate your source data. To estimate the required CU size, use the calculator.

Procedure

In this procedure, you will use Milvus Backup to prepare the backup files, upload them to the Zilliz Cloud internal stage, and migrate them to the specified target Zilliz Cloud cluster.

1. Download milvus-backup. Always use the latest release.

   Currently, you can migrate data from Milvus 2.2 and later versions to Zilliz Cloud clusters. For details on compatible source and target Milvus versions, refer to Milvus Backup Overview.

2. Create a configs folder side by side with the downloaded binary, and download backup.yaml into the configs folder.

   Once the step is done, the structure of your workspace folder should look like this:

   workspace
   ├── milvus-backup
   └── configs
        └── backup.yaml

3. Customize backup.yaml.

   1. Set the following configuration items:

      • cloud.address

        The Zilliz Cloud Control Plane endpoint, which is https://api.cloud.zilliz.com.

      • cloud.apikey

        A valid Zilliz Cloud API key with sufficient permissions to operate the target cluster of the migration. For details, refer to API Keys.

   2. Check whether the following configuration items are correct:

      • milvus.address

      • mivlus.port

      • minio.address

      • minio.port

      • minio.bucketName

      • minio.backupBucketName

      • rootPath

   📘Notes

   • For a Milvus instance installed using Docker Compose, minio.bucketName defaults to a-bucket and rootPath defaults to files.

   • For a Milvus instance installed on Kubernetes, minio.bucketName defaults to milvus-bucket and rootPath defaults to file.

4. Create a backup of your Milvus installation.

   ./milvus-backup --config backup.yaml create -n my_backup
   
   # my_backup is the name of the backup file 
   # and will be used in the migrate command

5. Create the target Zilliz Cloud cluster, note the cluster ID, and run the following command to initiate the migration.

   ./milvus-backup migrate --cluster_id inxx-xxxxxxxxxxxxxxx -n my_backup
   
   # cluster_id is the ID of the target Zilliz Cloud cluster
   # my_backup is the name of the backup file created in the above command
   
   # The command output is similar to the following:
   # Successfully triggered migration with backup name: my_backup target cluster: inxx-xxxxxxxxxxxxxxx migration job id: job-xxxxxxxxxxxxxxxxxxx.
   # You can check the progress of the migration job in Zilliz Cloud console.

   Upon executing this command, Milvus Backup uploads the prepared backup files to the Zilliz Cloud internal stage, creates a migration job, and returns the job ID as the command output.

   📘Notes

   The backup files uploaded to the internal stage will stay in the stage for 3 days after the upload and will then be deleted.

Monitor the migration process

Once you click Migrate, a migration job will be generated. You can check the migration progress on the Jobs page. When the job status switches from In Progress to Successful, the migration is complete.

📘Notes

After migration, verify that the number of collections and entities in the target cluster matches the data source. If discrepancies are found, delete the collections with missing entities and re-migrate them.

Post-migration

After the migration job is completed, note the following:

• Index Creation: The migration process automatically creates AUTOINDEX for migrated collections.

• Manual Loading Required: Despite automatic indexing, the migrated collections are not immediately available for search or query operations. You must manually load the collections in Zilliz Cloud to enable search and query functionalities. For details, refer to Load & Release.

Cancel migration job

If the migration process encounters any issues, you can take the following steps to troubleshoot and resume the migration:

1. On the Jobs page, identify the failed migration job and cancel it.

2. Click View Details in the Actions column to access the error log.