Migrate from Milvus to Zilliz Cloud Via Backup Files

Zilliz Cloud offers Milvus as a fully managed, cloud-hosted solution for users who want to use the Milvus vector database without the need to manage the infrastructure themselves. To enable smooth migration, you can migrate data from Milvus to Zilliz Cloud in these ways - connecting to source Milvus via database endpoint or uploading backup files directly.

This topic describes how to migrate from Milvus by uploading backup files directly. For information on how to migrate data via database endpoint, refer to Via Endpoint.

Before you start

Make sure the following prerequisites are met:

• You have made necessary preparations for migration based on the migration method:

  • From Local File: Prepare local backup files in advance. For information on how to prepare backup files, refer to Prepare backup files for migration.

  • From Object Storage: The public URL and access credentials for the Milvus object storage. You can choose long-term or temporary credentials. For detailed examples of an object storage URL, see FAQ.

• You have been granted the Organization Owner or Project Admin role. If you do not have the necessary permissions, contact your Zilliz Cloud Organization Owner.

• Make sure the CU size of the target cluster can accommodate your source data. To estimate the required CU size, use the calculator.

Prepare backup files for migration

To prepare migration data for Milvus 2.x,

1. Download milvus-backup. Always use the latest release.

   Currently, you can migrate data from Milvus 2.2 and later versions to Zilliz Cloud clusters. For details on compatible source and target Milvus versions, refer to Milvus Backup Overview.

2. Create a configs folder side by side with the downloaded binary, and download backup.yaml into the configs folder.

   Once the step is done, the structure of your workspace folder should look like this:

   workspace
   ├── milvus-backup
   └── configs
        └── backup.yaml

3. Customize backup.yaml.

   In normal cases, you do not need to customize this file. But before going on, check whether the following configuration items are correct:

   • milvus.address

   • mivlus.port

   • minio.address

   • minio.port

   • minio.bucketName

   • minio.backupBucketName

   • rootPath

   📘Notes

   • For a Milvus instance installed using Docker Compose, minio.bucketName defaults to a-bucket and rootPath defaults to files.

   • For a Milvus instance installed on Kubernetes, minio.bucketName defaults to milvus-bucket and rootPath defaults to file.

4. Create a backup of your Milvus installation.

   ./milvus-backup --config backup.yaml create -n my_backup

5. Get the backup file.

   ./milvus-backup --config backup.yaml get -n my_backup

6. Check the backup files.

   • If you set minio.address and minio.port to an S3 bucket, your backup file are already in the S3 bucket.

   • If you set minio.address and minio.port to a Minio bucket, you can download them using Minio Console or the mc client.

     • To download from Minio Console, log into Minio Console, locate the bucket specified in minio.address, select the files in the bucket, and click Download to download them.

     • If you prefer the mc client, do as follows:

       # configure a Minio host
       mc alias set my_minio https://<minio_endpoint> <accessKey> <secretKey>
       
       # List the available buckets
       mc ls my_minio
       
       # Download a file from the bucket
       mc cp --recursive my_minio/<your-bucket-path> <local_dir_path>

7. Decompress the downloaded archive and upload only the content of the backup folder to Zilliz Cloud.

Migrate data to Zilliz Cloud

With backup files ready, you can migrate the data from local files directly or from object storage.

📘Notes

During migration, the selected target database must contain data and cannot be empty.

Monitor the migration process

Once you click Migrate, a migration job will be generated. You can check the migration progress on the Jobs page. When the job status switches from In Progress to Successful, the migration is complete.

📘Notes

After migration, verify that the number of collections and entities in the target cluster matches the data source. If discrepancies are found, delete the collections with missing entities and re-migrate them.

Post-migration

After the migration job is completed, note the following:

• Index Creation: The migration process automatically creates AUTOINDEX for the migrated collections.

• Manual Loading Required: Despite automatic indexing, the migrated collections are not immediately available for search or query operations. You must manually load the collections in Zilliz Cloud to enable search and query functionalities. For details, refer to Load & Release.

Cancel migration job

If the migration process encounters any issues, you can take the following steps to troubleshoot and resume the migration:

1. On the Jobs page, identify the failed migration job and cancel it.

2. Click View Details in the Actions column to access the error log.

FAQ

1. What format of URL should I follow when migrating from a backup file stored in an object storage bucket?

   The following table provides an example of the URLs of different object storage services. Note that when migrating from a backup file, you can only choose a backup folder.

   Cloud Object Storage		URL Format
   Amazon S3	AWS Object URL, virtual-hosted–style	https://<bucket_name>.s3.<region-code>.amazonaws.com/<folder_name>/
   	AWS Object URL, path-style	https://s3.<region-code>.amazonaws.com/<bucket_name>/<folder_name>/
   	Amazon S3 URI	s3://<bucket_name>/<folder_name>/
   Google Cloud Storage	GSC public URL	https://storage.cloud.google.com/<bucket_name>/<folder_name>/
   	GSC gsutil URI	gs://<bucket_name>/<folder_name>/
   Azure Blob Storage		https://<storage_account>.blob.core.windows.net/<container>/<folder>/