Migrate from Elasticsearch to Zilliz Cloud

Elasticsearch is a highly scalable search and analytics engine known for its speed and flexibility in handling large volumes of data. By leveraging Zilliz Cloud's migration capabilities, you can seamlessly transfer data from your Elasticsearch instances to your Zilliz Cloud cluster.

This migration process involves establishing a connection with your existing Elasticsearch source and replicating its data indices to the corresponding target collections in Zilliz Cloud, preserving both the structure and performance of your original data while enabling advanced vector search functionalities.

Considerations

• Currently, you can migrate the following Elasticsearch data types: dense_vector, text, string, keyword, ip, date, timestamp, long, integer, short, byte, double, float, boolean, object, arrays. If your table has fields with unsupported data types, you can choose not to migrate those fields or submit a support ticket. For information on how Elasticsearch data types are mapped to Zilliz Cloud, refer to Field mapping reference.

• To ensure compatibility, Auto ID will be disabled and cannot be modified for each target collection on Zilliz Cloud.

• For each migration task, you can select only one vector field from each source index.

• Each migration task is limited to a single source Elasticsearch cluster. If you have data in multiple source clusters, you can set up separate migration jobs for each one.

Before you start

Make sure the following prerequisites are met:

• The source Elasticsearch cluster is running version 7.x or later and is accessible from the public internet.

• You have the necessary connection credentials for the source cluster:

  • For an Elastic Cloud cluster, obtain the Cloud ID and API key.

  • For an on-premises cluster, obtain the cluster URL, along with a username and password.

• You have been granted the Organization Owner or Project Admin role. If you do not have the necessary permissions, contact your Zilliz Cloud administrator.

Migrate from Elasticsearch to Zilliz Cloud

You can migrate source data to a Zilliz Cloud cluster of any plan tier, provided its CU size can accommodate the source data.

1. Log in to the Zilliz Cloud console.

2. Go to the target project page and select Migrations > Elasticsearch.

3. In the Connect to Data Source step, select Via Endpoint or Via Cloud ID as the connection method to interact with the source Elasticsearch cluster. Then, click Next.

   📘Notes

   Connect to your cluster and Get API key information can guide you on obtaining the required connection information.

4. In the Select Source and Target step, configure settings for the source Elasticsearch cluster and target Zilliz Cloud cluster. Then, click Next.

   📘Notes

   Each source index you choose to migrate from Elasticsearch must include a vector field.

5. In the Configure Schema step,

   1. Verify the data mapping between your Elasticsearch data and the corresponding Zilliz Cloud data types. Zilliz Cloud has a default mechanism for mapping Elasticsearch data types to its own, but you can review and make necessary adjustments. Currently, you can rename fields, but cannot change the underlying data types.

   2. In Advanced Settings, configure Dynamic Field and Partition Key. For more information, refer to Dynamic data fields and Use Partition Key.

   3. In Target Collection Name and Description, customize the target collection name and description. The collection name must be unique in each cluster. If the name duplicates an existing one, rename the collection.

6. Click Migrate.

Monitor the migration process

Once you click Migrate, a migration job will be generated. You can check the migration progress on the Jobs page. When the job status switches from IN PROGRESS to SUCCESSFUL, the migration is complete.

📘Notes

After migration, verify that the number of collections and entities in the target cluster matches the data source. If discrepancies are found, delete the collections with missing entities and re-migrate them.

Cancel migration job

If the migration process encounters any issues, you can take the following steps to troubleshoot and resume the migration:

1. On the Jobs page, identify the failed migration job and cancel it.

2. Click View Details in the Actions column to access the error log.

Field mapping reference

Review the table below to understand how Elasticsearch data types map to Zilliz Cloud field types.

Elasticsearch Field Type	Zilliz Cloud Field Type	Description
dense_vector	FloatVector	Vector dimensions remain unchanged. Specify L2 or IP as the metric type.
text, string, keyword, ip, date, timestamp	VarChar	Set Max Length (1 to 65,535). Strings exceeding the limit can trigger migration errors.
long	Int64	-
integer	Int32	-
short	int16	-
byte	int8	-
double	Double	-
float	Float	-
boolean	Bool	-
object	JSON	-
arrays	Array	-