Migrate from OpenSearch to Zilliz Cloud

OpenSearch is a distributed search and analytics engine that supports various use cases, from implementing a search box on a website to analyzing security data for threat detection. Zilliz Cloud enables seamless migration from OpenSearch, allowing you to leverage advanced analytics and AI-driven insights. This guide outlines how to transfer your OpenSearch data to Zilliz Cloud.

Considerations

• Zilliz Cloud supports migrating most OpenSearch data types. Before starting, verify that your data types are compatible. If your table has fields with unsupported data types, you can choose not to migrate those fields or submit a support ticket. For a full list of supported data types, Field mapping reference.

• Each migration task is limited to a single source OpenSearch cluster. If you have data in multiple source clusters, you can set up separate migration jobs for each one.

Before you start

• The source OpenSearch cluster is accessible from the internet.

• If you have an allowlist configured in your network environment, ensure that Zilliz Cloud IP addresses are added to it. For more information, refer to Zilliz Cloud IPs.

• You have been granted the Organization Owner or Project Admin role. If you do not have the necessary permissions, contact your Zilliz Cloud administrator.

Migrate from OpenSearch to Zilliz Cloud

You can migrate source data to a Zilliz Cloud cluster of any plan tier, provided its CU size can accommodate the source data.

1. Log in to the Zilliz Cloud console.

2. Go to the target project page and select Migrations > OpenSearch.

3. In the Connect to Data Source step, enter Cluster Endpoint (e.g. https://<ID>.<region>.es.amazonaws.com for AWS OpenSearch, https://<ip>:<port> for OpenSearch Community Edition), Username, and Password of the source OpenSearch cluster to establish connections. Then, click Next.

   📘Notes

   Need help finding your OpenSearch credentials? Check Communicate with OpenSearch.

4. In the Select Source and Target step, configure settings for the source OpenSearch cluster and target Zilliz Cloud cluster. Then, click Next.

   📘Notes

   Each source index you choose to migrate from OpenSearch must include a vector field.

5. In the Configure Schema step,

   1. Verify the data mapping between your OpenSearch data and the corresponding Zilliz Cloud data types. Zilliz Cloud has a default mechanism for mapping OpenSearch data types to its own, but you can review and make necessary adjustments. Currently, you can rename fields, but cannot change the underlying data types.

   2. In Advanced Settings, configure Dynamic Field and Partition Key. For more information, refer to Dynamic Field and Use Partition Key.

   3. In Target Collection Name and Description, customize the target collection name and description. The collection name must be unique in each cluster. If the name duplicates an existing one, rename the collection.

6. Click Migrate.

Monitor the migration process

Once you click Migrate, a migration job will be generated. You can check the migration progress on the Jobs page. When the job status switches from IN PROGRESS to SUCCESSFUL, the migration is complete.

📘Notes

After migration, verify that the number of collections and entities in the target cluster matches the data source. If discrepancies are found, delete the collections with missing entities and re-migrate them.

Cancel migration job

If the migration process encounters any issues, you can take the following steps to troubleshoot and resume the migration:

1. On the Jobs page, identify the failed migration job and cancel it.

2. Click View Details in the Actions column to access the error log.

Field mapping reference

Review the table below to understand how OpenSearch data types map to Zilliz Cloud field types.

OpenSearch Field Type	Zilliz Cloud Field Type	Description
k-NN vector	FLOAT_VECTOR	The float vector type from OpenSearch is mapped to FLOAT_VECTOR on Zilliz Cloud. Byte/Binary vectors from OpenSearch are not supported for migration. Vector dimensions remain unchanged.
Alias	Not supported	Alias fields are not supported.
Binary	VARCHAR	Binary data is stored as a string on Zilliz Cloud.
Numeric
byte	INT8	Directly mapped.
double	DOUBLE	Directly mapped.
float	FLOAT	Directly mapped.
half_float	FLOAT	Mapped to FLOAT.
integer	INT32	Directly mapped.
long	INT64	Directly mapped.
short	INT16	Directly mapped.
unsigned_long	Not supported	Not supported on Zilliz Cloud.
scaled_float	Not supported	Not supported on Zilliz Cloud.
Boolean	BOOL	Stores true or false.
Date	VARCHAR	Stored as a string. Ensure correct format conversion.
IP address	VARCHAR	Stored as a string.
Range	VARCHAR	Stored as a string.
Object	JSON	Serialized into JSON format.
String
keyword	VARCHAR	Stored as a string.
text	VARCHAR	Mapped to VARCHAR .
match_only_text	VARCHAR	Stored as a string.
token_count	VARCHAR	Stored as a string.
wildcard	Not supported	Not supported on Zilliz Cloud.
Autocomplete	VARCHAR	Stored as a string.
Geographic	VARCHAR	Stored as a string.
Rank	VARCHAR	Stored as a string.
Percolator	VARCHAR	Stored as a string.
Derived	Not supported	Derived fields are not supported on Zilliz Cloud.
Star-tree	Not supported	Star-tree fields are not supported on Zilliz Cloud.

• Potential data loss or truncation: When storing fields like Date, Range, IP address, or large text content in a VARCHAR column, consider any length limitations or indexing requirements on Zilliz Cloud.

• Unsupported field types: For OpenSearch types that are not supported, transform or exclude them before migrating to Zilliz Cloud.