Migrate from OpenSearch to Zilliz Cloud

OpenSearch is a distributed search and analytics engine that supports various use cases, from implementing a search box on a website to analyzing security data for threat detection. Zilliz Cloud enables seamless migration from OpenSearch, allowing you to leverage advanced analytics and AI-driven insights. This guide outlines how to transfer your OpenSearch data to Zilliz Cloud.

The migration process is structured into these steps:

1. Connect to data source: Enter your OpenSearch cluster endpoint to establish a connection.

2. Select source and target:

   • Choose one or more OpenSearch tables for migration.

   • Select an existing Zilliz Cloud cluster as the target.

3. Configure schema: Verify that field types are correctly mapped between OpenSearch and Zilliz Cloud. For detailed mapping rules, refer to Mapping rules.

Mapping rules

The following table summarizes how field types in OpenSearch are mapped to Zilliz Cloud field types, along with details on any customization options.

OpenSearch Field Type	Zilliz Cloud Field Type	Description
Primary key	Primary key	OpenSearch's primary key (_id) is automatically mapped as the primary key in Zilliz Cloud. When migrating data, you can enable Auto ID. However, if you do so, the original primary key values from your source table will be discarded.
k-NN vector	FLOAT_VECTOR	The float vector type from OpenSearch is mapped to FLOAT_VECTOR on Zilliz Cloud. Byte/Binary vectors from OpenSearch are not supported for migration. Vector dimensions remain unchanged.
Alias	Not supported	Alias fields are not supported.
Binary	VARCHAR	Binary data is stored as a string on Zilliz Cloud.
Numeric
byte	INT8	Directly mapped.
double	DOUBLE	Directly mapped.
float	FLOAT	Directly mapped.
half_float	FLOAT	Mapped to FLOAT.
integer	INT32	Directly mapped.
long	INT64	Directly mapped.
short	INT16	Directly mapped.
unsigned_long	Not supported	Not supported on Zilliz Cloud.
scaled_float	Not supported	Not supported on Zilliz Cloud.
Boolean	BOOL	Stores true or false.
Date	VARCHAR	Stored as a string. Ensure correct format conversion.
IP address	VARCHAR	Stored as a string.
Range	VARCHAR	Stored as a string.
Object	JSON	Serialized into JSON format.
String
keyword	VARCHAR	Stored as a string.
text	VARCHAR	Mapped to VARCHAR .
match_only_text	VARCHAR	Stored as a string.
token_count	VARCHAR	Stored as a string.
wildcard	Not supported	Not supported on Zilliz Cloud.
Autocomplete	VARCHAR	Stored as a string.
Geographic	VARCHAR	Stored as a string.
Rank	VARCHAR	Stored as a string.
Percolator	VARCHAR	Stored as a string.
Derived	Not supported	Derived fields are not supported on Zilliz Cloud.
Star-tree	Not supported	Star-tree fields are not supported on Zilliz Cloud.

Before you start

• The source OpenSearch cluster is accessible from the internet.

• If you have an allowlist configured in your network environment, ensure that Zilliz Cloud IP addresses are added to it. For more information, refer to Zilliz Cloud IPs.

• You have been granted the Organization Owner or Project Admin role. If you do not have the necessary permissions, contact your Zilliz Cloud administrator.

Migrate from OpenSearch to Zilliz Cloud

1. Log in to the Zilliz Cloud console.

2. Go to the target project page and select Migrations > OpenSearch.

   

3. In the Connect to Data Source step, enter Cluster Endpoint (e.g. https://<ID>.<region>.es.amazonaws.com for AWS OpenSearch, https://<ip>:<port> for OpenSearch Community Edition), Username, and Password of the source OpenSearch cluster to establish connections. Then, click Next.

   📘Notes

   Need help finding your OpenSearch credentials? Check Communicate with OpenSearch.

   

4. In the Select Source and Target step, configure settings for the source OpenSearch cluster and target Zilliz Cloud cluster. Then, click Next.

   📘Notes

   Each source index you choose to migrate from OpenSearch must include a vector field.

   

5. In the Configure Schema step, set up field mappings between Zilliz Cloud and OpenSearch:

   

   1. Confirm field mappings:

      • Zilliz Cloud automatically detects and displays your OpenSearch fields alongside their corresponding target fields. For details on how these fields are mapped, refer to Mapping rules.

      • Verify that each OpenSearch field is correctly paired with its corresponding target field. You can rename fields as needed, but note that the data type cannot be changed.

   2. Handle scalar fields:

      For scalar fields, configure the following attributes:

      • Nullable: Decide whether a field can accept null values. This feature is enabled by default. For details, refer to Nullable & Default.

      • Default Value: Specify a default value for a field. For details, refer to Nullable & Default.

      • Partition Key: Optionally designate an INT64 or VARCHAR field as the partition key. Note: Each collection supports only one partition key, and the selected field cannot be nullable. For details, refer to Use Partition Key.

   3. Enable dynamic field:

      • Dynamic fields are enabled by default. This allows you to include any scalar fields that are not defined in the collection schema.

      • If you disable it, you need to explicitly define each field in your entity before inserting data. For more information, refer to Dynamic Field.

   4. (Optional) Adjust shards:

      • Click Advanced Settings to configure the number of shards for your target collection.

      • For datasets of around 100 million rows, a single shard is typically sufficient.

      • If your dataset exceeds 1 billion rows, contact us to discuss optimal shard configuration for your use case.

6. Click Migrate.

Monitor the migration process

Once you click Migrate, a migration job will be generated. You can check the migration progress on the Jobs page. When the job status switches from In Progress to Successful, the migration is complete.

Post-migration

After the migration job is completed, note the following:

• Index Creation: The migration process does not automatically create indexes for vector fields when migrating from external data sources. You must manually create the index for each vector field. For details, refer to Index Vector Fields.

• Manual Loading Required: After creating the necessary indexes, manually load the collections to make them available for search and query operations. For details, refer to Load & Release.

📘Notes

Once you have completed indexing and loading, verify that the number of collections and entities in the target cluster matches the data source. If discrepancies are found, delete the collections with missing entities and re-migrate them.

Cancel migration job

If the migration process encounters any issues, you can take the following steps to troubleshoot and resume the migration:

1. On the Jobs page, identify the failed migration job and cancel it.

2. Click View Details in the Actions column to access the error log.