Skip to main content
Version: User Guides (BYOC)

Migrate from Pinecone to Zilliz Cloud

Pinecone is a vector database that allows for similarity searches. Migrating data from Pinecone to Zilliz Cloud can enhance capabilities for managing both dense and sparse vectors while taking advantage of Zilliz Cloud’s high-performance search and analytics.

This guide walks you through the process of migrating your data from Pinecone to Zilliz Cloud, including connecting to Pinecone, configuring data mappings, and troubleshooting potential issues.

Considerations​

  • When you migrate data from Pinecone to Zilliz Cloud, vector fields are transferred directly, while metadata fields from Pinecone are stored as JSON in a dynamic field on Zilliz Cloud. For details on the dynamic field feature, refer to Dynamic Field.

  • To ensure compatibility, Auto ID will be disabled and cannot be modified for each target collection on Zilliz Cloud.

  • This migration only supports Pinecone serverless indexes.

  • Each migration task is limited to a single source Pinecone index. If you have data in multiple source indexes, you can set up separate migration jobs for each one.

Before you start​

  • The source Pinecone index is accessible from the public internet.

  • If you have an allowlist configured in your network environment, ensure that Zilliz Cloud IP addresses are added to it. For more information, refer to Zilliz Cloud IPs.

  • You have obtained the API key to access the target Pinecone project.

  • You have been granted the Organization Owner or Project Admin role on Zilliz Cloud. If you do not have the necessary permissions, contact your Zilliz Cloud administrator.

Migrate from Pinecone to Zilliz Cloud​

migrate_from_pinecone

You can migrate source data to a Zilliz Cloud cluster of any plan tier, provided its CU size can accommodate the source data.

  1. Log in to the Zilliz Cloud console.

  2. Go to the target project page and select Migrations > Pinecone.

  3. In the Connect to Data Source step, enter the API key that can be used to access the target Pinecone project. Then, click Next.

    📘Notes

    Authentication can guide you in obtaining the required connection information.

  4. In the Select Source and Target step, configure settings for the source Pinecone index and target Zilliz Cloud cluster. Then, click Next.

    📘Notes

    Each source index you choose to migrate from Pinecone must include a vector field.

  5. In the Configure Schema step,

    1. In Schema Preview, verify the field mapping between your Pinecone index and the corresponding Zilliz Cloud collection.

      📘Notes

      • The Auto ID is disbaled and cannot be modified.

      • The record ID from Pinecone will be mapped to a VARCHAR field on Zilliz Cloud as the primary field, with a max_length range of 1 to 65,535 bytes. When inserting or upserting entities, ensure that VARCHAR field values stay within this limit.

      • You may rename fields, but the data types are fixed and cannot be changed.

    2. In Advanced Settings, verify the settings of Dynamic Field and Partition Key.

      1. Dynamic Field: Enabled by default and cannot be modified. It stores metadata from the source index, ensuring consistency and maintaining flexibility.

      2. Partition Key: Disabled by default and cannot be modified. This is because metadata from Pinecone is stored as JSON in a dynamic field, which cannot serve as a partition key. In Zilliz Cloud, only scalar fields that are explicitly defined in the schema can be used as partition keys.

    3. In Target Collection Name and Description, customize the target collection name and description. The collection name must be unique in each cluster. If the name duplicates an existing one, rename the collection.

  6. Click Migrate.

Monitor the migration process​

Once you click Migrate, a migration job will be generated. You can check the migration progress on the Jobs page. When the job status switches from IN PROGRESS to SUCCESSFUL, the migration is complete.

📘Notes

After migration, verify that the number of collections and entities in the target cluster matches the data source. If discrepancies are found, delete the collections with missing entities and re-migrate them.

verify_collection

Cancel migration job​

If the migration process encounters any issues, you can take the following steps to troubleshoot and resume the migration:

  1. On the Jobs page, identify the failed migration job and cancel it.

  2. Click View Details in the Actions column to access the error log.