Integrate with AWS S3

Private Preview

Zilliz Cloud allows you to integrate with Amazon Simple Storage Service (Amazon S3) to export backup files or audit logs to designated S3 buckets.

📘Notes

This feature is available only for clusters on the Dedicated-Enterprise plan. To upgrade your plan tier, refer to Manage Cluster.

Before you start

• To integrate Zilliz Cloud with AWS S3, you must have Organization Owner or Project Admin access to the project. If you do not have necessary permissions, contact your Zilliz Cloud Organization Owner.

• You have administrative access to the AWS Management Console.

Step 1: Start integration in Zilliz Cloud console

1. Log in to the Zilliz Cloud console.

2. On your project page, navigate to Integrations from the left-side navigation pane.

3. Under the Amazon S3 section, click + Integration.

4. In the dialog box that appears, configure Basic Settings:

   • Integration Name: A unique name for this integration (e.g., integration_0819).

   • Integration Description (optional): A description for this integration (e.g., for export backupfile).

5. Click Next. You'll be redirected to the Create Amazon S3 Bucket step:

   1. In the Zilliz Cloud Cluster Region field, select the cloud region where your Zilliz Cloud cluster resides. The bucket you create later must be in the same region as your Zilliz Cloud cluster.

   2. Open the S3 console and proceed to step 2.

Step 2: Create S3 bucket in AWS console

1. In the upper-right corner of the Amazon S3 console, choose the AWS region that matches your Zilliz Cloud cluster’s region.

   📘Notes

   • The AWS region to create a bucket should be consistent with the region where your Zilliz Cloud cluster resides. For Zilliz Cloud-supported regions, refer to Cloud Providers & Regions.

   • For clusters running in different regions, create separate integrations for each region to ensure backup files or audit logs can be exported properly.

2. In the left navigation pane, choose General purpose buckets, and then click Create bucket.

3. Configure bucket settings:

   1. Under Bucket type, choose General purpose.

   2. For Bucket name, enter a name for your bucket (e.g., zilliz-bucket-for-integration-0819). Remember this bucket name, as you will need it for future steps.

   3. Keep other settings as default and click Create bucket.

   For more information, refer to Creating a bucket.

Once the bucket is created, go back to the Zilliz Cloud console, and do the following:

1. In the Bucket Name field, enter the name of the bucket you just created (in this example, zilliz-bucket-for-integration-0819). Then, click Next.

2. In the Create IAM Policy step, copy the JSON policy. It will be required in step 3.

3. Once completed, open the IAM console and proceed to step 3.

Step 3: Create IAM policy in AWS console

To give Zilliz Cloud access to AWS S3, create an IAM policy. This policy should include specific actions and resources to facilitate the transfer of backup files between Zilliz Cloud and your S3 bucket.

For simplicity, create a policy using the JSON editor.

1. In the IAM console, choose Policies > Create policy.

2. In the Policy editor section, choose the JSON option.

3. Copy and paste the JSON policy document provided by Zilliz Cloud to the policy editor. Then, click Next.

   The following is a sample JSON policy document. For the exact policy tailored to your integration, refer to the Create IAM Policy step on the Zilliz Cloud console.

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "Statement1",
               "Effect": "Allow",
               "Action": [
                   "s3:GetObject",
                   "s3:PutObject",
                   "s3:ListBucket",
                   "s3:GetBucketLocation"
               ],
               "Resource": [
                   "arn:aws:s3:::<bucket>",
                   "arn:aws:s3:::<bucket>/*"
               ]
           }
       ]
   }

   However, when you have enabled server-side encryption for the bucket using AWS KMS, you need to add another IAM policy to allow the kms:GenerateDataKey action. In this case, use the following JSON policy.

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "Statement1",
               "Effect": "Allow",
               "Action": [
                   "s3:GetObject",
                   "s3:PutObject",
                   "s3:ListBucket",
                   "s3:GetBucketLocation"
               ],
               "Resource": [
                   "arn:aws:s3:::<bucket>",
                   "arn:aws:s3:::<bucket>/*"
               ]
           },
           {
               "Sid": "AllowKMSGenerateDataKey",
               "Effect": "Allow",
               "Action": [
                   "kms:GenerateDataKey"
               ],
               "Resource": "arn:aws:kms:<region>:<account_id>:key/<key_id>"
           }
       ]
   }

   📘Notes

   • <bucket> should be replaced with the actual name of your S3 bucket.

   • <region>, <account_id>, and <key_id> should be replaced with their actual values. For details, refer to Key identifiers in AWS docs.

4. On the Review and create page, enter a Policy Name (e.g. zilliz-policy-for-integration-0819) and a Description (optional) for the policy that you are creating, and review Permissions defined in this policy. Remember the policy name, as you will need it for future steps.

5. Choose Create policy to save your new policy. Once completed, proceed to step 4.

Step 4: Create IAM role

Before creating an IAM role in AWS console, do the following in the Zilliz Cloud console:

1. In the Zilliz Cloud console, click Next to proceed to the Create IAM Role step.

2. In Select trusted entity, copy the JSON content, then go to the IAM console.

Once that's done, do the following to create an IAM role:

1. In the IAM console, choose Roles > Create role.

2. Choose the Custom trust policy role type.

3. In the Custom trust policy section, copy and paste the custom trust policy for the role. Then, click Next.

   The following is a sample JSON trust policy. For the exact trust policy tailored to your integration, refer to the Create IAM Role step on the Zilliz Cloud console.

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Action": "sts:AssumeRole",
               "Principal": {
                   "AWS": "965570967084"
               },
               "Condition": {
                   "StringEquals": {
                       "sts:ExternalId": "my-external-id"
                   }
               }
           }
       ]
   }

   📘Notes

   965570967084 and my-external-id should be replaced with the actual AWS account ID and external ID shown in the Create IAM Role step on the Zilliz Cloud console.

4. Under Permissions policies of the Add permissions step, search for and select the policy you created in step 3 to add permissions. Then, click Next.

5. In the Name, review, and create step, enter a role name (e.g. zilliz-integration-role-0819) and review the settings. Then, click Create role.

6. Go to the details page of the created role, copy the ARN corresponding to the role. This will be required in Zilliz Cloud console in step 5.

Step 5: Validate and add integration

1. In the Create IAM Role step of the Zilliz Cloud console, paste the ARN you copied from the IAM console in the previous step.

2. Then, click Validate Integration to confirm the S3 bucket and IAM role settings.

3. Once the status changes to Successful, the integration works. Then, click Add.

You can now use this integration to export backup files or forward audit logs to your Amazon S3 bucket. For more information, refer to Export Backup Files or Audit Logging.

Manage integrations

Once the integration is added, you can view its details or remove the integration as needed.

Obtain the integration ID

If you need to use the RESTful API to export backup files to one of your AWS S3 buckets integrated with Zilliz Cloud, click View Details to display the details of an integration and copy its integration ID.

Troubleshooting

If you encounter issues during the integration process, here are some common error messages and their solutions.

Bucket region mismatch

Description: The following example error occurs when the region of the S3 bucket does not match the region of your Zilliz Cloud cluster.

"bucket region not match, want[us-west-1] got[us-west-2]"

Solution:

• Ensure that the AWS region where your S3 bucket is located matches the region of your Zilliz Cloud cluster.

• If needed, create a new bucket in the correct region or adjust your cluster's region to match the bucket’s region.

Bucket not found

Description: This error occurs when the specified S3 bucket does not exist or the bucket name is incorrect.

check bucket failed: get bucket location: operation error S3: GetBucketLocation, https response error StatusCode: 404, RequestID: ..., HostID: ..., api error NoSuchBucket: The specified bucket does not exis

Solution:

• Double-check the bucket name in both the Zilliz Cloud console and the AWS S3 console.

• Confirm that the bucket exists and that the name is correctly entered in your Zilliz Cloud configuration.

Access denied for bucket location

Description: This error occurs when the IAM role does not have the required permissions to access the S3 bucket's location.

check bucket failed: get bucket location: operation error S3: GetBucketLocation, https response error StatusCode: 403 ...

Solution:

• Review the IAM policy attached to the role being used by Zilliz Cloud.

• Ensure the policy includes the s3:GetBucketLocation permission along with other necessary permissions, such as s3:GetObject, s3:PutObject, and s3:ListBucket.

Role assumption failure

Description: This error occurs when there is an issue with assuming the IAM role due to incorrect role ARN, external ID, or trust policy.

try assume role from[zilliz-role] to [arn:aws:iam::041623484421:role/testoss121703] with externalId[zilliz-external-1umVCIK7q96kzDE] failed

Solution:

• Verify that the role ARN and external ID on the Zilliz Cloud console match the corresponding values in the IAM trust policy.

• Ensure that the trust policy in the IAM role allows Zilliz Cloud to assume the role.