Skip to main content
Version: User Guides (BYOC)

Quickstart

This guide explains how to set up your Zilliz Cloud cluster and perform CRUD operations in minutes.

Before you start​

Zilliz Cloud provides a Bring-Your-Own-Cloud (BYOC) solution, allowing organizations to host applications and data in their own cloud accounts instead of using Zilliz Cloud's infrastructure. For details about our BYOC solution, read BYOC Overview.

The following figure illustrates the procedures to start using our BYOC solution.

ChT3woJqYhkzj1bipPxcXNZrnbc

Before going through this quick start, ensure that:

  • You have registered an account with Zilliz Cloud.

    For instructions, refer to Register with Zilliz Cloud.

  • You have contacted Zilliz Cloud sales and provided your account to us.

    📘Notes

    Currently, Zilliz BYOC is in Private Preview. To participate in the trial, contact Zilliz Cloud support.

  • You have created a project in the BYOC organization and deployed the data plane infrastructure for the project.

    Zilliz BYOC operates within your Virtual Private Cloud (VPC), requiring you to initiate the deployment of data plane components. You can deploy the data plane in your VPCs hosted on the following cloud providers:

    If your cloud provider is not available above, contact Zilliz Cloud support.

Install an SDK​

Zilliz Cloud supports the Milvus SDKs and all RESTful API endpoints. You can use the RESTful API directly, or choose one of the following SDKs to start with:

Create a Cluster​

You can create a cluster with the subscription plan of your choice using either the RESTful API endpoints or on the Zilliz Cloud console.

The following demonstrates how to create a dedicated cluster using the RESTful API.

curl --request POST \
    --url "https://api.cloud.zilliz.com/v2/clusters/createDedicated" \
    --header "Authorization: Bearer ${API_KEY}" \
    --header "Accept: application/json" \
    --header "Content-Type: application/json" \
    --data-raw '{
        "clusterName": "Cluster-05",
        "projectId": "proj-xxxxxxxxxxxxxxxxxxxxxx",
        "regionId": "aws-us-west-2",
        "plan": "Standard",
        "cuType": "Performance-optimized",
        "cuSize": 1
    }'
    
# {
#     "code": 0,
#     "data": {
#         "clusterId": "inxx-xxxxxxxxxxxxxxx",
#         "username": "db_admin",
#         "password": "*************",
#         "prompt": "successfully submitted, cluster is being created. You can access data about the creation progress and status of your cluster by DescribeCluster API. Once the cluster status is RUNNING, you may access your vector database using the SDK with the admin account and the initial password you specified."
#     }
# }

You can find the cloud region and project ID on Zilliz Cloud console. If you prefer to create a free cluster on the Zilliz Cloud console, refer to Create Cluster.

Once your cluster is running, you will be prompted with the cluster credentials for once. Download and save it in a safe place. You will need it to connect to your cluster later.

Connect to Zilliz Cloud cluster​

Once you have obtained the cluster credentials, you can use it to connect to your cluster now.

from pymilvus import MilvusClient, DataType

CLUSTER_ENDPOINT = "YOUR_CLUSTER_ENDPOINT"
TOKEN = "YOUR_CLUSTER_TOKEN"

# 1. Set up a Milvus client
client = MilvusClient(
    uri=CLUSTER_ENDPOINT,
    token=TOKEN 
)
📘Notes

Due to language differences, you should include your code in the main function if you prefer to code in Java or Node.js.

Create a Collection​

On Zilliz Cloud, you need to store your vector embeddings in collections. All vector embeddings stored in a collection share the same dimensionality and distance metric for measuring similarity. You can create a collection in either of the following manners.

Quick setup​

To set up a collection in quick setup mode, you only need to set the collection name and the dimension of the vector field of the collection.

# 2. Create a collection in quick setup mode
client.create_collection(
    collection_name="quick_setup",
    dimension=5 # The dimensionality should be an integer greater than 1.
)

In the above setup,

  • The primary and vector fields use their default names (id and vector).

  • The metric type is also set to its default value (COSINE).

  • The primary field accepts integers and does not automatically increments.

  • A reserved JSON field named $meta is used to store non-schema-defined fields and their values.

📘Notes

Collections created using the RESTful API supports a minimum of 32-dimensional vector field.

Customized setup​

To define the collection schema by yourself, use the customized setup. In this manner, you can define the attributes of each field in the collection, including its name, data type, and extra attributes of a specific field.

# 3. Create a collection in customized setup mode

# 3.1. Create schema
schema = MilvusClient.create_schema(
    auto_id=False,
    enable_dynamic_field=True,
)

# 3.2. Add fields to schema
schema.add_field(field_name="my_id", datatype=DataType.INT64, is_primary=True)
schema.add_field(field_name="my_vector", datatype=DataType.FLOAT_VECTOR, dim=5)

# 3.3. Prepare index parameters
index_params = client.prepare_index_params()

# 3.4. Add indexes
index_params.add_index(
    field_name="my_id"
)

index_params.add_index(
    field_name="my_vector", 
    index_type="AUTOINDEX",
    metric_type="IP"
)

# 3.5. Create a collection
client.create_collection(
    collection_name="customized_setup",
    schema=schema,
    index_params=index_params
)

In the above setup, you have the flexibility to define various aspects of the collection during its creation, including its schema and index parameters.

  • Schema

    The schema defines the structure of a collection. Except for adding pre-defined fields and setting their attributes as demonstrated above, you have the option of enabling and disabling

    • Auto ID

      Whether to enable the collection to automatically increment the primary field.

    • Dynamic Field

      Whether to use the reserved JSON field $meta to store non-schema-defined fields and their values.

    For a detailed explanation of the schema, refer to Schema Explained.

  • Index parameters

    Index parameters dictate how Zilliz Cloud organizes your data within a collection. You can assign specific indexes to fields by configuring their metric types and index types.

    • For the vector field, you can use AUTOINDEX as the index type and use COSINE, L2, or IP as the metric_type.

    • For scalar fields, including the primary field, Zilliz Cloud uses TRIE for integers and STL_SORT for strings.

    For additional insights into index types, refer toAUTOINDEX Explained.

📘Notes

The collection created in the preceding code snippets are automatically loaded. If you prefer not to create an automatically loaded collection, refer to Create Collection.

Collections created using the RESTful API are always automatically loaded.

Insert Data​

Collections created in either of the preceding ways have been indexed and loaded. Once you are ready, insert some example data.

# 4. Insert data into the collection
# 4.1. Prepare data
data=[
    {"id": 0, "vector": [0.3580376395471989, -0.6023495712049978, 0.18414012509913835, -0.26286205330961354, 0.9029438446296592], "color": "pink_8682"},
    {"id": 1, "vector": [0.19886812562848388, 0.06023560599112088, 0.6976963061752597, 0.2614474506242501, 0.838729485096104], "color": "red_7025"},
    {"id": 2, "vector": [0.43742130801983836, -0.5597502546264526, 0.6457887650909682, 0.7894058910881185, 0.20785793220625592], "color": "orange_6781"},
    {"id": 3, "vector": [0.3172005263489739, 0.9719044792798428, -0.36981146090600725, -0.4860894583077995, 0.95791889146345], "color": "pink_9298"},
    {"id": 4, "vector": [0.4452349528804562, -0.8757026943054742, 0.8220779437047674, 0.46406290649483184, 0.30337481143159106], "color": "red_4794"},
    {"id": 5, "vector": [0.985825131989184, -0.8144651566660419, 0.6299267002202009, 0.1206906911183383, -0.1446277761879955], "color": "yellow_4222"},
    {"id": 6, "vector": [0.8371977790571115, -0.015764369584852833, -0.31062937026679327, -0.562666951622192, -0.8984947637863987], "color": "red_9392"},
    {"id": 7, "vector": [-0.33445148015177995, -0.2567135004164067, 0.8987539745369246, 0.9402995886420709, 0.5378064918413052], "color": "grey_8510"},
    {"id": 8, "vector": [0.39524717779832685, 0.4000257286739164, -0.5890507376891594, -0.8650502298996872, -0.6140360785406336], "color": "white_9381"},
    {"id": 9, "vector": [0.5718280481994695, 0.24070317428066512, -0.3737913482606834, -0.06726932177492717, -0.6980531615588608], "color": "purple_4976"}
]

# 4.2. Insert data
res = client.insert(
    collection_name="quick_setup",
    data=data
)

print(res)

# Output
#
# {
#     "insert_count": 10,
#     "ids": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
# }

The provided code assumes that you have created a collection in the Quick Setup manner. As shown in the above code,

  • The data to insert is organized into a list of dictionaries, where each dictionary represents a data record, termed as an entity.

  • Each dictionary contains a non-schema-defined field named color.

  • Each dictionary contains the keys corresponding to both pre-defined and dynamic fields.

📘Notes

Collections created using RESTful API enabled AutoID, and therefore you need to skip the primary field in the data to insert.

Insert more data​

You can safely skip this section if you prefer to search with the inserted 10 entities later. To learn more about the search performance of Zilliz Cloud clusters, you are advised use the following code snippet to add more randomly generated entities into the collection.

import time

# 5. Insert more data into the collection
# 5.1. Prepare data

colors = ["green", "blue", "yellow", "red", "black", "white", "purple", "pink", "orange", "brown", "grey"]
data = [ {
    "id": i, 
    "vector": [ random.uniform(-1, 1) for _ in range(5) ], 
    "color": f"{random.choice(colors)}_{str(random.randint(1000, 9999))}" 
} for i in range(1000) ]

# 5.2. Insert data
res = client.insert(
    collection_name="quick_setup",
    data=data[10:]
)

print(res)

# Output
#
# {
#     "insert_count": 990
# }

# Wait for a while
time.sleep(5)
📘Notes

You can insert a maximum of 100 entities in a batch upon each call to the Insert RESTful API.

You can conduct similarity searches based on one or more vector embeddings.

📘Notes

The insert operations are asynchronous, and conducting a search immediately after data insertions may result in empty result set. To avoid this, you are advised to wait for a few seconds.

The value of the query_vectors variable is a list containing a sub-list of floats. The sub-list represents a vector embedding of 5 dimensions.

# 6.1. Prepare query vectors
query_vectors = [
    [0.041732933, 0.013779674, -0.027564144, -0.013061441, 0.009748648]
]

# 6.2. Start search
res = client.search(
    collection_name="quick_setup",     # target collection
    data=query_vectors,                # query vectors
    limit=3,                           # number of returned entities
)

print(res)

# Output
#
# [
#     [
#         {
#             "id": 551,
#             "distance": 0.08821295201778412,
#             "entity": {}
#         },
#         {
#             "id": 296,
#             "distance": 0.0800950899720192,
#             "entity": {}
#         },
#         {
#             "id": 43,
#             "distance": 0.07794742286205292,
#             "entity": {}
#         }
#     ]
# ]

The output is a list containing a sub-list of three dictionaries, representing the returned entities with their IDs and distances.

You can also include multiple vector embeddings in the query_vectors variable to conduct a batch similarity search.

# 7. Search with multiple vectors
# 7.1. Prepare query vectors
query_vectors = [
    [0.041732933, 0.013779674, -0.027564144, -0.013061441, 0.009748648],
    [0.0039737443, 0.003020432, -0.0006188639, 0.03913546, -0.00089768134]
]

# 7.2. Start search
res = client.search(
    collection_name="quick_setup",
    data=query_vectors,
    limit=3,
)

print(res)

# Output
#
# [
#     [
#         {
#             "id": 551,
#             "distance": 0.08821295201778412,
#             "entity": {}
#         },
#         {
#             "id": 296,
#             "distance": 0.0800950899720192,
#             "entity": {}
#         },
#         {
#             "id": 43,
#             "distance": 0.07794742286205292,
#             "entity": {}
#         }
#     ],
#     [
#         {
#             "id": 730,
#             "distance": 0.04431751370429993,
#             "entity": {}
#         },
#         {
#             "id": 333,
#             "distance": 0.04231833666563034,
#             "entity": {}
#         },
#         {
#             "id": 232,
#             "distance": 0.04221535101532936,
#             "entity": {}
#         }
#     ]
# ]

The output should be a list of two sub-lists, each of which contains three dictionaries, representing the returned entities with their IDs and distances.

Filtered searches​

  • With schema-defined fields

    You can also enhance the search result by including a filter and specifying certain output fields in the search request.

    # 8. Search with a filter expression using schema-defined fields
    # 1 Prepare query vectors
    query_vectors = [
        [0.041732933, 0.013779674, -0.027564144, -0.013061441, 0.009748648]
    ]

    # 2. Start search
    res = client.search(
        collection_name="quick_setup",
        data=query_vectors,
        filter="500 < id < 800",
        limit=3
    )

    print(res)

    # Output
    #
    # [
    #     [
    #         {
    #             "id": 551,
    #             "distance": 0.08821295201778412,
    #             "entity": {}
    #         },
    #         {
    #             "id": 760,
    #             "distance": 0.07432225346565247,
    #             "entity": {}
    #         },
    #         {
    #             "id": 539,
    #             "distance": 0.07279646396636963,
    #             "entity": {}
    #         }
    #     ]
    # ]

    The output should be a list containing a sub-list of three dictionaries, each representing a searched entity with its ID, distance, and the specified output fields.

  • With non-schema-defined fields

    You can also include dynamic fields in a filter expression. In the following code snippet, color is a non-schema-defined field. You can include them either as keys in the magic $meta field, such as $meta["color"], or directly use it like a schema-defined field, such as color.

    # 9. Search with a filter expression using custom fields
    # 9.1.Prepare query vectors
    query_vectors = [
        [0.041732933, 0.013779674, -0.027564144, -0.013061441, 0.009748648]
    ]

    # 9.2.Start search
    res = client.search(
        collection_name="quick_setup",
        data=query_vectors,
        filter='$meta["color"] like "red%"',
        limit=3,
        output_fields=["color"]
    )

    print(res)

    # Output
    #
    # [
    #     [
    #         {
    #             "id": 263,
    #             "distance": 0.0744686871767044,
    #             "entity": {
    #                 "color": "red_9369"
    #             }
    #         },
    #         {
    #             "id": 381,
    #             "distance": 0.06509696692228317,
    #             "entity": {
    #                 "color": "red_9315"
    #             }
    #         },
    #         {
    #             "id": 360,
    #             "distance": 0.057343415915966034,
    #             "entity": {
    #                 "color": "red_6066"
    #             }
    #         }
    #     ]
    # ]

Scalar Query​

Unlike a vector similarity search, a query retrieves vectors via scalar filtering based on filter expressions.

  • With filter using schema-defined fields

    # 10. Query with a filter expression using a schema-defined field
    res = client.query(
        collection_name="quick_setup",
        filter="10 < id < 15",
        output_fields=["color"]
    )

    print(res)

    # Output
    #
    # [
    #     {
    #         "color": "yellow_4104",
    #         "id": 11
    #     },
    #     {
    #         "color": "blue_7278",
    #         "id": 12
    #     },
    #     {
    #         "color": "orange_7136",
    #         "id": 13
    #     },
    #     {
    #         "color": "pink_7776",
    #         "id": 14
    #     }
    # ]

  • With filter using non-schema-defined fields.

    # 11. Query with a filter expression using a custom field
    res = client.query(
        collection_name="quick_setup",
        filter='$meta["color"] like "brown_8%"',
        output_fields=["color"],
        limit=5
    )

    print(res)

    # Output
    #
    # [
    #     {
    #         "color": "brown_8454",
    #         "id": 17
    #     },
    #     {
    #         "color": "brown_8390",
    #         "id": 35
    #     },
    #     {
    #         "color": "brown_8442",
    #         "id": 309
    #     },
    #     {
    #         "color": "brown_8429",
    #         "id": 468
    #     },
    #     {
    #         "color": "brown_8020",
    #         "id": 472
    #     }
    # ]

Get Entities​

If you know the IDs of the entities to retrieve, you can get entities by their IDs as follows:

# 12. Get entities by IDs
res = client.get(
    collection_name="quick_setup",
    ids=[1,2,3],
    output_fields=["vector"]
)

print(res)

# Output
#
# [
#     {
#         "vector": [
#             0.19886813,
#             0.060235605,
#             0.6976963,
#             0.26144746,
#             0.8387295
#         ],
#         "id": 1
#     },
#     {
#         "vector": [
#             0.43742132,
#             -0.55975026,
#             0.6457888,
#             0.7894059,
#             0.20785794
#         ],
#         "id": 2
#     },
#     {
#         "vector": [
#             0.3172005,
#             0.97190446,
#             -0.36981148,
#             -0.48608947,
#             0.9579189
#         ],
#         "id": 3
#     }
# ]
📘Notes

Currently, the RESTful API does not provide a get endpoint.

Delete Entities​

Zilliz Cloud allows deleting entities by IDs and by filters.

  • Delete entities by IDs.

    # 13. Delete entities by IDs
    res = client.delete(
        collection_name="quick_setup",
        ids=[0,1,2,3,4]
    )

    print(res)

    # Output
    #
    # {
    #     "delete_count": 5
    # }

  • Delete entities by filter

    # 14. Delete entities by a filter expression
    res = client.delete(
        collection_name="quick_setup",
        filter="id in [5,6,7,8,9]"
    )

    print(res)

    # Output
    #
    # {
    #     "delete_count": 5
    # }
    📘Notes

    Currently, the delete endpoint of the RESTful API does not support filters.

Drop the collection​

The Free plan allows up to two collections in a cluster. Once you have done this guide, you can drop the collection as follows:

# 15. Drop collection
client.drop_collection(
    collection_name="quick_setup"
)

client.drop_collection(
    collection_name="customized_setup"
)

Recaps​

  • There are two ways to create a collection. The first is the quick setup, which only requires you to provide a name and the dimension of the vector field. The second is the customized setup, which allows you to customize almost every aspect of the collection.

  • The data insertion process may take some time to complete. It is recommended to wait a few seconds after inserting data and before conducting similarity searches.

  • Filter expressions can be used in both search and query requests. However, they are mandatory for query requests.