Skip to main content
Version: User Guides (BYOC)

Use JSON Fields

This guide explains how to use the JSON fields, such as inserting JSON values as well as searching and querying in JSON fields with basic and advanced operators.

Overview

JSON stands for Javascript Object Notation, which is a lightweight and simple text-based data format. Data in JSON is structured in key-value pairs, where every key is a string that maps to a value of a number, string, boolean, list, or array. With Zilliz Cloud clusters, it's possible to store dictionaries as a field value in collections.

For example, the following code randomly generates key-value pairs, each containing a JSON field with the key color.

# 3. Insert randomly generated vectors 
colors = ["green", "blue", "yellow", "red", "black", "white", "purple", "pink", "orange", "brown", "grey"]
data = []

for i in range(1000):
current_color = random.choice(colors)
current_tag = random.randint(1000, 9999)
current_coord = [ random.randint(0, 40) for _ in range(3) ]
current_ref = [ [ random.choice(colors) for _ in range(3) ] for _ in range(3) ]
data.append({
"id": i,
"vector": [ random.uniform(-1, 1) for _ in range(5) ],
"color": {
"label": current_color,
"tag": current_tag,
"coord": current_coord,
"ref": current_ref
}
})

print(data[0])

You can view the structure of the generated data by checking its first entry.

{
"id": 0,
"vector": [
-0.8017921296923975,
0.550046715206634,
0.764922589768134,
0.6371433836123146,
0.2705233937454232
],
"color": {
"label": "blue",
"tag": 9927,
"coord": [
22,
36,
6
],
"ref": [
[
"blue",
"green",
"white"
],
[
"black",
"green",
"pink"
],
[
"grey",
"black",
"brown"
]
]
}
}
📘Notes
  • Ensure that all values in a list or array are of the same data type.

  • Any nested dictionaries in a JSON field value will be considered strings.

  • Use only alphanumeric characters and underscores to name JSON keys, as other characters may cause problems with scalar-filtering and searches.

Define JSON field

To define a JSON field, simply follow the same procedure as defining fields of other types.

import random, time
from pymilvus import connections, MilvusClient, DataType

CLUSTER_ENDPOINT = "YOUR_CLUSTER_ENDPOINT"
TOKEN = "YOUR_CLUSTER_TOKEN"

# 1. Set up a Milvus client
client = MilvusClient(
uri=CLUSTER_ENDPOINT,
token=TOKEN
)

# 2. Create a collection
schema = MilvusClient.create_schema(
auto_id=False,
enable_dynamic_field=False,
)

schema.add_field(field_name="id", datatype=DataType.INT64, is_primary=True)
# The dim value should be greater than 1
schema.add_field(field_name="vector", datatype=DataType.FLOAT_VECTOR, dim=5)
# Define a JSON field
schema.add_field(field_name="color", datatype=DataType.JSON)

index_params = MilvusClient.prepare_index_params()

index_params.add_index(
field_name="id",
index_type="STL_SORT"
)

index_params.add_index(
field_name="vector",
index_type="AUTOINDEX",
metric_type="L2"
)

client.create_collection(
collection_name="test_collection",
schema=schema,
index_params=index_params
)

res = client.get_load_state(
collection_name="test_collection"
)

print(res)

# Output
#
# {
# "state": "<LoadState: Loaded>"
# }

Insert field values

After creating a collection, you can insert dictionaries such as the one demonstrated in Overview.

res = client.insert(
collection_name="test_collection",
data=data
)

print(res)

# Output
#
# {
# "insert_count": 1000,
# "ids": [
# 0,
# 1,
# 2,
# 3,
# 4,
# 5,
# 6,
# 7,
# 8,
# 9,
# "(990 more items hidden)"
# ]
# }

Basic scalar filtering

Once all of your data has been added, you can conduct searches and queries using the keys in the JSON field in the same manner as you would with a standard scalar field.

# 4. Basic search with a JSON field
query_vectors = [ [ random.uniform(-1, 1) for _ in range(5) ]]

res = client.search(
collection_name="test_collection",
data=query_vectors,
filter='color["label"] in ["red"]',
search_params={
"metric_type": "L2",
"params": {"nprobe": 16}
},
output_fields=["id", "color"],
limit=3
)

print(res)

# Output
#
# [
# [
# {
# "id": 460,
# "distance": 0.4016231596469879,
# "entity": {
# "id": 460,
# "color": {
# "label": "red",
# "tag": 5030,
# "coord": [14, 32, 40],
# "ref": [
# [ "pink", "green", "brown" ],
# [ "red", "grey", "black"],
# [ "red", "yellow", "orange"]
# ]
# }
# }
# },
# {
# "id": 785,
# "distance": 0.451080858707428,
# "entity": {
# "id": 785,
# "color": {
# "label": "red",
# "tag": 5290,
# "coord": [31, 13, 23],
# "ref": [
# ["yellow", "pink", "pink"],
# ["purple", "grey", "orange"],
# ["grey", "purple", "pink"]
# ]
# }
# }
# },
# {
# "id": 355,
# "distance": 0.5839247703552246,
# "entity": {
# "id": 355,
# "color": {
# "label": "red",
# "tag": 8725,
# "coord": [5, 10, 22],
# "ref": [
# ["white", "purple", "yellow"],
# ["white", "purple", "white"],
# ["orange", "white", "pink"]
# ]
# }
# }
# }
# ]
# ]

Advanced scalar filtering

Zilliz Cloud provides a set of advanced filters for scalar filtering in JSON fields. These filters are JSON_CONTAINS, JSON_CONTAINS_ALL, and JSON_CONTAINS_ANY.

  • Filters all entities that have ["blue", "brown", "grey"] as the reference color set.

    # 5. Advanced search within a JSON field

    res = client.query(
    collection_name="test_collection",
    data=query_vectors,
    filter='JSON_CONTAINS(color["ref"], ["blue", "brown", "grey"])',
    output_fields=["id", "color"],
    limit=3
    )

    print(res)

    # Output
    #
    # [
    # {
    # "id": 79,
    # "color": {
    # "label": "orange",
    # "tag": 8857,
    # "coord": [
    # 10,
    # 14,
    # 5
    # ],
    # "ref": [
    # [
    # "yellow",
    # "white",
    # "green"
    # ],
    # [
    # "blue",
    # "purple",
    # "purple"
    # ],
    # [
    # "blue",
    # "brown",
    # "grey"
    # ]
    # ]
    # }
    # },
    # {
    # "id": 371,
    # "color": {
    # "label": "black",
    # "tag": 1324,
    # "coord": [
    # 2,
    # 18,
    # 32
    # ],
    # "ref": [
    # [
    # "purple",
    # "orange",
    # "brown"
    # ],
    # [
    # "blue",
    # "brown",
    # "grey"
    # ],
    # [
    # "purple",
    # "blue",
    # "blue"
    # ]
    # ]
    # }
    # },
    # {
    # "id": 590,
    # "color": {
    # "label": "red",
    # "tag": 3340,
    # "coord": [
    # 13,
    # 21,
    # 13
    # ],
    # "ref": [
    # [
    # "yellow",
    # "yellow",
    # "red"
    # ],
    # [
    # "blue",
    # "brown",
    # "grey"
    # ],
    # [
    # "pink",
    # "yellow",
    # "purple"
    # ]
    # ]
    # }
    # }
    # ]
  • Filters entities that have the coordinator of [4, 5].

    res = client.query(
    collection_name="test_collection",
    data=query_vectors,
    filter='JSON_CONTAINS_ALL(color["coord"], [4, 5])',
    output_fields=["id", "color"],
    limit=3
    )

    print(res)

    # Output
    #
    # [
    # {
    # "id": 281,
    # "color": {
    # "label": "red",
    # "tag": 3645,
    # "coord": [
    # 5,
    # 33,
    # 4
    # ],
    # "ref": [
    # [
    # "orange",
    # "blue",
    # "pink"
    # ],
    # [
    # "purple",
    # "blue",
    # "purple"
    # ],
    # [
    # "black",
    # "brown",
    # "yellow"
    # ]
    # ]
    # }
    # },
    # {
    # "id": 464,
    # "color": {
    # "label": "brown",
    # "tag": 6261,
    # "coord": [
    # 5,
    # 9,
    # 4
    # ],
    # "ref": [
    # [
    # "purple",
    # "purple",
    # "brown"
    # ],
    # [
    # "black",
    # "pink",
    # "white"
    # ],
    # [
    # "brown",
    # "grey",
    # "brown"
    # ]
    # ]
    # }
    # },
    # {
    # "id": 567,
    # "color": {
    # "label": "green",
    # "tag": 4589,
    # "coord": [
    # 5,
    # 39,
    # 4
    # ],
    # "ref": [
    # [
    # "purple",
    # "yellow",
    # "white"
    # ],
    # [
    # "yellow",
    # "yellow",
    # "brown"
    # ],
    # [
    # "blue",
    # "red",
    # "yellow"
    # ]
    # ]
    # }
    # }
    # ]
  • Filters entities that have the coordinator containing either 4 or 5.

    res = client.query(
    collection_name="test_collection",
    data=query_vectors,
    filter='JSON_CONTAINS_ANY(color["coord"], [4, 5])',
    output_fields=["id", "color"],
    limit=3
    )

    print(res)

    # Output
    #
    # [
    # {
    # "id": 0,
    # "color": {
    # "label": "yellow",
    # "tag": 6340,
    # "coord": [
    # 40,
    # 4,
    # 40
    # ],
    # "ref": [
    # [
    # "purple",
    # "yellow",
    # "orange"
    # ],
    # [
    # "green",
    # "grey",
    # "purple"
    # ],
    # [
    # "black",
    # "white",
    # "yellow"
    # ]
    # ]
    # }
    # },
    # {
    # "id": 2,
    # "color": {
    # "label": "brown",
    # "tag": 9359,
    # "coord": [
    # 38,
    # 21,
    # 5
    # ],
    # "ref": [
    # [
    # "red",
    # "brown",
    # "white"
    # ],
    # [
    # "purple",
    # "red",
    # "brown"
    # ],
    # [
    # "pink",
    # "grey",
    # "black"
    # ]
    # ]
    # }
    # },
    # {
    # "id": 7,
    # "color": {
    # "label": "green",
    # "tag": 3560,
    # "coord": [
    # 5,
    # 9,
    # 5
    # ],
    # "ref": [
    # [
    # "blue",
    # "orange",
    # "green"
    # ],
    # [
    # "blue",
    # "blue",
    # "black"
    # ],
    # [
    # "green",
    # "purple",
    # "green"
    # ]
    # ]
    # }
    # }
    # ]

Reference on JSON filters

When working with JSON fields, you can either use the JSON fields as filters or some of its specific keys.

📘Notes
  • Zilliz Cloud stores string values in the JSON field as is without performing semantic escape or conversion.

For instance, 'a"b', "a'b", 'a\'b', and "a\"b" will be saved as is, while 'a'b' and "a"b" will be treated as invalid values.

  • To build filter expressions using a JSON field, you can utilize the keys within the field.

  • If a key's value is an integer or a float, you can compare it with another integer or float key or an INT32/64 or FLOAT32/64 field.

  • If a key's value is a string, you can compare it only with another string key or a VARCHAR field.

Basic Operators in JSON Fields

The following table assumes that the value of a JSON field named json_key has a key named A. Use it as a reference when constructing boolean expressions using JSON field keys.

Operator

Examples

Remarks

<

'json_field["A"] < 3'

This expression evaluates to true if the value of json_field["A"] is less than 3.

>

'json_field["A"] > 1'

This expression evaluates to true if the value of json_field["A"] is greater than 1.

==

'json_field["A"] == 1'

This expression evaluates to true if the value of json_field["A"] is equal to 1.

!=

'json_field["A"][0]' != "abc"'

This expression evaluates to true if

  • json_field does not have a key named A.

  • json_field has a key named A but json_field["A"] is not an array.

  • json_field["A"] is an empty array.

  • json_field["A"] is an array but the first element is not abc.

<=

'json_field["A"] <= 5'

This expression evaluates to true if the value of json_field["A"] is less than or equal to 5.

>=

'json_field["A"] >= 1'

This expression evaluates to true if the value of json_field["A"] is greater than or equal to 1.

not

'not json_field["A"] == 1'

This expression evaluates to true if

  • json_field does not have a key named A.

  • json_field["A"] is not equal to 1.

in

'json_field["A"] in [1, 2, 3]'

This expression evaluates to true if the value of json_field["A"] is 1, 2, or 3.

and (&&)

'json_field["A"] > 1 && json_field["A"] < 3'

This expression evaluates to true if the value of json_field["A"] is greater than 1 and less than 3.

or (||)

'json_field["A"] > 1 || json_field["A"] < 3'

This expression evaluates to true if the value of json_field["A"] is greater than 1 or less than 3.

exists

'exists json_field["A"]'

This expression evaluates to true if json_field does not have a key named A.

Advanced Operators

The following operators are specific to JSON fields:

  • json_contains(identifier, jsonExpr)

    This operator filters entities whose identifier contains the specified JSON expression.

    • Example 1: {"x": [1,2,3]}

      json_contains(x, 1) # => True (x contains 1.)
      json_contains(x, "a") # => False (x does not contain a member "a".)
    • Example 2: {"x", [[1,2,3], [4,5,6], [7,8,9]]}

      json_contains(x, [1,2,3]) # => True (x contains [1,2,3].)
      json_contains(x, [3,2,1]) # => False (x does contain a member [3,2,1].)
  • json_contains_all(identifier, jsonExpr)

    This operator filters entities whose identifier contains all the members of the JSON expression.

    Example: {"x": [1,2,3,4,5,7,8]}

    json_contains_all(x, [1,2,8]) # => True (x contains 1, 2, and 8.)
    json_contains_all(x, [4,5,6]) # => False (x does not has a member 6.)
  • json_contains_any(identifier, jsonExpr)

    This operator filters entities whose identifier contains any members of the JSON expression.

    Example: {"x": [1,2,3,4,5,7,8]}

    json_contains_any(x, [1,2,8]) # => True (x contains 1, 2, and 8.)
    json_contains_any(x, [4,5,6]) # => True (x contains 4 and 5.)
    json_contains_any(x, [6,9]) # => False (x contains none of 6 and 9.)