Import from a JSON file
JSON (JavaScript Object Notation) is a lightweight, human-readable data format that machines can parse and generate easily. Language-independent, it follows conventions familiar to C-family language programmers, making it an ideal data interchange format.
You are advised to use the BulkWriter tool to prepare your raw data into JSON files. The following figure demonstrates how your raw data can be mapped into a JSON file.
- Whether to enable AutoID
The id field serves as the primary field of the collection. To make the primary field automatically increment, you can enable AutoID in the schema. In this case, you should exclude the id field from each row in the source data.
- Whether to enable dynamic fields
When the target collection enables dynamic fields, if you need to store fields that are not included in the pre-defined schema, you can specify the $meta column during the write operation and provide the corresponding key-value data.
- Case-sensitive
Dictionary keys and collection field names are case-sensitive. Ensure that the dictionary keys in your data exactly match the field names in the target collection. If there is a field named id in the target collection, each entity dictionary should have a key named id. Using ID or Id results in errors.
Directory structure​
If you prefer to prepare your data into JSON files, place all JSON files directly into the source data folder as shown in the tree diagram below.
├── json-folder
│ ├── 1.json
│ └── 2.json
Import data​
Once your data is ready, you can use either of the following methods to import them into your Zilliz Cloud collection.
If your files are relatively small, it is recommended to use the folder or multiple-path method to import them all at once. This approach allows for internal optimizations during the import process, which helps reduce resource consumption later.
You can also import your data on the Zilliz Cloud console using Milvus SDKs. For details, refer to Import Data (Console) and Import Data (SDK).
Import files from multiple paths (Recommended)​
When importing files from multiple paths, include each JSON file path in a separate list, then group all the lists into a higher-level list as in the following code example.
curl --request POST \
--url "https://api.cloud.zilliz.com/v2/vectordb/jobs/import/create" \
--header "Authorization: Bearer ${TOKEN}" \
--header "Accept: application/json" \
--header "Content-Type: application/json" \
-d '{
"clusterId": "inxx-xxxxxxxxxxxxxxx",
"collectionName": "medium_articles",
"partitionName": "",
"objectUrls": [
["s3://bucket-name/json-folder-1/1.json"],
["s3://bucket-name/json-folder-2/1.json"],
["s3://bucket-name/json-folder-3/"]
],
"accessKey": "",
"secretKey": ""
}'
Import files from a folder​
If the source folder contains only the JSON files to import, you can simply include the source folder in the request as follows:
curl --request POST \
--url "https://api.cloud.zilliz.com/v2/vectordb/jobs/import/create" \
--header "Authorization: Bearer ${TOKEN}" \
--header "Accept: application/json" \
--header "Content-Type: application/json" \
-d '{
"clusterId": "inxx-xxxxxxxxxxxxxxx",
"collectionName": "medium_articles",
"partitionName": "",
"objectUrls": [
["s3://bucket-name/json-folder/"]
],
"accessKey": "",
"secretKey": ""
}'
Import a single file​
If your prepared data file is a single JSON file, import it as demonstrated in the following code example.
curl --request POST \
--url "https://api.cloud.zilliz.com/v2/vectordb/jobs/import/create" \
--header "Authorization: Bearer ${TOKEN}" \
--header "Accept: application/json" \
--header "Content-Type: application/json" \
-d '{
"clusterId": "inxx-xxxxxxxxxxxxxxx",
"collectionName": "medium_articles",
"partitionName": "",
"objectUrls": [
["s3://bucket-name/json-folder/1.json"]
],
"accessKey": "",
"secretKey": ""
}'
Storage paths​
Zilliz Cloud supports data import from your cloud storage. The table below lists the possible storage paths for your data files.
Cloud | Quick Examples |
|---|---|
AWS S3 |
|
Google Cloud Storage |
|
Azure Bolb |
|
Limits​
There are some limits you need to observe when you import data in the JSON format from your cloud storage. Note that a valid JSON file has a root key named rows, the corresponding value of which is a list of dictionaries, each representing an entity that matches the schema of the target collection.
Item | Description |
|---|---|
Multiple files per import | Yes |
Maximum file size per import | Free cluster: 512 MB in total Serverless and Dedicated cluster:
|
Applicable data file locations | Local and remote files |
You can either rebuild your data on your own by referring to Prepare the data file or use the BulkWriter tool to generate the source data file. Click here to download the prepared sample data based on the schema in the above diagram.