Skip to main content

Create Pipeline

Create a pipeline.


POST
https://controller.api.{cloud-region}.zillizcloud.com/v1/pipelines

Example

Notes
  • This API requires an API Key as the authentication token.

Currently, data of the JSON and Array types are not supported in RESTful API requests..

  • Create a data ingestion pipeline.

    curl --request POST \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer ${API_KEY}" \
    --url "https://controller.api.{CLOUD_REGION}.zillizcloud.com/v1/pipelines" \
    -d '{
    "projectId": "proj-**********************",
    "name": "my_doc_ingestion_pipeline",
    "description": "A pipeline that splits a text file into chunks and generates embeddings. It also stores the publish_year with each chunk.",
    "type": "INGESTION",
    "functions": [
    {
    "name": "index_my_doc",
    "action": "INDEX_DOC",
    "inputField": "doc_url",
    "language": "ENGLISH",
    "chunkSize": 500
    },
    {
    "name": "keep_doc_info",
    "action": "PRESERVE",
    "inputField": "publish_year",
    "outputField": "publish_year",
    "fieldType": "Int16"
    }
    ],
    "clusterId": "${CLUSTER_ID}",
    "newCollectionName": "my_new_collection"
    }'

    Possible response:

    {
    "code": 200,
    "data": {
    "pipelineId": "pipe-**********************",
    "name": "my_doc_ingestion_pipeline",
    "type": "INGESTION",
    "description": "A pipeline that splits a text file into chunks and generates embeddings. It also stores the publish_year with each chunk.",
    "status": "SERVING",
    "totalTokenUsage": 0,
    "functions": [
    {
    "action": "INDEX_DOC",
    "name": "index_my_doc",
    "inputField": "doc_url",
    "language": "ENGLISH",
    "chunkSize": 500
    },
    {
    "action": "PRESERVE",
    "name": "keep_doc_info",
    "inputField": "publish_year",
    "outputField": "publish_year",
    "fieldType": "Int16"
    }
    ],
    "clusterId": "in03-***************",
    "newCollectionName": "my_new_collection"
    }
    }
  • Create a search pipeline

    curl --request POST \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer ${API_KEY}" \
    --url "https://controller.api.{CLOUD_REGION}.zillizcloud.com/v1/pipelines" \
    -d '{
    "name": "my_text_search_pipeline",
    "description": "A pipeline that receives text and search for semantically similar doc chunks",
    "type": "SEARCH",
    "functions": [
    {
    "name": "search_chunk_text_and_title",
    "action": "SEARCH_DOC_CHUNK",
    "inputField": "query_text",
    "clusterId": "${CLUSTER_ID}",
    "collectionName": "my_new_collection"
    }
    ]
    }'

    Possible response

    {
    "code": 200,
    "data": {
    "pipelineId": "pipe-**********************",
    "name": "my_text_search_pipeline",
    "type": "SEARCH",
    "description": "A pipeline that receives text and search for semantically similar doc chunks",
    "status": "SERVING",
    "functions": [
    {
    "action": "SEARCH_DOC_CHUNK",
    "name": "search_chunk_text_and_title",
    "inputField": "query_text",
    "clusterId": "in03-***************",
    "collectionName": "my_new_collection"
    }
    ]
    }
    }
  • Create a deletion pipeline

    curl --request POST \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer ${API_KEY}" \
    --url "https://controller.api.{CLOUD_REGION}.zillizcloud.com/v1/pipelines" \
    -d '{
    "name": "my_doc_deletion_pipeline",
    "description": "A pipeline that deletes all info associated with a doc",
    "type": "DELETION",
    "functions": [
    {
    "name": "purge_chunks_by_doc_name",
    "action": "PURGE_DOC_INDEX",
    "inputField": "doc_name"
    }
    ],

    "clusterId": "${CLUSTER_ID}",
    "collectionName": "my_new_collection"
    }'

    Possible response

    {
    "code": 200,
    "data": {
    "pipelineId": "pipe-**********************",
    "name": "my_doc_deletion_pipeline",
    "type": "DELETION",
    "description": "A pipeline that deletes all info associated with a doc",
    "status": "SERVING",
    "functions": [
    {
    "action": "PURGE_DOC_INDEX",
    "name": "purge_chunks_by_doc_name",
    "inputField": "doc_name"
    }
    ],
    "clusterId": "in03-***************",
    "collectionName": "my_new_collection"
    }
    }

Request

Parameters

  • No query parameters required

  • No path parameters required

Request Body

{
"projectId": "string",
"name": "string",
"type": "string",
"description": "string",
"functions": [
{
"name": "string",
"action": "string",
"inputField": "string",
"outputField": "string",
"fieldType": "string"
}
],
"clusterId": "string",
"newCollectionName": "string"
}
ParameterDescription
projectIdstring
ID of the target project of this operation.
namestring
Name of the pipeline to create.
typestring
Type of the pipeline to create. For an ingestion pipeline, the value should be INGESTION.
descriptionstring
Description of the pipeline to create.
functionsarray
Actions to take in the pipeline to create. For an ingestion pipeline, you can add only one doc-indexing function and multilpe preserve functions.
functions[]object
clusterIdstring
ID of a target cluster. You can find it in cluster details on Zilliz Cloud console.
newCollectionNamestring
Name of the collection to create in the specified cluster. Zilliz Cloud creates a new collection and name it using this value.
{
"projectId": "string",
"name": "string",
"description": "string",
"type": "string",
"functions": [
{
"name": "string",
"action": "string",
"clusterId": "string",
"collectionName": "string"
}
]
}
ParameterDescription
projectIdstring
ID of the target project of this operation.
namestring
Name of the pipeline to create.
descriptionstring
Description of the pipeline to create.
typestring
Type of the pipeline to create. For a search pipeline, the value should be SEARCH.
functionsarray
Actions to take in the search pipeline to create. You can define multiple functions to retrieve results from different collections.
functions[]object
functions[].namestring
Name of the function to create.
functions[].actionstring
Type of the function to create. For a search pipeline, possible value is SEARCH_DOC_CHUNK.
functions[].clusterIdstring
ID of a target collection in which Zilliz Cloud concducts the search.
functions[].collectionNamestring
Name of the collection in which ZIlliz Cloud conducts the search.
{
"projectId": "string",
"name": "string",
"description": "string",
"type": "string",
"functions": [
{
"name": "string",
"action": "string",
"inputField": "string"
}
],
"clusterId": "string",
"collectionName": "string"
}
ParameterDescription
projectIdstring
ID of the target project of this operation.
namestring
Name of the pipeline to create.
descriptionstring
Description of the pipeline to create.
typestring
Type of the pipeline to create. For a deletion pipeline, the value should be DELETION
functionsarray
Actions to take in the pipeline to create.
functions[]object
functions[].namestring
Name of the function to create.
functions[].actionstring
Type of the function to create. For a delete pipeline, possible value is PURGE_DOC_INDEX.
functions[].inputFieldstring
Name the field according to your needs. In a delete pipeline, use it for the name of a document to delete.
clusterIdstring
ID of a target cluster. You can find it in cluster details on Zilliz Cloud console.
collectionNamestring
Name of the collection to create in the specified cluster. Zilliz Cloud creates a new collection and name it using this value.

Response

Returns information about the pipeline just created.

Response Bodies

  • Response body if we process your request successfully
{
"code": "integer",
"data": {
"pipelineId": "integer",
"name": "string",
"type": "string",
"description": "string",
"status": "string",
"totalTokenUsage": "integer",
"clusterID": "string",
"collectionName": "string"
}
}
  • Response body if we failed to process your request
{
"code": integer,
"message": string
}

Properties

The properties in the returned response are listed in the following table.

PropertyDescription
codeinteger
dataobject
data.pipelineIdinteger
A pipeline ID.
data.namestring
Name of the pipeline.
data.typestring
Type of the pipeline. For an ingestion pipeline, the value should be INGESTION.
data.descriptionstring
Description of the pipeline.
data.statusstring
Current status of the pipeline. If the value is other than SERVING, the pipeline is not working.
data.totalTokenUsageinteger
Number of consumed tokens in this operation.
functionsobject
data.clusterIDstring
The target cluster to which the pipeline applies.
data.collectionNamestring
The target collection to which the pipeline applies.
messagestring
Indicates the possible reason for the reported error.

Possible Errors

CodeError Message
10041(Possible pipeline errors are all under this error code.)