English

The english analyzer in Zilliz Cloud is designed to process English text, applying language-specific rules for tokenization and filtering.

Definition

The english analyzer uses the following components:

• Tokenizer: Uses the standard tokenizer to split text into discrete word units.

• Filters: Includes multiple filters for comprehensive text processing:

  • lowercase: Converts all tokens to lowercase, enabling case-insensitive searches.

  • stemmer: Reduces words to their root form to support broader matching (e.g., "running" becomes "run").

  • stop_words: Removes common English stop words to focus on key terms in text.

The functionality of the english analyzer is equivalent to the following custom analyzer configuration:

Python

python

analyzer_params = {
    "tokenizer": "standard",
    "filter": [
        "lowercase",
        {
            "type": "stemmer",
            "language": "english"
        },
        {
            "type": "stop",
            "stop_words": ["_english_"]
        }
    ]
}

Java

java

import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.Map;

Map<String, Object> analyzerParams = new HashMap<>();
analyzerParams.put("tokenizer", "standard");
analyzerParams.put("filter",
        Arrays.asList("lowercase",
                new HashMap<String, Object>() {{
                    put("type", "stemmer");
                    put("language", "english");
                }},
                new HashMap<String, Object>() {{
                    put("type", "stop");
                    put("stop_words", Collections.singletonList("_english_"));
                }}
        )
);

Go

go

analyzerParams := map[string]any{"tokenizer": "standard",
        "filter": []any{"lowercase", map[string]any{
            "type":     "stemmer",
            "language": "english",
        }, map[string]any{
            "type":       "stop",
            "stop_words": []string{"_english_"},
        }}}

Rust

rust

use serde_json::json;

let analyzer_params = json!({
    "tokenizer": "standard",
    "filter": [
        "lowercase",
        { "type": "stemmer", "language": "english" },
        { "type": "stop", "stop_words": ["_english_"] }
    ]
});

C++

cpp

nlohmann::json analyzer_params = {
    {"tokenizer", "standard"},
    {"filter", {
        "lowercase",
        {{"type", "stemmer"}, {"language", "english"}},
        {{"type", "stop"}, {"stop_words", {"_english_"}}}
    }}
};

NodeJS

javascript

const analyzer_params = {
    "tokenizer": "standard",
    "filter": [
        "lowercase",
        { "type": "stemmer", "language": "english" },
        { "type": "stop", "stop_words": ["_english_"] }
    ]
};

CSharp

csharp

using System.Collections.Generic;

var analyzerParams = new Dictionary<string, object>
{
    ["tokenizer"] = "standard",
    ["filter"] = new object[]
    {
        "lowercase",
        new Dictionary<string, object>
        {
            ["type"] = "stemmer",
            ["language"] = "english"
        },
        new Dictionary<string, object>
        {
            ["type"] = "stop",
            ["stop_words"] = new[] { "_english_" }
        }
    }
};

cURL

bash

# restful
analyzerParams='{
  "tokenizer": "standard",
  "filter": [
    "lowercase",
    {
      "type": "stemmer",
      "language": "english"
    },
    {
      "type": "stop",
      "stop_words": [
        "_english_"
      ]
    }
  ]
}'

Configuration

To apply the english analyzer to a field, simply set type to english in analyzer_params, and include optional parameters as needed.

Python

python

analyzer_params = {
    "type": "english",
}

Java

java

Map<String, Object> analyzerParams = new HashMap<>();
analyzerParams.put("type", "english");

Go

go

analyzerParams := map[string]any{"type": "english"}

Rust

rust

use serde_json::json;

let analyzer_params = json!({
    "type": "english"
});

C++

cpp

nlohmann::json analyzer_params = {
    {"type", "english"}
};

NodeJS

javascript

const analyzer_params = {
    "type": "english",
};

CSharp

csharp

var analyzerParams = new Dictionary<string, object>
{
    ["type"] = "english"
};

cURL

bash

# restful
analyzerParams='{
  "type": "english"
}'

The english analyzer accepts the following optional parameters:

Parameter	Description
stop_words	An array containing a list of stop words, which will be removed from tokenization. Defaults to english, a built-in set of common English stop words.

Example configuration with custom stop words:

Python

python

analyzer_params = {
    "type": "english",
    "stop_words": ["a", "an", "the"]
}

Java

java

Map<String, Object> analyzerParams = new HashMap<>();
analyzerParams.put("type", "english");
analyzerParams.put("stop_words", Arrays.asList("a", "an", "the"));

Go

go

analyzerParams := map[string]any{"type": "english", "stop_words": []string{"a", "an", "the"}}

Rust

rust

use serde_json::json;

let analyzer_params = json!({
    "type": "english",
    "stop_words": ["a", "an", "the"]
});

C++

cpp

nlohmann::json analyzer_params = {
    {"type", "english"},
    {"stop_words", {"a", "an", "the"}}
};

NodeJS

javascript

const analyzer_params = {
    "type": "english",
    "stop_words": ["a", "an", "the"]
};

CSharp

csharp

var analyzerParams = new Dictionary<string, object>
{
    ["type"] = "english",
    ["stop_words"] = new[] { "a", "an", "the" }
};

cURL

bash

# restful
analyzerParams='{
  "type": "english",
  "stop_words": [
    "a",
    "an",
    "the"
  ]
}'

After defining analyzer_params, you can apply them to a VARCHAR field when defining a collection schema. This allows Zilliz Cloud to process the text in that field using the specified analyzer for efficient tokenization and filtering. For details, refer to Example use.

Examples

Before applying the analyzer configuration to your collection schema, verify its behavior using the run_analyzer method.

Analyzer configuration

Python

python

analyzer_params = {
    "type": "english", # Uses the default `_english_` stop-word list
}

Java

java

Map<String, Object> analyzerParams = new HashMap<>();
analyzerParams.put("type", "english");

Go

go

analyzerParams := map[string]any{"type": "english"}

Rust

rust

use serde_json::json;

let analyzer_params = json!({
    "type": "english"
});

C++

cpp

nlohmann::json analyzer_params = {
    {"type", "english"}
};

NodeJS

javascript

const analyzer_params = {
    "type": "english", // Uses the default `_english_` stop-word list
};

CSharp

csharp

var analyzerParams = new Dictionary<string, object>
{
    ["type"] = "english" // Uses the default `_english_` stop-word list
};

cURL

bash

# restful
analyzerParams='{
  "type": "english"
}'

Verification using run_analyzer

Python

python

from pymilvus import (
    MilvusClient,
)

client = MilvusClient(
    uri="YOUR_CLUSTER_ENDPOINT",
    token="YOUR_CLUSTER_TOKEN"
)

# Sample text to analyze
sample_text = "Milvus is a vector database built for scale!"

# Run the english analyzer with the defined configuration
result = client.run_analyzer(sample_text, analyzer_params)
print("English analyzer output:", result)

Java

java

import java.util.*;
import io.milvus.v2.client.ConnectConfig;
import io.milvus.v2.client.MilvusClientV2;
import io.milvus.v2.service.vector.request.RunAnalyzerReq;
import io.milvus.v2.service.vector.response.RunAnalyzerResp;

ConnectConfig config = ConnectConfig.builder()
        .uri("YOUR_CLUSTER_ENDPOINT")
        .token("YOUR_CLUSTER_TOKEN")
        .build();
MilvusClientV2 client = new MilvusClientV2(config);

List<String> texts = new ArrayList<>();
texts.add("Milvus is a vector database built for scale!");

RunAnalyzerResp resp = client.runAnalyzer(RunAnalyzerReq.builder()
        .texts(texts)
        .analyzerParams(analyzerParams)
        .build());
List<RunAnalyzerResp.AnalyzerResult> results = resp.getResults();

for (RunAnalyzerResp.AnalyzerResult result : results) {
    for (RunAnalyzerResp.AnalyzerToken token : result.getTokens()) {
        System.out.println("English analyzer output: " + token.getToken());
    }
}

Go

go

import (
    "context"
    "fmt"

    "github.com/milvus-io/milvus/client/v3/milvusclient"
)

ctx := context.Background()

client, err := milvusclient.New(ctx, &milvusclient.ClientConfig{
    Address: "YOUR_CLUSTER_ENDPOINT",
    APIKey:  "YOUR_CLUSTER_TOKEN",
})
if err != nil {
    fmt.Println(err.Error())
    // handle error
}

texts := []string{"Milvus is a vector database built for scale!"}
option := milvusclient.NewRunAnalyzerOption(texts...).
    WithAnalyzerParams(analyzerParams)

result, err := client.RunAnalyzer(ctx, option)
if err != nil {
    fmt.Println(err.Error())
    // handle error
}

for _, r := range result {
    for _, token := range r.Tokens {
        fmt.Println("English analyzer output:", token.Text)
    }
}

Rust

rust

use milvus::v2::prelude::*;

let config = ConnectConfig::new().uri("YOUR_CLUSTER_ENDPOINT").token("YOUR_CLUSTER_TOKEN");
let client = ClientV2::new(&config).await?;

let response = client
    .run_analyzer(
        RunAnalyzerRequest::builder()
            .texts(["Milvus is a vector database built for scale!"])
            .analyzer_params(analyzer_params)
            .build()?,
    )
    .await?;

for result in response.results() {
    for token in result.get_tokens() {
        println!("English analyzer output: {}", token.get_text());
    }
}

C++

cpp

#include "milvus/MilvusClientV2.h"
#include <iostream>
#include <string>

auto client = milvus::MilvusClientV2::Create();

milvus::ConnectParam connect_param{"YOUR_CLUSTER_ENDPOINT", "YOUR_CLUSTER_TOKEN"};
auto status = client->Connect(connect_param);
if (!status.IsOk()) {
    std::cout << status.Message() << std::endl;
}

std::string text = "Milvus is a vector database built for scale!";
auto request = milvus::RunAnalyzerRequest()
                       .AddText(text)
                       .WithAnalyzerParams(analyzer_params);

milvus::RunAnalyzerResponse response;
status = client->RunAnalyzer(request, response);
if (!status.IsOk()) {
    std::cout << status.Message() << std::endl;
}

for (const auto& result : response.Results()) {
    for (const auto& token : result.Tokens()) {
        std::cout << "English analyzer output: " << token.token_ << std::endl;
    }
}

NodeJS

javascript

import { MilvusClient } from '@zilliz/milvus2-sdk-node';

const client = new MilvusClient({
  address: 'YOUR_CLUSTER_ENDPOINT',
  token: 'YOUR_CLUSTER_TOKEN',
});

const sample_text = 'Milvus is a vector database built for scale!';
const result = await client.runAnalyzer({
  text: sample_text,
  analyzer_params,
});
const tokens = result.results.flatMap(r => r.tokens.map(t => t.token));
console.log('English analyzer output:', tokens);

CSharp

csharp

using Milvus.Client.V2;
using Milvus.Client.V2.Requests.Utility;
using Milvus.Client.V2.Responses.Utility;
using Milvus.Client.V2.Types;

using var client = new MilvusClientV2(new ConnectConfig
{
    Uri = "YOUR_CLUSTER_ENDPOINT",
    ApiKey = "YOUR_CLUSTER_TOKEN"
});
await client.ConnectAsync();

// Sample text to analyze
var sampleText = "Milvus is a vector database built for scale!";

// Run the english analyzer with the defined configuration
RunAnalyzerResp result = await client.RunAnalyzerAsync(new RunAnalyzerReq
{
    AnalyzerParams = analyzerParams,
    Texts = new[] { sampleText }
});

foreach (AnalyzerResult r in result.Results)
{
    foreach (AnalyzerToken t in r.Tokens)
    {
        Console.WriteLine($"English analyzer output: {t.Token}");
    }
}

cURL

bash

curl --request POST \
  --url "YOUR_CLUSTER_ENDPOINT/v2/vectordb/common/run_analyzer" \
  --header "Authorization: Bearer YOUR_CLUSTER_TOKEN" \
  --header "Content-Type: application/json" \
  --data-raw '{
    "analyzerParams": "{\"type\": \"english\"}",
    "text": ["Milvus is a vector database built for scale!"]
  }'

Expected output

For the sample text, the analyzer produces the following tokens in order. SDK response objects and printed formatting vary by language; the REST response contains these tokens at data.results[].tokens[].token.

plaintext

['milvus', 'vector', 'databas', 'built', 'scale']