Skip to main content

Chinese

The chinese analyzer is designed specifically to handle Chinese text, providing effective segmentation and tokenization.

Definition​

The chinese analyzer consists of:

  • Tokenizer: Uses the jieba tokenizer to segment Chinese text into tokens based on vocabulary and context. For more information, refer to Jieba.

  • Filter: Uses the cnalphanumonly filter to remove tokens that contain any characters other than Chinese characters, English letters, or digits. For more information, refer to Cnalphanumonly.

The functionality of the chinese analyzer is equivalent to the following custom analyzer configuration:

📘Notes

The built-in chinese analyzer does not emit Pinyin tokens. To match Chinese text with Pinyin query terms, use a custom analyzer with the jieba tokenizer and the pinyin filter.

python
analyzer_params = {
"tokenizer": "jieba",
"filter": ["cnalphanumonly"]
}

Configuration​

To apply the chinese analyzer to a field, simply set type to chinese in analyzer_params.

python
analyzer_params = {
"type": "chinese",
}
📘Notes

The chinese analyzer does not accept any optional parameters.

Examples​

Before applying the analyzer configuration to your collection schema, verify its behavior using the run_analyzer method.

Analyzer configuration​

python
analyzer_params = {
"type": "chinese",
}

Verification using run_analyzer​

python
from pymilvus import (
MilvusClient,
)

client = MilvusClient(
uri="YOUR_CLUSTER_ENDPOINT",
token="YOUR_CLUSTER_TOKEN"
)

# Sample text to analyze
sample_text = "Milvus 是一个高性能、可扩展的向量数据库!"

# Run the chinese analyzer with the defined configuration
result = client.run_analyzer(sample_text, analyzer_params)
print("Chinese analyzer output:", result)

Expected output​

For the sample text, the analyzer produces the following tokens in order. SDK response objects and printed formatting vary by language; the REST response contains these tokens at data.results[].tokens[].token.

plaintext
['Milvus', '是', '一个', '高性', '性能', '高性能', '可', '扩展', '的', '向量', '数据', '据库', '数据库']