Jieba

The jieba tokenizer processes Chinese text by breaking it down into its component words.

📘Notes

The jieba tokenizer preserves punctuation marks as separate tokens in the output. For example, "你好！世界。" becomes ["你好", "！", "世界", "。"]. To remove these standalone punctuation tokens, use the removepunct filter.

Configuration

Python

# Simple configuration: only specifying the tokenizer name
analyzer_params = {
    "tokenizer": "jieba",  # Use the default settings: dict=["_default_"], mode="search", hmm=True
}

Java

Map<String, Object> analyzerParams = new HashMap<>();
analyzerParams.put("tokenizer", "jieba");

NodeJS

const analyzer_params = {
    "tokenizer": "jieba",
};

Go

analyzerParams = map[string]any{"tokenizer": "jieba"}

cURL

# restful
analyzerParams='{
  "tokenizer": "jieba"
}'

Examples

Analyzer configuration

Python

analyzer_params = {
    "tokenizer": {
        "type": "jieba",
        "dict": ["结巴分词器"],
        "mode": "exact",
        "hmm": False
    }
}

Java

Map<String, Object> analyzerParams = new HashMap<>();
analyzerParams.put("type", "jieba");
analyzerParams.put("dict", Collections.singletonList("结巴分词器"));
analyzerParams.put("mode", "exact");
analyzerParams.put("hmm", false);

NodeJS

// javascript

Go

analyzerParams = map[string]any{"type": "jieba", "dict": []any{"结巴分词器"}, "mode": "exact", "hmm": false}

cURL

# restful

Expected output

['milvus', '结巴分词器', '中', '文', '测', '试']