ランダムサンプリング

大規模なデータセットを扱う場合、インサイトを得たりフィルタリングロジックをテストしたりするために、すべてのデータを処理する必要は通常ありません。ランダムサンプリングは、統計的に代表的なデータのサブセットを扱えるようにすることで解決策を提供し、クエリ時間とリソース消費を大幅に削減します。

ランダムサンプリングはセグメントレベルで動作し、効率的なパフォーマンスを確保しながら、コレクションのデータ分布全体でサンプルのランダム性を維持します。

主なユースケース：

• データ探索：最小限のリソース使用量でコレクションの構造と内容をすばやくプレビューできます

• 開発時のテスト：完全にデプロイする前に、扱いやすいデータサンプルで複雑なフィルタリングロジックをテストできます

• リソースの最適化：探索的クエリと統計分析の計算コストを削減できます

構文

Python

python

filter = "RANDOM_SAMPLE(sampling_factor)"

Java

java

String filter = "RANDOM_SAMPLE(sampling_factor)";

Go

go

filter := "RANDOM_SAMPLE(sampling_factor)"

Rust

rust

let filter = "RANDOM_SAMPLE(sampling_factor)";

C++

c++

auto filter = "RANDOM_SAMPLE(sampling_factor)";

NodeJS

javascript

const filter = "RANDOM_SAMPLE(sampling_factor)";

cURL

bash

# restful
export filterRandomSample='RANDOM_SAMPLE(sampling_factor)'

パラメーター：

• sampling_factor: （0, 1）の範囲（境界値を除く）のサンプリング係数です。たとえば、RANDOM_SAMPLE(0.001) は結果の約 0.1% を選択します。

重要なルール：

• 式では大文字と小文字が区別されません（RANDOM_SAMPLE または random_sample）

• サンプリング係数は（0, 1）の範囲（境界値を除く）に指定する必要があります

他のフィルターとの組み合わせ

ランダムサンプリング演算子は、論理 AND を使用して他のフィルタリング式と組み合わせる必要があります。フィルターを組み合わせる場合、Milvus は最初に他の条件を適用し、その後で結果セットに対してランダムサンプリングを実行します。

Python

python

# Correct: Filter first, then sample
filter = 'color == "red" AND RANDOM_SAMPLE(0.001)'
# Processing: Find all red items → Sample 0.1% of those red items

# Incorrect: OR doesn't make logical sense
filter = 'color == "red" OR RANDOM_SAMPLE(0.001)'  # ❌ Invalid logic
# This would mean: "Either red items OR sample everything" - which is meaningless

Java

java

// Correct: Filter first, then sample
String filter = "color == \"red\" AND RANDOM_SAMPLE(0.001)";
// Processing: Find all red items → Sample 0.1% of those red items

// Incorrect: OR doesn't make logical sense
String filterIncorrect = "color == \"red\" OR RANDOM_SAMPLE(0.001)";  // ❌ Invalid logic
// This would mean: "Either red items OR sample everything" - which is meaningless

Go

go

// Correct: Filter first, then sample
filter := "color == \"red\" AND RANDOM_SAMPLE(0.001)"
// Processing: Find all red items → Sample 0.1% of those red items

filter = "color == \"red\" OR RANDOM_SAMPLE(0.001)" // ❌ Invalid logic
// This would mean: "Either red items OR sample everything" - which is meaningless

Rust

rust

// Correct: Filter first, then sample
let filter_correct = r#"color == "red" AND RANDOM_SAMPLE(0.001)"#;
// Processing: Find all red items → Sample 0.1% of those red items

// Incorrect: OR doesn't make logical sense
let filter_incorrect = r#"color == "red" OR RANDOM_SAMPLE(0.001)"#;  // ❌ Invalid logic
// This would mean: "Either red items OR sample everything" - which is meaningless

C++

c++

auto filter_sample_correct = R"(color == "red" AND RANDOM_SAMPLE(0.001))";
auto filter_sample_incorrect = R"(color == "red" OR RANDOM_SAMPLE(0.001))";

NodeJS

javascript

// Correct: Filter first, then sample
const filterCorrect = 'color == "red" AND RANDOM_SAMPLE(0.001)';
// Processing: Find all red items → Sample 0.1% of those red items

// Incorrect: OR doesn't make logical sense
const filterIncorrect = 'color == "red" OR RANDOM_SAMPLE(0.001)';  // ❌ Invalid logic
// This would mean: "Either red items OR sample everything" - which is meaningless

cURL

bash

# restful
# Correct: Filter first, then sample
export filterSampleCorrect='color == "red" AND RANDOM_SAMPLE(0.001)'
# Processing: Find all red items → Sample 0.1% of those red items

# Incorrect: OR doesn't make logical sense
export filterSampleIncorrect='color == "red" OR RANDOM_SAMPLE(0.001)'  # ❌ Invalid logic
# This would mean: "Either red items OR sample everything" - which is meaningless

例

例 1：データ探索

コレクションの構造をすばやくプレビューします：

Python

python

from pymilvus import MilvusClient

client = MilvusClient(uri="YOUR_CLUSTER_ENDPOINT")

# Sample approximately 1% of the entire collection
result = client.query(
    collection_name="product_catalog",
    filter="RANDOM_SAMPLE(0.01)",
    output_fields=["id", "product_name"],
    limit=10
)

print(f"Sampled {len(result)} products from collection")

Java

java

import io.milvus.v2.client.ConnectConfig;
import io.milvus.v2.client.MilvusClientV2;
import io.milvus.v2.service.vector.request.QueryReq;
import io.milvus.v2.service.vector.response.QueryResp;
import java.util.Arrays;
import java.util.List;

ConnectConfig config = ConnectConfig.builder()
        .uri("YOUR_CLUSTER_ENDPOINT")
        .build();
MilvusClientV2 client = new MilvusClientV2(config);

QueryReq queryReq = QueryReq.builder()
        .collectionName("product_catalog")
        .filter("RANDOM_SAMPLE(0.01)")
        .outputFields(Arrays.asList("id", "product_name"))
        .limit(10)
        .build();

QueryResp queryResp = client.query(queryReq);

List<QueryResp.QueryResult> results = queryResp.getQueryResults();
for (QueryResp.QueryResult result : results) {
    System.out.println(result.getEntity());
}

Go

go

import (
    "context"
    "fmt"

    "github.com/milvus-io/milvus/client/v3/entity"
    "github.com/milvus-io/milvus/client/v3/milvusclient"
)

ctx, cancel := context.WithCancel(context.Background())
defer cancel()

milvusAddr := "YOUR_CLUSTER_ENDPOINT"
client, err := milvusclient.New(ctx, &milvusclient.ClientConfig{
    Address: milvusAddr,
})
if err != nil {
    fmt.Println(err.Error())
    // handle error
}
defer client.Close(ctx)

resultSet, err := client.Query(ctx, milvusclient.NewQueryOption("product_catalog").
    WithFilter("RANDOM_SAMPLE(0.01)").
    WithOutputFields("id", "product_name"))
if err != nil {
    fmt.Println(err.Error())
    // handle error
}

fmt.Println("id: ", resultSet.GetColumn("id").FieldData().GetScalars())
fmt.Println("product_name: ", resultSet.GetColumn("product_name").FieldData().GetScalars())

Rust

rust

use milvus::v2::prelude::*;

let client = ClientV2::new(&ConnectConfig::new().uri("YOUR_CLUSTER_ENDPOINT")).await?;

// Sample approximately 1% of the entire collection
let res = client
    .query(
        QueryRequest::builder()
            .collection_name("product_catalog")
            .filter("RANDOM_SAMPLE(0.01)")
            .output_fields(["id", "product_name"])
            .limit(10)
            .build()?,
    )
    .await?;

println!("Sampled {} products from collection", res.results().get_row_count());

C++

c++

#include "milvus/MilvusClientV2.h"

auto client = milvus::MilvusClientV2::Create();

milvus::ConnectParam connect_param{"YOUR_CLUSTER_ENDPOINT"};
auto status = client->Connect(connect_param);
if (!status.IsOk()) {
    std::cout << status.Message() << std::endl;
}

auto request = milvus::QueryRequest()
                       .WithCollectionName("product_catalog")
                       .WithFilter("RANDOM_SAMPLE(0.01)")
                       .AddOutputField("id")
                       .AddOutputField("product_name")
                       .WithLimit(10);

milvus::QueryResponse response;
status = client->Query(request, response);
if (!status.IsOk()) {
    std::cout << status.Message() << std::endl;
}

NodeJS

javascript

import { MilvusClient } from "@zilliz/milvus2-sdk-node";

const client = new MilvusClient({ address: "YOUR_CLUSTER_ENDPOINT" });

// Sample approximately 1% of the entire collection
const res = await client.query({
    collection_name: "product_catalog",
    filter: "RANDOM_SAMPLE(0.01)",
    output_fields: ["id", "product_name"],
    limit: 10
});

console.log(`Sampled ${res.data.length} products from collection`);

cURL

bash

# restful
export TOKEN="YOUR_CLUSTER_TOKEN"
export CLUSTER_ENDPOINT="YOUR_CLUSTER_ENDPOINT"
export filterSample='RANDOM_SAMPLE(0.01)'

curl --request POST \
--url "${CLUSTER_ENDPOINT}/v2/vectordb/entities/query" \
--header "Authorization: Bearer ${TOKEN}" \
--header "Content-Type: application/json" \
--data "{
  \"collectionName\": \"product_catalog\",
  \"filter\": \"$filterSample\",
  \"outputFields\": [\"id\", \"product_name\"],
  \"limit\": 10
}"

例 2：ランダムサンプリングによるフィルターの組み合わせ

扱いやすいサブセットでフィルタリングロジックをテストします：

Python

python

# First filter by category and price, then sample 0.5% of results
filter_expression = 'category == "electronics" AND price > 100 AND RANDOM_SAMPLE(0.005)'

result = client.query(
    collection_name="product_catalog",
    filter=filter_expression,
    output_fields=["product_name", "price", "rating"],
    limit=10
)

print(f"Found {len(result)} electronics products in sample")

Java

java

import io.milvus.v2.service.vector.request.QueryReq;
import io.milvus.v2.service.vector.response.QueryResp;
import java.util.Arrays;

String filter = "category == \"electronics\" AND price > 100 AND RANDOM_SAMPLE(0.005)";

QueryReq queryReq = QueryReq.builder()
        .collectionName("product_catalog")
        .filter(filter)
        .outputFields(Arrays.asList("product_name", "price", "rating"))
        .limit(10)
        .build();

QueryResp queryResp = client.query(queryReq);

Go

go

import (
    "fmt"

    "github.com/milvus-io/milvus/client/v3/milvusclient"
)

filter := "category == \"electronics\" AND price > 100 AND RANDOM_SAMPLE(0.005)"

_, err := client.Query(ctx, milvusclient.NewQueryOption("product_catalog").
    WithFilter(filter).
    WithOutputFields("product_name", "price", "rating"))
if err != nil {
    fmt.Println(err.Error())
    // handle error
}

Rust

rust

// First filter by category and price, then sample 0.5% of results
let filter_expression = r#"category == "electronics" AND price > 100 AND RANDOM_SAMPLE(0.005)"#;

let res = client
    .query(
        QueryRequest::builder()
            .collection_name("product_catalog")
            .filter(filter_expression)
            .output_fields(["product_name", "price", "rating"])
            .limit(10)
            .build()?,
    )
    .await?;

println!("Found {} electronics products in sample", res.results().get_row_count());

C++

c++

#include "milvus/MilvusClientV2.h"

auto filter = R"(category == "electronics" AND price > 100 AND RANDOM_SAMPLE(0.005))";
auto request = milvus::QueryRequest()
                       .WithCollectionName("product_catalog")
                       .WithFilter(filter)
                       .AddOutputField("product_name")
                       .AddOutputField("price")
                       .AddOutputField("rating")
                       .WithLimit(10);

milvus::QueryResponse response;
auto status = client->Query(request, response);
if (!status.IsOk()) {
    std::cout << status.Message() << std::endl;
}

NodeJS

javascript

// First filter by category and price, then sample 0.5% of results
const filter_expression = 'category == "electronics" AND price > 100 AND RANDOM_SAMPLE(0.005)';

const res = await client.query({
    collection_name: "product_catalog",
    filter: filter_expression,
    output_fields: ["product_name", "price", "rating"],
    limit: 10
});

console.log(`Found ${res.data.length} electronics products in sample`);

cURL

bash

# restful
export filterComplex="category == \\\"electronics\\\" AND price > 100 AND RANDOM_SAMPLE(0.005)"

curl --request POST \
--url "${CLUSTER_ENDPOINT}/v2/vectordb/entities/query" \
--header "Authorization: Bearer ${TOKEN}" \
--header "Content-Type: application/json" \
--data "{
  \"collectionName\": \"product_catalog\",
  \"filter\": \"$filterComplex\",
  \"outputFields\": [\"product_name\", \"price\", \"rating\"]
}"

例 3：クイック分析

フィルタリングされたデータに対して迅速な統計分析を実行します：

Python

python

# Get insights from ~0.1% of premium customer data
filter_expression = 'customer_tier == "premium" AND region == "North America" AND RANDOM_SAMPLE(0.001)'

result = client.query(
    collection_name="customer_profiles",
    filter=filter_expression,
    output_fields=["purchase_amount", "satisfaction_score", "last_purchase_date"],
    limit=10
)

# Analyze sample for quick insights
if result:
    average_purchase = sum(r["purchase_amount"] for r in result) / len(result)
    average_satisfaction = sum(r["satisfaction_score"] for r in result) / len(result)

    print(f"Sample size: {len(result)}")
    print(f"Average purchase amount: ${average_purchase:.2f}")
    print(f"Average satisfaction score: ${average_satisfaction:.2f}")

Java

java

import io.milvus.v2.service.vector.request.QueryReq;
import io.milvus.v2.service.vector.response.QueryResp;
import java.util.Arrays;

String filter = "customer_tier == \"premium\" AND region == \"North America\" AND RANDOM_SAMPLE(0.001)";

QueryReq queryReq = QueryReq.builder()
        .collectionName("customer_profiles")
        .filter(filter)
        .outputFields(Arrays.asList("purchase_amount", "satisfaction_score", "last_purchase_date"))
        .limit(10)
        .build();

QueryResp queryResp = client.query(queryReq);

Go

go

import (
    "fmt"

    "github.com/milvus-io/milvus/client/v3/milvusclient"
)

filter := "customer_tier == \"premium\" AND region == \"North America\" AND RANDOM_SAMPLE(0.001)"

_, err := client.Query(ctx, milvusclient.NewQueryOption("customer_profiles").
    WithFilter(filter).
    WithOutputFields("purchase_amount", "satisfaction_score", "last_purchase_date"))
if err != nil {
    fmt.Println(err.Error())
    // handle error
}

Rust

rust

// Get insights from ~0.1% of premium customer data
let filter_expression = r#"customer_tier == "premium" AND region == "North America" AND RANDOM_SAMPLE(0.001)"#;

let res = client
    .query(
        QueryRequest::builder()
            .collection_name("customer_profiles")
            .filter(filter_expression)
            .output_fields(["purchase_amount", "satisfaction_score", "last_purchase_date"])
            .limit(10)
            .build()?,
    )
    .await?;

println!("Sample size: {}", res.results().get_row_count());

C++

c++

#include "milvus/MilvusClientV2.h"

auto filter = R"(customer_tier == "premium" AND region == "North America" AND RANDOM_SAMPLE(0.001))";
auto request = milvus::QueryRequest()
                       .WithCollectionName("customer_profiles")
                       .WithFilter(filter)
                       .AddOutputField("purchase_amount")
                       .AddOutputField("satisfaction_score")
                       .AddOutputField("last_purchase_date")
                       .WithLimit(10);

milvus::QueryResponse response;
auto status = client->Query(request, response);
if (!status.IsOk()) {
    std::cout << status.Message() << std::endl;
}

NodeJS

javascript

// Get insights from ~0.1% of premium customer data
const filter_expression = 'customer_tier == "premium" AND region == "North America" AND RANDOM_SAMPLE(0.001)';

const res = await client.query({
    collection_name: "customer_profiles",
    filter: filter_expression,
    output_fields: ["purchase_amount", "satisfaction_score", "last_purchase_date"],
    limit: 10
});

// Analyze sample for quick insights
if (res.data.length > 0) {
    const average_purchase = res.data.reduce((sum, r) => sum + r.purchase_amount, 0) / res.data.length;
    const average_satisfaction = res.data.reduce((sum, r) => sum + r.satisfaction_score, 0) / res.data.length;

    console.log(`Sample size: ${res.data.length}`);
    console.log(`Average purchase amount: $${average_purchase.toFixed(2)}`);
    console.log(`Average satisfaction score: ${average_satisfaction.toFixed(2)}`);
}

cURL

bash

# restful
export TOKEN="YOUR_CLUSTER_TOKEN"
export CLUSTER_ENDPOINT="YOUR_CLUSTER_ENDPOINT"
export filterCustomer="customer_tier == \\\"premium\\\" AND region == \\\"North America\\\" AND RANDOM_SAMPLE(0.001)"

curl --request POST \
--url "${CLUSTER_ENDPOINT}/v2/vectordb/entities/query" \
--header "Authorization: Bearer ${TOKEN}" \
--header "Content-Type: application/json" \
--data "{
  \"collectionName\": \"customer_profiles\",
  \"filter\": \"$filterCustomer\",
  \"outputFields\": [\"purchase_amount\", \"satisfaction_score\", \"last_purchase_date\"],
  \"limit\": 10
}"

例 4：ベクトル検索との組み合わせ

フィルター付き検索のシナリオでランダムサンプリングを使用します：

Python

python

# Search for similar products within a sampled subset
search_results = client.search(
    collection_name="product_catalog",
    data=[[0.1, 0.2, 0.3, 0.4, 0.5]],  # query vector
    filter='category == "books" AND RANDOM_SAMPLE(0.01)',
    search_params={"params": {}},
    output_fields=["title", "author", "price"],
    limit=10
)

print(f"Found {len(search_results[0])} similar books in sample")

Java

java

import io.milvus.v2.client.ConnectConfig;
import io.milvus.v2.client.MilvusClientV2;
import io.milvus.v2.service.vector.request.SearchReq;
import io.milvus.v2.service.vector.request.data.FloatVec;
import io.milvus.v2.service.vector.response.SearchResp;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;

MilvusClientV2 client = new MilvusClientV2(ConnectConfig.builder()
        .uri("YOUR_CLUSTER_ENDPOINT")
        .token("YOUR_CLUSTER_TOKEN")
        .build());

FloatVec queryVector = new FloatVec(new float[]{0.1f, 0.2f, 0.3f, 0.4f, 0.5f});
SearchReq searchReq = SearchReq.builder()
        .collectionName("product_catalog")
        .data(Collections.singletonList(queryVector))
        .topK(10)
        .filter("category == \"books\" AND RANDOM_SAMPLE(0.01)")
        .outputFields(Arrays.asList("title", "author", "price"))
        .build();

SearchResp searchResp = client.search(searchReq);

List<List<SearchResp.SearchResult>> searchResults = searchResp.getSearchResults();
for (List<SearchResp.SearchResult> results : searchResults) {
    System.out.println("TopK results:");
    for (SearchResp.SearchResult result : results) {
        System.out.println(result);
    }
}

Go

go

import (
    "fmt"

    "github.com/milvus-io/milvus/client/v3/entity"
    "github.com/milvus-io/milvus/client/v3/milvusclient"
)

queryVector := []float32{0.1, 0.2, 0.3, 0.4, 0.5}

resultSets, err := client.Search(ctx, milvusclient.NewSearchOption(
    "product_catalog", // collectionName
    10,               // limit
    []entity.Vector{entity.FloatVector(queryVector)},
).WithConsistencyLevel(entity.ClStrong).
    WithFilter("category == \"books\" AND RANDOM_SAMPLE(0.01)").
    WithOutputFields("title", "author", "price"))
if err != nil {
    fmt.Println(err.Error())
    // handle error
}

for _, resultSet := range resultSets {
    fmt.Println("title: ", resultSet.GetColumn("title").FieldData().GetScalars())
    fmt.Println("author: ", resultSet.GetColumn("author").FieldData().GetScalars())
    fmt.Println("price: ", resultSet.GetColumn("price").FieldData().GetScalars())
}

Rust

rust

// Search for similar products within a sampled subset
let res = client
    .search(
        SearchRequest::builder()
            .collection_name("product_catalog")
            .vector_field("vector")
            .vectors(SearchVectors::Float(vec![vec![0.1, 0.2, 0.3, 0.4, 0.5]]))
            .filter(r#"category == "books" AND RANDOM_SAMPLE(0.01)"#)
            .limit(10)
            .output_fields(["title", "author", "price"])
            .build()?,
    )
    .await?;

println!("Found {} similar books in sample", res.results().len());

C++

c++

#include "milvus/MilvusClientV2.h"

std::vector<float> query_vector = {0.1, 0.2, 0.3, 0.4, 0.5};
auto request = milvus::SearchRequest()
                   .WithCollectionName("product_catalog")
                   .WithLimit(10)
                   .WithFilter(R"(category == "books" AND RANDOM_SAMPLE(0.01))")
                   .AddOutputField("title")
                   .AddOutputField("author")
                   .AddOutputField("price")
                   .AddFloatVector(query_vector);

milvus::SearchResponse response;
auto status = client->Search(request, response);
if (!status.IsOk()) {
    std::cout << status.Message() << std::endl;
}

NodeJS

javascript

// Search for similar products within a sampled subset
const res = await client.search({
    collection_name: "product_catalog",
    data: [[0.1, 0.2, 0.3, 0.4, 0.5]],  // query vector
    filter: 'category == "books" AND RANDOM_SAMPLE(0.01)',
    limit: 10,
    output_fields: ["title", "author", "price"]
});

console.log(`Found ${res.results.length} similar books in sample`);

cURL

bash

# restful
export TOKEN="YOUR_CLUSTER_TOKEN"
export CLUSTER_ENDPOINT="YOUR_CLUSTER_ENDPOINT"

curl --request POST \
--url "${CLUSTER_ENDPOINT}/v2/vectordb/entities/search" \
--header "Authorization: Bearer ${TOKEN}" \
--header "Content-Type: application/json" \
--data '{
  "collectionName": "product_catalog",
  "data": [[0.1, 0.2, 0.3, 0.4, 0.5]],
  "annsField": "vector",
  "limit": 10,
  "filter": "category == \"books\" AND RANDOM_SAMPLE(0.01)",
  "outputFields": ["title", "author", "price"]
}' 

ベストプラクティス

• 小規模から始める：最初の探索には、より小さいサンプリング係数（0.001-0.01）を使用します

• 開発ワークフロー：開発中はサンプリングを使用し、本番環境のクエリではサンプリングを削除します

• 統計的な妥当性：サンプルが大きいほど、より正確な統計的表現が得られます

• パフォーマンステスト：クエリのパフォーマンスを監視し、必要に応じてサンプリング係数を調整します