Skip to main content
Version: User Guides (Cloud)

Sparse Vector

Sparse vectors are an important method of data representation in information retrieval and natural language processing. While dense vectors are popular for their excellent semantic understanding capabilities, sparse vectors often provide more accurate results when it comes to applications that require precise matching of keywords or phrases.

Overview

A sparse vector is a special representation of high-dimensional vectors where most elements are zero, and only a few dimensions have non-zero values. This characteristic makes sparse vectors particularly effective in handling large-scale, high-dimensional, but sparse data. Common applications include:

  • Text Analysis: Representing documents as bag-of-words vectors, where each dimension corresponds to a word, and only words that appear in the document have non-zero values.

  • Recommendation Systems: User-item interaction matrices, where each dimension represents a user's rating for a particular item, with most users interacting with only a few items.

  • Image Processing: Local feature representation, focusing only on key points in the image, resulting in high-dimensional sparse vectors.

As shown in the diagram below, dense vectors are typically represented as continuous arrays where each position has a value (e.g., [0.3, 0.8, 0.2, 0.3, 0.1]). In contrast, sparse vectors store only non-zero elements and their indices, often represented as key-value pairs (e.g., [{2: 0.2}, ..., {9997: 0.5}, {9999: 0.7}]). This representation significantly reduces storage space and increases computational efficiency, especially when dealing with extremely high-dimensional data (e.g., 10,000 dimensions).

RHlkwqC3Mh1tBSb7G3gcMc8PnUO

Sparse vectors can be generated using various methods, such as TF-IDF (Term Frequency-Inverse Document Frequency) and BM25 in text processing. Additionally, Zilliz Cloud offers convenient methods to help generate and process sparse vectors.

After vectorization, the data can be stored in Zilliz Cloud for management and vector retrieval. The diagram below illustrates the basic process.

QcKewe4XchpFUUblEyOcgEUpnbc

📘Notes

In addition to sparse vectors, Zilliz Cloud also supports dense vectors and binary vectors. Dense vectors are ideal for capturing deep semantic relationships, while binary vectors excel in scenarios like quick similarity comparisons and content deduplication. For more information, refer to Dense Vector and Binary Vector.

Use sparse vectors

Zilliz Cloud supports representing sparse vectors in any of the following formats:

  • Sparse Matrix (using the scipy.sparse class)

    from scipy.sparse import csr_matrix

    # Create a sparse matrix
    row = [0, 0, 1, 2, 2, 2]
    col = [0, 2, 2, 0, 1, 2]
    data = [1, 2, 3, 4, 5, 6]
    sparse_matrix = csr_matrix((data, (row, col)), shape=(3, 3))

    # Represent sparse vector using the sparse matrix
    sparse_vector = sparse_matrix.getrow(0)
  • List of Dictionaries (formatted as {dimension_index: value, ...})

    # Represent sparse vector using a dictionary
    sparse_vector = [{1: 0.5, 100: 0.3, 500: 0.8, 1024: 0.2, 5000: 0.6}]
  • List of Tuple Iterators (formatted as [(dimension_index, value)])

    # Represent sparse vector using a list of tuples
    sparse_vector = [[(1, 0.5), (100, 0.3), (500, 0.8), (1024, 0.2), (5000, 0.6)]]

Add vector field

To use sparse vectors in Zilliz Cloud clusters, define a field for storing sparse vectors when creating a collection. This process includes:

  1. Setting datatype to the supported sparse vector data type, SPARSE_FLOAT_VECTOR.

  2. No need to specify the dimension.

from pymilvus import MilvusClient, DataType

client = MilvusClient(uri="YOUR_CLUSTER_ENDPOINT")

client.drop_collection(collection_name="my_sparse_collection")

schema = client.create_schema(
auto_id=True,
enable_dynamic_fields=True,
)

schema.add_field(field_name="pk", datatype=DataType.VARCHAR, is_primary=True, max_length=100)
schema.add_field(field_name="sparse_vector", datatype=DataType.SPARSE_FLOAT_VECTOR)

In this example, a vector field named sparse_vector is added for storing sparse vectors. The data type of this field is SPARSE_FLOAT_VECTOR.

Set index params for vector field

The process of creating an index for sparse vectors is similar to that for dense vectors, but with differences in the specified index type (index_type), distance metric (metric_type), and index parameters (params).

index_params = client.prepare_index_params()

index_params.add_index(
field_name="sparse_vector",
index_name="sparse_inverted_index",
index_type="SPARSE_INVERTED_INDEX",
metric_type="IP",
params={"drop_ratio_build": 0.2},
)

In the example above:

  • An index of type SPARSE_INVERTED_INDEX is created for the sparse vector. For sparse vectors, you can specify SPARSE_INVERTED_INDEX or SPARSE_WAND. For details, refer to xx.

  • For sparse vectors, metric_type only supports IP (Inner Product), used to measure the similarity between two sparse vectors. For more information on similarity, refer to Metric Types.

  • drop_ratio_build is an optional index parameter specifically for sparse vectors. It controls the proportion of small vector values excluded during index building. For example, with {"drop_ratio_build": 0.2}, the smallest 20% of vector values will be excluded during index creation, reducing computational effort during searches.

Create collection

Once the sparse vector and index settings are complete, you can create a collection that contains sparse vectors. The example below uses the create_collection method to create a collection named my_sparse_collection.

client.create_collection(
collection_name="my_sparse_collection",
schema=schema,
index_params=index_params
)

Insert data

After creating the collection, insert data containing sparse vectors.

sparse_vectors = [
{"sparse_vector": {1: 0.5, 100: 0.3, 500: 0.8}},
{"sparse_vector": {10: 0.1, 200: 0.7, 1000: 0.9}},
]

client.insert(
collection_name="my_sparse_collection",
data=sparse_vectors
)

To perform similarity search using sparse vectors, prepare the query vector and search parameters.

# Prepare search parameters
search_params = {
"params": {"drop_ratio_search": 0.2}, # Additional optional search parameters
}

# Prepare the query vector
query_vector = [{1: 0.2, 50: 0.4, 1000: 0.7}]

In this example, drop_ratio_search is an optional parameter specifically for sparse vectors, allowing fine-tuning of small values in the query vector during the search. For example, with {"drop_ratio_search": 0.2}, the smallest 20% of values in the query vector will be ignored during the search.

Then, execute the similarity search using the search method:

res = client.search(
collection_name="my_sparse_collection",
data=query_vector,
limit=3,
output_fields=["pk"],
search_params=search_params,
)

print(res)

# Output
# data: ["[{'id': '453718927992172266', 'distance': 0.6299999952316284, 'entity': {'pk': '453718927992172266'}}, {'id': '453718927992172265', 'distance': 0.10000000149011612, 'entity': {'pk': '453718927992172265'}}]"]

For more information on similarity search parameters, refer to Basic ANN Search.