DOC-14050: Document GSI sparse vector indexing

modules/vector-index/pages/composite-vector-index.adoc

@@ -53,13 +53,9 @@ See xref:vector-index:vectors-and-indexes-overview.adoc#vector_similarity[Vector
 include::vector-search:partial$download-sample-partial.adoc[]
 
 [#create-index]
-// TODO: For uptake in Couchbase Capella
-////
-include::vector-index:partial$create-composite-index-capella-ui.adoc[opts=optional]
+include::vector-index:partial$create-composite-index-ui.adoc[opts=optional]
 
 == Create a Composite Vector Index Using {sqlpp}
-////
-== Create a Composite Vector Index
 
 Creating a Composite Vector index is similar to creating a non-vector GSI index.
 See xref:guides:create-index.adoc[] for an overview of creating indexes.

modules/vector-index/pages/hyperscale-vector-index.adoc

@@ -64,13 +64,9 @@ See xref:vector-index:vectors-and-indexes-overview.adoc#vector_similarity[Vector
 include::vector-search:partial$download-sample-partial.adoc[]
 
 [#create-index]
-// TODO: For uptake in Couchbase Capella
-////
-include::vector-index:partial$create-hyperscale-index-capella-ui.adoc[opts=optional]
+include::vector-index:partial$create-hyperscale-index-ui.adoc[opts=optional]
 
 == Create a Hyperscale Vector Index Using {sqlpp}
-////
-== Create a Hyperscale Vector Index
 
 Use the `CREATE VECTOR INDEX` statement to create a Hyperscale Vector Index.
 This statement is similar to the `CREATE INDEX` statement that you use to create Global Secondary Indexes (GSI).

modules/vector-index/pages/use-vector-indexes.adoc

@@ -5,6 +5,9 @@
 :description: Use Couchbase {page-product-name}'s vector indexes to find documents based on content similarity or semantic meaning.
 :page-toclevels: 3
 
+// TEMP
+:totoro: the next major release of Couchbase Server
+
 [abstract]
 {description}
 
@@ -172,7 +175,6 @@ Hyperscale Vector indexes can handle the large datasets and perform these vector
 
 === Hyperscale Vector Index Application Workflow
 
-
 To create and use a Hyperscale Vector index, your application follows the workflow shown in the following diagram:
 
 .Application Workflow with Hyperscale Vector Indexes
@@ -193,6 +195,10 @@ The steps shown in the diagram are:
 It sends the results of the scan back to the Query Service.
 . The Query Service returns query results to your application.
 
+In {totoro} and later, you can create a Hyperscale Vector Index with dense vectors or sparse vectors.
+You can use a dense Hyperscale Vector Index or a sparse Hyperscale Vector Index for individual queries.
+You can also combine the results of a query using a dense Hyperscale Vector Index and a query using a sparse Hyperscale Vector Index using Reciprocal Rank Fusion (RRF).
+
 See xref:vector-index:hyperscale-vector-index.adoc[] for more information about Hyperscale Vector indexes.
 
 === Composite Vector Index Applications 
@@ -236,6 +242,10 @@ It then performs the vector similarity search on the results of the scalar index
 Finally, it sends the results of the similarity search back to the Query Service.
 . The Query Service returns the query results to your application.
 
+In {totoro} and later, you can create a Composite Vector Index with dense vectors or sparse vectors.
+You can use a dense Composite Vector Index or a sparse Composite Vector Index for individual queries.
+You can also combine the results of a query using a dense Composite Vector Index and a query using a sparse Composite Vector Index using Reciprocal Rank Fusion (RRF).
+
 See xref:vector-index:composite-vector-index.adoc[] for more information about Composite Vector indexes.
 
 [#fts]

modules/vector-index/pages/vectors-and-indexes-overview.adoc

@@ -5,9 +5,10 @@
 :page-product-name: {product-name}
 :description: This page is a high-level overview of vectors and how they work in indexes.
 :page-toclevels: 3
+:imagesdir: ../assets/images
 
 // TEMP
-:totoro: the next major release of Couchbase Server
+:totoro: The next major release of Couchbase Server
 
 [abstract]
 {description}
@@ -155,7 +156,7 @@ However, it does not take the square root of the sum distances between the vecto
 
 
 Euclidean Distance Formula::
- latexmath:[L2(x, y) = \sqrt{\sum_{i=1}^n (x_i - y_i)^2}]
+latexmath:[L2(x, y) = \sqrt{\sum_{i=1}^n (x_i - y_i)^2}]
 
 
 Euclidean Squared Distance Formula::
@@ -196,7 +197,7 @@ The magnitude of the vectors is important when determining the strength of the r
 For example:
 
 * Recommending related products based on a shopper's previous purchases.
-* Personalizing a music service's user home page  based on the user's likes and dislikes of artists and songs.
+* Personalizing a music service's user home page based on the user's likes and dislikes of artists and songs.
 * Targeting ads based on a user's interests and previous interactions with the service.
 
 [#cosine]
@@ -420,3 +421,30 @@ After Couchbase {product-name} builds and trains the new index, you can drop the
 You should consider the need to retrain your indexes when choosing which quantization to use.
 For example, the PQ quantization requires more resources to train than SQ.
 If your dataset evolves rapidly, you may choose to not use PQ because you'll need to perform more frequent retraining.
+
+== Dense Vectors and Sparse Vectors
+
+[.status]
+{totoro}
+
+By default, Hyperscale Vector Indexes and Composite Vector Indexes are ((dense vector indexes)).
+Dense vectors have hundreds or thousands of dimensions, and most dimensions contain a value.
+This means that dense vector indexes capture semantic similarity and generalize across concepts, but that they may miss exact matches for keywords or product-specific terms.
+
+{totoro} introduces ((sparse vector indexes)).
+In a sparse vector, most of the dimensions have zero values.
+Only a few of the dimensions are non-zero, and each non-zero dimensions maps a specific token to a weight.
+A token is equivalent to a word, part of a word, or a contiguous sequence of words or characters.
+This means that a sparse vector enables you to search for rare keywords or specialized terms.
+
+=== Combine Dense Vector Search and Sparse Vector Search
+
+By itself, a dense vector search or a sparse vector search may not provide the results you need.
+A dense vector search is good for broad semantic queries, but may not match rare, specialized tokens; whereas a sparse vector search can match specific search terms and brand names but may miss general semantic similarities.
+
+For most AI applications, including retrieval-augmented generation (RAG) pipelines and agentic AI systems, a multi-stage approach is required.
+This chains together dense vector search and sparse vector search:
+
+. Find the dense scores: cosine similarity of dense embeddings.
+. Find the sparse scores: dot product of term weights.
+. Use Reciprocal Rank Fusion (RRF) to combine the dense scores and sparse scores in a single ranking.

modules/vector-index/partials/create-composite-index-ui.adoc

@@ -0,0 +1,104 @@
+== Create a Composite Vector Index with the Capella UI
+
+To create a Composite Vector index with the Capella UI: 
+
+. On the *Operational Clusters* page, select the operational cluster where you want to create a Composite Vector index.
+. Go to menu:Data Tools[Vector Indexes].
+. Click btn:[Create Vector Index].
+. Choose btn:[Composite Vector Index].
+. In the *Index Name* field, enter a name for your new Vector index.
++
+[NOTE]
+==== 
+Your index name must start with an alphabetic character (a-z or A-Z). It can only contain alphanumeric characters (a-z, A-Z, or 0-9), hyphens (-), or underscores (_).
+
+Your index name must be unique inside your selected bucket and scope.
+You cannot have 2 indexes with the same name inside the same bucket and scope.
+====
+. In the *Bucket*, *Scope*, and *Collection* lists, select the bucket, scope, and collection where you have documents that contain a vector field.
+
+. Under *Vector Field*, in the *Select Field* list, select the vector field to use in your Composite Vector index.
+
+. [[filter-fields]](Optional) To add a field to use as a filter on the data included in your Composite Vector index, click btn:[+ Add Filter Field].
+
+.. Under *Filter Fields*, select the field you want to use to filter your documents before running vector comparisons.
+
+.. Under *Include in Index*, choose whether the previously selected field should be included in your index.
++
+You can still choose to partition using this field and not include it in your index.
+
+.. Under *Partition by Hash*, choose whether the previously selected field should be used to partition your index, based on the values in your documents.
++
+For example, you could partition your Composite Vector index based on the value of a `supplier` field in your documents.
+If you choose *No*, you can still filter using a `WHERE` clause, but your index will not be partitioned based on field values.
+
+.. (Optional) Under *Add WHERE clause*, enter a clause for the specific values that a document must contain in your filter field to be included in your index.
++
+For example, you could filter your index so that only documents with `supplier = "HP"` or `brightness > 0.5` are included.
+
+. Under *Index Configuration*, choose options to configure additional settings for your Composite Vector index: 
+
+.. In the *Similarity Metric* list, select the similarity metric that you want to use for comparing vectors in your index.
+For the best accuracy, use the same similarity that you plan to use to query your data.
++
+For more information about vector similarity measures, see xref:vector-index:vectors-and-indexes-overview.adoc#vector_similarity[Vector Similarity].
+
+.. In the *Quantization* list, select the type of quantization to use for simplifying the vectors stored in your index.
++
+For more information about quantization, see xref:vector-index:vectors-and-indexes-overview.adoc#quantization[Quantization].
+For more information about how to choose quantization on your index, see xref:vector-index:vectors-and-indexes-overview.adoc#choosing-a-quantization-method[Choosing a Quantization Method] and xref:vector-index:vector-index-best-practices.adoc#quantization[Tuning Index Creation - Quantization].
+
+.. [[dimensions]] In the *Dimensions* field, enter the exact dimension of the vectors in your data.
++
+Your embedding model determines this value, and Capella automatically configures it based on your chosen *Vector Field*.
+
+.. In the *Replicas* list, select the number of replicas you want to create for your index.
++
+Replicas affect the query throughput, memory footprint, and fault tolerance for your Composite Vector index.
+More replicas increase the required memory for your index, but can increase query throughput.
+
+.. [[training-list]](Optional) In the *Training List* field, enter or select the number of vectors to consider when searching for centroids in your data.
++
+[TIP]
+====
+If you have less than 10,000 vectors in your index, Couchbase sets the *Training List* value to your number of vectors.
+If you have more than 10,000 vectors up to a maximum of 1,000,000, Couchbase sets the *Training List* value to either 10% of your total number of vectors or 10 times your number of centroids - whichever value is higher, up to a maximum of 1,000,000.
+
+You can change your *Training List* value from these recommended defaults.
+Increasing the value can improve recall, but increase build times.
+Decreasing the value can improve build times, but reduces recall.
+====
+
+
+.. [[probes]](Optional) In the *Probes to scan* field, enter or select the number of centroids to check for similar vectors for each query.
++
+The default value is *1*.
+A higher value increases search times, but also increases accuracy.
+You can also choose to override this value in your queries.
+
+.. (Optional) In the *Number of centroids* field, enter or select the number of centroids to create in your index.
++
+A larger value creates smaller centroids.
+By default, Couchbase sets the *Number of centroids* to the number of vectors in your data divided by 1000.
+For more information about how to set this value, see xref:vector-index:composite-vector-index.adoc#number-of-centroids[Number of Centroids].
++
+////
+.. To store full vectors in the index, select *Persist Full Vector*.
++
+This increases the storage required for your index but increases accuracy.
+////
+. Click btn:[Review Index].
+. Confirm your Composite Vector Index configuration. 
+. Click btn:[Create Index] or copy the {sqlpp} syntax to create your index with the Query Workbench.
+
+=== Resolve Index Creation Errors 
+
+If an error occurs while Capella tries to create your Composite Vector index, on the *Vector Indexes* page, the *Status* of your index changes to *Error*. 
+
+To view the details of the error: 
+
+. Click the *Error* status to open the Query Workbench.
+. Click btn:[Run] to rebuild the index.
+. In the query results, find the error details. 
+
+Depending on the error, you might need to try recreating your index with a new configuration. 

modules/vector-index/partials/create-hyperscale-index-ui.adoc

@@ -0,0 +1,101 @@
+== Create a Hyperscale Vector Index with the Capella UI
+
+To create a Hyperscale Vector index with the Capella UI: 
+
+. On the *Operational Clusters* page, select the operational cluster where you want to create a Hyperscale Vector index.
+. Go to menu:Data Tools[Vector Indexes].
+. Click btn:[Create Vector Index].
+. Choose btn:[Hyperscale Vector Index].
+. In the *Index Name* field, enter a name for your new Vector index.
++
+[NOTE]
+==== 
+Your index name must start with an alphabetic character (a-z or A-Z). It can only contain alphanumeric characters (a-z, A-Z, or 0-9), hyphens (-), or underscores (_).
+
+Your index name must be unique inside your selected bucket and scope.
+You cannot have 2 indexes with the same name inside the same bucket and scope.
+====
+. In the *Bucket*, *Scope*, and *Collection* lists, select the bucket, scope, and collection where you have documents that contain a vector field.
+
+. Under *Vector Field*, in the *Select Field* list, select the vector field to use in your Hyperscale Vector index.
+
+. [[filter-fields]](Optional) To add a field to use as a filter on the data included in your Hyperscale Vector index, click btn:[+ Add Filter Field].
+
+.. Under *Filter Fields*, select the field you want to use to filter your documents.
+
+.. Under *Include in Index*, choose whether the previously selected field should be included in your index.
++
+You can still choose to partition using this field and not include it in your index.
+
+.. Under *Partition by Hash*, choose whether the previously selected field should be used to partition your index, based on the values in your documents.
++
+For example, you could partition your Hyperscale Vector index based on the value of a `supplier` field in your documents.
+If you choose *No*, you can still filter using a `WHERE` clause, but your index will not be partitioned based on field values.
+
+.. (Optional) Under *Add WHERE clause*, enter a clause for the specific values that a document must contain in your filter field to be included in your index.
++
+For example, you could filter your index so that only documents with `supplier = "HP"` or `brightness > 0.5` are included.
+
+. Under *Index Configuration*, choose options to configure additional settings for your Hyperscale Vector index: 
+
+.. In the *Similarity Metric* list, select the similarity metric that you want to use for comparing vectors in your index.
+For the best accuracy, use the same similarity that you plan to use to query your data.
++
+For more information about vector similarity measures, see xref:vector-index:vectors-and-indexes-overview.adoc#vector_similarity[Vector Similarity].
+
+.. In the *Quantization* list, select the type of quantization to use for simplifying the vectors stored in your index.
++
+For more information about quantization, see xref:vector-index:vectors-and-indexes-overview.adoc#quantization[Quantization].
+For more information about how to choose quantization on your index, see xref:vector-index:vectors-and-indexes-overview.adoc#choosing-a-quantization-method[Choosing a Quantization Method] and xref:vector-index:vector-index-best-practices.adoc#quantization[Tuning Index Creation - Quantization].
+
+.. [[dimensions]] In the *Dimensions* field, enter the exact dimension of the vectors in your data.
++
+Your embedding model determines this value, and Capella automatically configures it based on your chosen *Vector Field*.
+
+.. In the *Replicas* list, select the number of replicas you want to create for your index.
++
+Replicas affect the query throughput, memory footprint, and fault tolerance for your Hyperscale Vector index.
+More replicas increase the required memory for your index, but can increase query throughput.
+.. [[training-list]](Optional) In the *Training List* field, enter or select the number of vectors to consider when searching for centroids in your data.
++
+[TIP]
+====
+If you have less than 10,000 vectors in your index, Couchbase sets the *Training List* value to your number of vectors.
+If you have more than 10,000 vectors up to a maximum of 1,000,000, Couchbase sets the *Training List* value to either 10% of your total number of vectors or 10 times your number of centroids - whichever value is higher, up to a maximum of 1,000,000.
+
+You can change your *Training List* value from these recommended defaults.
+Increasing the value can improve recall, but increase build times.
+Decreasing the value can improve build times, but reduces recall.
+====
+.. [[probes]](Optional) In the *Probes to scan* field, enter or select the number of centroids to check for similar vectors for each query.
++
+The default value is *1*. 
+A higher value increases search times, but also increases accuracy.
+You can also choose to override this value in your queries.
+
+.. (Optional) In the *Number of centroids* field, enter or select the number of centroids to create in your index.
++
+A larger value creates smaller centroids.
+By default, Couchbase sets the *Number of centroids* to the number of vectors in your data divided by 1000.
+For more information about how to set this value, see xref:vector-index:hyperscale-vector-index.adoc#number-of-centroids[Number of Centroids].
+
+.. To store full vectors in the index, select *Persist Full Vector*.
++
+This increases the storage required for your index but increases accuracy.
+
++
+. Click btn:[Review Index].
+. Confirm your Hyperscale Vector Index configuration. 
+. Click btn:[Create Index] or copy the {sqlpp} syntax to create your index with the Query Workbench.
+
+=== Resolve Index Creation Errors 
+
+If an error occurs while Capella tries to create your Hyperscale Vector index, on the *Vector Indexes* page, the *Status* of your index changes to *Error*. 
+
+To view the details of the error: 
+
+. Click the *Error* status to open the Query Workbench.
+. Click btn:[Run] to rebuild the index.
+. In the query results, find the error details. 
+
+Depending on the error, you might need to try recreating your index with a new configuration.