Sentence Transformers v6.0 - Multi Vector Encoders by tomaarsen · Pull Request #3501 · huggingface/blog

tomaarsen · 2026-08-12T09:46:21Z

Congratulations! You've made it this far! Once merged, the article will appear at https://huggingface.co/blog. Official articles
require additional reviews. Alternatively, you can write a community article following the process here.

Preparing the Article

You're not quite done yet, though. Please make sure to follow this process (as documented here):

Add an entry to _blog.yml.
Add a thumbnail. There are no requirements here, but there is a template if it's helpful.
Check you use a short title and blog path.
Upload any additional assets (such as images) to the Documentation Images repo. This is to reduce bloat in the GitHub base repo when cloning and pulling. Try to have small images to avoid a slow or expensive user experience.
Add metadata (such as authors) to your md file. You can also specify guest or org for the authors.
Ensure the publication date is correct.
Preview the content. A quick way is to paste the markdown content in https://huggingface.co/new-blog. Do not click publish, this is just a way to do an early check.

Here is an example of a complete PR: #2382

Getting a Review

cc @NohTow @raphaelsty Please review! I'm not 100% sure on the release date yet, but it'll be nice to get the reviews in whenever you can 🤗

Tom Aarsen

raphaelsty

Hi @tomaarsen, I love the blog post, it's very well written and I'm sure it will be soon the bible of anyone wanting to experiments with multi-vector models.

I added some comments from my first read, I'll do a second one, feel free to just ignore them.

I would add two things to the BP, I think it would be interesting to add a table with some scores for the models to help people choose the right model at the end of the page maybe. I know it might not age well so feel free to ignore this feedback as well, but I felt that it was a nice feature for me at the time when discovering sentence-transformers library within the documentation.

A second point to me is to make clear that vision multi-vector models can be used with multi-vector index weither it's weaviate qdrant or fast-plaid. I think it's quite important as it might not be crystal clear for some readers who are new to the field

I'll need to re-read it again to add more comments but honestly it's a pleasure to read

tomaarsen · 2026-08-13T10:08:59Z

Thanks for the reviews @raphaelsty! I've tackled all except for a scoring table. I think that would be a great addition, also for the sbert.net docs. Most of these don't have MTEB scores, so I think I'll run each model on NanoBEIR despite that benchmark having its own issues.

NohTow · 2026-08-15T10:02:37Z

+
+A multi-vector model (also called a late-interaction or ColBERT-style model, after the [ColBERT paper](https://arxiv.org/abs/2004.12832)) skips that compression. It runs the same transformer, but instead of pooling the token embeddings into one vector, it projects each token embedding down to a small dimension (classically 128) and keeps all of them. A 9-token document becomes a 9x128 matrix, not a 1x128 vector.
+
+The interaction between query and document is then deferred until scoring time, which is where the name "late interaction" comes from. A cross-encoder interacts early (both texts go through the model together, which is accurate but too slow to precompute). A bi-encoder barely interacts at all (one dot product between two finished summaries). Late interaction sits in between: documents are still encoded independently and can be indexed offline, but scoring compares every query token against every document token.


but too slow to precompute

but we cannot precompute anything as the user query is required for encoding, making it too slow to apply at scale

A bi-encoder barely interacts at all (one dot product between two finished summaries)
, but allows to precompute all the documents representations, enabling the indexation of large collections and fast querying.

but scoring compares every query token against every document token.

, enabling more interaction and thus increasing the capacity.

NohTow

The amount of work is insane, not only on the blogpost which is insanely thorough and easy to read (while introducing people to LI and its trade-off), but the amount of thoughts in the design of the integration is wild. You included most (all?) of the design I needed to push into PyLate but never got time/motivation to do (fix similarity function, add explainability, ...).
Congratulations on the work, Tom, really!

I added a few comments, in the intro about details and also because I think maybe pushing LateOn rather than GTE is good because its a net gain (same backbone, better perf).

On the things we might want to add (it's already dense, so honestly nothing mandatory), a (very) small paragraph in the early BP about PyLate to introduce it to people unfamiliar with it. It's not to get traction onto the lib (that's the whole point of merging into ST), but rather to give a bit of background for totally newcomers that reads "yeah it's PyLate compatible" but don't know what it is. Something along "ST supported dense models but not LI, while LI is just a few differences away, so LightOn built PyLate, a ST "fork" to add those capabilities and allows to train and experiments with these models. It allowed the community to build a few strong models and they also created an ecosystem around it, notably the indexes through fast-plaid. Now that it maturated, it was time to merge these capabilities back to ST." No worries if you think it's too much, I just think not all ST people just know what PyLate is in the first place.

Finally, one additional thing that could be cool would be to add LateOn/mLateOn boilerplates into the mix (maybe as external links). Again, it's not really for publicity (although this time a bit 😇 ), but rather because I think it's a very nice ressource for practitioners, because the data is strong and it also highlights a few "tricks" (e.g, split_batches/temperature) required to train strong models. Pre-training scripts are somewhat more messy and hard to read, but I think fine-tuning scripts are a really good base.

Again, all of this is suggestions, I think you already did an amazing job at handling both the release and the BP, so feel free to merge whatever you think is cool!

NohTow · 2026-08-15T10:03:37Z

+
+A multi-vector model (also called a late-interaction or ColBERT-style model, after the [ColBERT paper](https://arxiv.org/abs/2004.12832)) skips that compression. It runs the same transformer, but instead of pooling the token embeddings into one vector, it projects each token embedding down to a small dimension (classically 128) and keeps all of them. A 9-token document becomes a 9x128 matrix, not a 1x128 vector.
+
+The interaction between query and document is then deferred until scoring time, which is where the name "late interaction" comes from. A cross-encoder interacts early (both texts go through the model together, which is accurate but too slow to precompute). A bi-encoder barely interacts at all (one dot product between two finished summaries). Late interaction sits in between: documents are still encoded independently and can be indexed offline, but scoring compares every query token against every document token.


A bi-encoder barely interacts at all (one dot product between two finished summaries)
, but allows to precompute all the documents representations, enabling the indexation of large collections and fast querying.

NohTow · 2026-08-15T10:04:10Z

+
+A multi-vector model (also called a late-interaction or ColBERT-style model, after the [ColBERT paper](https://arxiv.org/abs/2004.12832)) skips that compression. It runs the same transformer, but instead of pooling the token embeddings into one vector, it projects each token embedding down to a small dimension (classically 128) and keeps all of them. A 9-token document becomes a 9x128 matrix, not a 1x128 vector.
+
+The interaction between query and document is then deferred until scoring time, which is where the name "late interaction" comes from. A cross-encoder interacts early (both texts go through the model together, which is accurate but too slow to precompute). A bi-encoder barely interacts at all (one dot product between two finished summaries). Late interaction sits in between: documents are still encoded independently and can be indexed offline, but scoring compares every query token against every document token.


but scoring compares every query token against every document token.

, enabling more interaction and thus increasing the capacity.

NohTow · 2026-08-15T10:04:49Z

+
+Because the token embeddings are L2-normalized, each of those dot products is a cosine similarity in `[-1, 1]`, so the whole sum lands within `[-num_query_tokens, num_query_tokens]`.
+
+You can read the operator as a soft alignment: every query token points at the one document token that best explains it, and the score is how well the document explains the query overall.


explains and supports the query overall.

NohTow · 2026-08-15T10:05:58Z

+
+You can read the operator as a soft alignment: every query token points at the one document token that best explains it, and the score is how well the document explains the query overall.
+
+The alignment doesn't have to be lexical, since the token embeddings are contextualized. Encode "Where do penguins live?" against "Penguins inhabit Antarctica." with [`lightonai/mLateOn`](https://huggingface.co/lightonai/mLateOn) and the query token `live` finds its best match on `inhabit` at 0.94, a word it shares no characters with! It isn't one-to-one either, since several query tokens routinely settle on the same document token. But when an exact match does matter to you (a product code, a surname, a function name), MaxSim has a token sitting right there to match it, where a single-vector model had to fold it into an average.


a word it shares no characters with! This thus overcomes the main limitation of traditional lexical approaches that requires an exact match and thus fail with synonyms or other languages.

NohTow · 2026-08-15T10:07:20Z

+
+### What You Gain, and What It Costs
+
+You gain retrieval quality, particularly on queries where one specific piece of a document is what makes it relevant, on multi-requirement queries like the sofa above where each requirement gets to find its own evidence, and on out-of-domain data where a dense model's compression was tuned for a different distribution.


Indeed, the dense model has to learn the compression and thus to select what could be useful in the document during the training. It thus learns to discard information that was not required by training queries but could be very useful for production queries. This gets worse as the document gets longer, as the compression becomes more dramatic.

NohTow · 2026-08-15T10:08:56Z

+pip install -U sentence-transformers
+```
+
+For ColPali-style visual document retrieval, you also need the image dependencies (see [Installation](https://sbert.net/docs/installation.html) for all extras):


Since sentence-transformers now also supports multimodality (link to the release), you can also use late interaction with, e.g, images (also known as ColPali). For ColPali-style...

NohTow · 2026-08-15T10:10:04Z

+```python
+from sentence_transformers import MultiVectorEncoder
+
+model = MultiVectorEncoder("lightonai/GTE-ModernColBERT-v1")


I think the default in any snippet should be LateOn
It's a net gain over GTE-ModernColBERT
(i would go as far as going for mLateOn, but it's a bit bigger. LateOn is the same size as GTE (as it's the same ModernBERT backbone))

NohTow · 2026-08-15T10:15:29Z

+
+## Token Pooling
+
+If the index footprint worries you, the most effective knob is to store fewer token vectors. `HierarchicalTokenPooling` implements the [token pooling](https://www.answer.ai/posts/colbert-pooling.html) technique from Answer.AI: it clusters each document's token vectors with Ward linkage on cosine similarity and replaces each cluster with its mean, keeping roughly `1 / pool_factor` of the tokens. Within one document a lot of token vectors end up close to each other, so much of what you drop is redundancy rather than signal:


technique from Answer.AI

I take no offense, but I've worked on the paper as well 😇

NohTow · 2026-08-15T10:17:26Z

+| gte-modernbert-base (dense, 768d) | 149M | 0.6446 | 0.3474 | 0.7237 | **0.6198** | 0.8203 | 0.6312 |
+| all-MiniLM-L6-v2 (dense, 384d) | 23M | 0.5540 | 0.3323 | 0.5904 | 0.4775 | 0.7265 | 0.5361 |
+
+Late interaction wins on four of the five subsets and on the mean! FiQA2018 (financial question answering) goes the other way, which is the shape of the tradeoff you should expect: a solid gain in retrieval quality for the same model size, paid for in index footprint, rather than a universal win on every dataset.


Here again I think integrating LateOn might be better to let the people know GTE-ModernColBERT is an old model and there are better alternatives nowadays

Sentence Transformers v6.0 - Multi Vector Encoders

4057d3b

tomaarsen marked this pull request as draft August 12, 2026 09:47

raphaelsty reviewed Aug 12, 2026

View reviewed changes

Comment thread multi-vector-encoder.md

Comment thread multi-vector-encoder.md Outdated

Comment thread multi-vector-encoder.md Outdated

Comment thread multi-vector-encoder.md

raphaelsty reviewed Aug 12, 2026

View reviewed changes

Comment thread multi-vector-encoder.md

raphaelsty reviewed Aug 12, 2026

View reviewed changes

Comment thread multi-vector-encoder.md Outdated

tomaarsen added 2 commits August 13, 2026 10:57

Add Audio and Video Retrieval sections, process review feedback

b82ed97

Add tentative publish date

b2f6777

NohTow reviewed Aug 15, 2026

View reviewed changes


		A multi-vector model (also called a late-interaction or ColBERT-style model, after the [ColBERT paper](https://arxiv.org/abs/2004.12832)) skips that compression. It runs the same transformer, but instead of pooling the token embeddings into one vector, it projects each token embedding down to a small dimension (classically 128) and keeps all of them. A 9-token document becomes a 9x128 matrix, not a 1x128 vector.

		The interaction between query and document is then deferred until scoring time, which is where the name "late interaction" comes from. A cross-encoder interacts early (both texts go through the model together, which is accurate but too slow to precompute). A bi-encoder barely interacts at all (one dot product between two finished summaries). Late interaction sits in between: documents are still encoded independently and can be indexed offline, but scoring compares every query token against every document token.


		Because the token embeddings are L2-normalized, each of those dot products is a cosine similarity in `[-1, 1]`, so the whole sum lands within `[-num_query_tokens, num_query_tokens]`.

		You can read the operator as a soft alignment: every query token points at the one document token that best explains it, and the score is how well the document explains the query overall.


		You can read the operator as a soft alignment: every query token points at the one document token that best explains it, and the score is how well the document explains the query overall.

		The alignment doesn't have to be lexical, since the token embeddings are contextualized. Encode "Where do penguins live?" against "Penguins inhabit Antarctica." with [`lightonai/mLateOn`](https://huggingface.co/lightonai/mLateOn) and the query token `live` finds its best match on `inhabit` at 0.94, a word it shares no characters with! It isn't one-to-one either, since several query tokens routinely settle on the same document token. But when an exact match does matter to you (a product code, a surname, a function name), MaxSim has a token sitting right there to match it, where a single-vector model had to fold it into an average.


		### What You Gain, and What It Costs

		You gain retrieval quality, particularly on queries where one specific piece of a document is what makes it relevant, on multi-requirement queries like the sofa above where each requirement gets to find its own evidence, and on out-of-domain data where a dense model's compression was tuned for a different distribution.


		## Token Pooling

		If the index footprint worries you, the most effective knob is to store fewer token vectors. `HierarchicalTokenPooling` implements the [token pooling](https://www.answer.ai/posts/colbert-pooling.html) technique from Answer.AI: it clusters each document's token vectors with Ward linkage on cosine similarity and replaces each cluster with its mean, keeping roughly `1 / pool_factor` of the tokens. Within one document a lot of token vectors end up close to each other, so much of what you drop is redundancy rather than signal:

Uh oh!

Conversation

tomaarsen commented Aug 12, 2026

Preparing the Article

Getting a Review

Uh oh!

raphaelsty left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tomaarsen commented Aug 13, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

NohTow left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants