Skip to content

rotate: add an optional random rotation feature in INT8 quantization method#483

Open
zzlin237 wants to merge 44 commits into
alibaba:mainfrom
zzlin237:rotate-int8
Open

rotate: add an optional random rotation feature in INT8 quantization method#483
zzlin237 wants to merge 44 commits into
alibaba:mainfrom
zzlin237:rotate-int8

Conversation

@zzlin237

@zzlin237 zzlin237 commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator

New Feature: Added an optional rotation parameter to INT8 quantization, now supporting HNSW, Flat, IVF, and Vamana indexes.
Background: The quantization error in INT8 is directly proportional to the data range (max - min). By applying a random rotation, we can distribute the variance evenly across all vector dimensions. This helps reduce the overall range in certain cases, thereby improving INT8 quantization accuracy.
Benefits: Recall is improved while maintaining the same QPS. Using the cohere-1m dataset as an example:

  1. HNSW: QPS [14,431], Recall [0.9285] -> QPS [14,452], Recall [0.9397] (m=15, ef_construction=180, ef=180)
  2. Flat: Recall [0.9695] -> Recall [0.9881]
  3. Vamana: Recall [0.9576] -> Recall [0.9734]

Known Issues: We encountered issues during IVF testing and discovered pre-existing bugs in the IVF index:

  1. The nprobe parameter at the Python layer is currently ineffective; the index defaults to a fixed scan_ratio=0.1.
  2. When combining IVF with INT8, most points are assigned to a very small number of clusters (e.g., 70% of the data falls into a single cluster), which significantly degrades overall performance.

@CLAassistant

CLAassistant commented Jun 9, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@feihongxu0824

Copy link
Copy Markdown
Collaborator
  1. pr title需要能够说明这个pr的作用和目的;2. description里面需要注明一些背景、问题、收益等信息

Comment thread src/core/quantizer/integer_quantizer_reformer.cc Outdated
Comment thread src/db/proto/zvec.proto
@zzlin237 zzlin237 changed the title Rotate int8 An optional random rotation feature in INT8 quantization method Jun 15, 2026
@zzlin237 zzlin237 changed the title An optional random rotation feature in INT8 quantization method rotate: add an optional random rotation feature in INT8 quantization method Jun 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants