Embeddings are non-deterministic: identical input returns different vectors, and processing a longer document permanently changes the embeddings of shorter ones

[vegah]

Embeddings are non-deterministic: identical input returns different vectors, and processing a longer document permanently changes the embeddings of shorter ones

Summary

On flm serve gemma3:4b --embed 1, the /v1/embeddings endpoint returns a different 768-dim vector for the same input depending on the request history. Over 60 identical back-to-back requests we observed 8 distinct vectors, with cosine similarity against the first response as low as 0.31.

Two separate effects are involved:

1. Sporadic corruption correlated with a slow path. Most requests complete in ~200 ms and return a consistent vector. A minority take 9–39 seconds and each returns a unique, badly different vector.
2. Persistent contamination by sequence length. After a longer document is embedded, the vectors returned for all shorter documents change and stay changed.

The same underlying instability is visible in chat: at temperature: 0 (which sampler.cpp implements as a hard argmax) long generations are not reproducible, while short ones are.

This makes the embedding endpoint unusable for retrieval, since the vector stored for a document depends on what was embedded before it.

Environment

FLM version	v0.9.46
Command	flm serve gemma3:4b --embed 1 (also reproduced with --pmode turbo)
Embedding model	embed-gemma:300m
Endpoint	POST /v1/embeddings
CPU / NPU	AMD Ryzen AI 9 HX 370 (Strix Point), XDNA2
NPU driver	32.0.20102.3930
OS	Windows 11 Pro 10.0.26200

Reproduction 1 — identical requests, different vectors

Send the same string 60 times, one input per request, and hash each returned vector.

Save as repro.mjs and run with node repro.mjs:

import { createHash } from 'node:crypto';

const URL = 'http://127.0.0.1:<port>/v1/embeddings';
const TEXT = 'The quick brown fox jumps over the lazy dog.';

const embed = async () => {
  const t0 = performance.now();
  const r = await fetch(URL, {
    method: 'POST', headers: { 'content-type': 'application/json' },
    body: JSON.stringify({ model: 'embed-gemma:300m', input: TEXT }),
  });
  return { vec: (await r.json()).data[0].embedding, ms: performance.now() - t0 };
};

for (let i = 1; i <= 60; i++) {
  const { vec, ms } = await embed();
  const h = createHash('sha1').update(vec.join(',')).digest('hex').slice(0, 12);
  console.log(i, ms.toFixed(0) + 'ms', h);
}

Expected: 60 identical vectors.
Actual: 8 distinct vectors on a freshly started server.

526e70051360  x53      <- dominant, ~185-220 ms
83c676aa67cf  x1       <- request #1 only (first call after model load)
ffd8d0d67f40  x1       <- request  #3, 9344 ms,  cos vs #1 = 0.679
218abf90db5a  x1       <- request #22, 27847 ms, cos vs #1 = 0.337
ed43d7a4de00  x1       <- request #23, 39420 ms, cos vs #1 = 0.395
b414a7643c98  x1       <- request #37, 214 ms,   cos vs #1 = 0.688
8a95a2609d0d  x1       <- request #49, 19279 ms, cos vs #1 = 0.313
ccafcd2df460  x1       <- request #51, 10255 ms, cos vs #1 = 0.705

latency p50 = 212 ms, max = 39420 ms

Five of the six corrupt responses coincide with a latency spike of 9–39 s, against a p50 of 212 ms. Whatever the server does on that slow path appears to also produce a wrong result. There are no NaN or Inf values, and the vector norm stays ~0.997 throughout, so the output looks superficially valid.

Reproduction 2 — a longer document permanently shifts shorter ones

Embed three short inputs, then introduce a longer one, then re-embed the short inputs.

A = "The quick brown fox jumps over the lazy dog."            (44 chars)
B = "Kubernetes autoscaling reduces cloud spend during off-peak hours."  (65 chars)
C = "x"                                                        (1 char)
D = <224 chars>
E = <700 chars>

sequence: A B C A B C | D | A B C A B C | E | A B C A B C

With --pmode turbo, where the slow path above does not occur, the result is clean enough to read directly — inputs B and C return exactly one vector per era, and a new era begins the moment a longer input is processed:

input B:  930cfcaffa36   baseline
          7287b5dac16b   after D
          0437f7826906   after E
input C:  a44c24a7ab8a   baseline
          bf593d66c773   after D
          53bc724cb711   after E

Embedding a single longer document permanently changes the embedding returned for every shorter document, and the new value is then stable until something longer is processed. On the default power mode the same run produces 5–6 distinct vectors per input, because effect (1) is superimposed.

This is consistent with buffer state (padding, or a residual region beyond the current sequence length) not being cleared between requests, so a shorter sequence sees residue from a longer predecessor. Inputs A, B and C do not contaminate each other, which would fit them falling into the same padded length bucket.

Reproduction 3 — chat is also affected at temperature: 0

AutoModel::set_temperature (src/common/AutoModel/automodel.cpp:614-620) rejects only negative values, so 0 passes through unclamped, and Sampler::sampler_temp_apply (src/common/modules/sampler.cpp:276-287) implements temp == 0 as a hard argmax by setting all but the top logit to -inf. Decoding is therefore expected to be exactly reproducible.

Short generations are:

prompt: "Count from 1 to 40, separated by commas. Output nothing else."
8/8 identical responses (39 chars, ~3.16 s each)

Long generations are not:

prompt: "Explain in detail how a CPU cache hierarchy works ... at least 300 words."
6/6 distinct responses, 3808-4227 chars, 49-54 s each
all diverge from the reference at character 58

Since the sampler is a deterministic argmax at temperature: 0, the differing token sequences mean the model is producing genuinely different logits across identical requests. The response lengths differ too, so this is not a display or serialization artifact.

Effect of --pmode turbo

Running with --pmode turbo did not fix correctness, but it did suppress the slow path. Over 50 requests we saw no 9–39 s spikes and only 2 distinct vectors instead of 8, and latency was stable at ~180–260 ms. The length-contamination effect (Reproduction 2) was unaffected.

We also could not reproduce the ~19 s per-embedding figure as a baseline cost — normal requests are ~200 ms on both power modes. The multi-second timings appear to be the corrupt path specifically.

Notes that may help localise it

• NPUAccessManager (src/server/server.cpp:133-149) serialises NPU access and the NPU Locked! / NPU Lock Released! log lines show no overlap, so this does not look like a plainly missing mutex at the server layer.
• bytes::sync_from_device() exists (src/include/buffer.hpp:310) and npu_utils.hpp:240-255 defines a safe_run() that syncs all BOs before and after a kernel run — but safe_run is not called anywhere in the open source tree, and the forward pass ships only as src/lib/gemma_embedding.dll, so we cannot tell from outside whether the output BO is synced or whether padding is cleared between runs. Gemma_Embedding::embed (src/common/AutoEmbeddingModel/modeling_gemma_embedding.cpp:30-35) reads y[i] directly with no sync on the caller side.
• The first request for a given input after a fresh start is reproducible across server restarts and across power modes — input A returned 83c676aa67cf as request Merge all source file in. #1 in independent runs. If that is the correct value, then the dominant steady-state value 526e70051360 (cos 0.759 against it) is wrong, and almost every embedding the server returns in normal use is wrong. We cannot determine which is correct from outside; src/test/gemma_embedding/test.cpp compares against a golden reference and would answer this immediately.

Two smaller API issues found while investigating

1. The model field is ignored. src/server/rest_handler.cpp:848 reads request["model"] and only echoes it back at :880; it never selects anything. Requesting "model": "this-model-does-not-exist-12345" returns HTTP 200 with a 768-dim vector. Validating it would make misconfiguration much easier to spot.
2. The embedding task type is hardcoded. src/server/rest_handler.cpp:866 always passes embedding_task_type_t::task_query. EmbeddingGemma is trained asymmetrically and modeling_gemma_embedding.hpp implements all ten prefixes, but none except query is reachable over REST, so documents can never be embedded with title: none | text: . For retrieval use this is a correctness problem independent of the bug above. An optional request field would solve it.

What we would expect

Repeated identical requests should return bit-identical vectors, and the vector for a document should not depend on which documents were embedded before it.