+ We are publishing the retrieval results behind Delphi, the + corpora they were measured on, and the per-query traces they came + from. On {BENCHMARK.name}, the shipped build returns more of the + answer set than the leading hosted context engine and does it{" "} + {HEAD_TO_HEAD.latencyRatio.toFixed(1)} times faster on a laptop. + It does not lead on every metric, and this piece is specific + about which. +
+ +Results
+ ++ Delphi is an open-source context engine: it indexes + repositories, documentation, papers, and datasets, and answers an + agent's question with the files that answer it. The number + that matters for an agent is whether the file it needs is in the + window it gets, which is what these measure. +
+ ++ Recall@20 is the difference that survives:{" "} + {RECALL20.diff.toFixed(3)} in Delphi's favour, 95% interval{" "} + [{RECALL20.lo.toFixed(3)}, {RECALL20.hi.toFixed(3)}], winning{" "} + {RECALL20.wins} cases to {RECALL20.losses}. Latency is the other: + {" "}{(HEAD_TO_HEAD.delphi.latencyMs / 1000).toFixed(1)}s against{" "} + {(HEAD_TO_HEAD.nia.latencyMs / 1000).toFixed(1)}s. +
+ ++ MRR reads {HEAD_TO_HEAD.delphi.mrr.toFixed(3)} against{" "} + {HEAD_TO_HEAD.nia.mrr.toFixed(3)}, which looks like a loss and is + not one we can claim: the interval runs from{" "} + {MRR_ROW.lo.toFixed(3)} to {MRR_ROW.hi.toFixed(3)} and the cases + split {MRR_ROW.wins} to {MRR_ROW.losses} with {MRR_ROW.ties}{" "} + ties. At sixty cases that is a coin flip. We say so rather than + reporting it either way.1 +
+ ++ The two engines also answer differently. The comparator returns + about {HEAD_TO_HEAD.nia.meanPaths.toFixed(0)} files per query and + Delphi returns {HEAD_TO_HEAD.delphi.meanPaths.toFixed(0)}. Short + lists concentrate on rank one, long lists cover more of the + answer. Which you want depends on whether your agent gets one + shot or can keep reading. +
+ ++ Against the benchmark's own published baselines on the + development split, scored by the same code: +
+ +
+ Delphi does not lead Recall@20 outright either. Plain grep gets{" "}
+ {GREP.recall20.toFixed(3)}. We would rather print that than drop
+ the column. If your engine finds the right file somewhere in
+ twenty results, you have not beaten grep -r. The
+ argument has to be won at the top of the list, and that is where
+ we still have work to do.
+
A corpus orthogonal to itself
+ ++ The symptom was that semantic queries returned nonsense. We asked + a 68-repository corpus about TLS configuration in a gRPC proxy and + got back an interval tree, a systemd journal wrapper, and some + file-locking utilities. Exact symbol lookups worked perfectly. + Anything that went through the embedding did not. +
+ ++ Two embedding models can produce vectors of the same width. When + they do, pgvector will compute a cosine between them without + complaining. The query succeeds, results come back ranked, and the + ranking is noise, because the two spaces have nothing to do with + each other. There is no error to catch. The only thing wrong is + the answer. +
+ +The check that found it
+ ++ The check is embarrassingly simple. Take a chunk out of the index. + Embed its own content with whatever model answers queries today. + Compare that vector against the one already stored. If the two + sides agree, a chunk has to be nearly identical to itself. +
+ +
+ It scored {EMBEDDING_MISMATCH.cosineBefore.toFixed(4)}. That is
+ orthogonal, the number you get from two random vectors.
+ Pointing the query path at{" "}
+ {EMBEDDING_MISMATCH.indexedWith}, the model that had
+ actually built the index, moved the same comparison to{" "}
+ {EMBEDDING_MISMATCH.cosineAfter.toFixed(3)}. The vector branch was
+ weighted 0.5, the largest weight in the pipeline. Half the ranking
+ signal had been random for the entire evaluation.
+
+ The database had known all along.{" "}
+ repositories.embedding_model records which model
+ indexed each repository; it simply was never compared against the
+ model answering queries. Delphi now makes that comparison on every
+ search and reports it on /backend-health, because an
+ engine that returns confident nonsense is worse than one that
+ returns an error.
+
Comparing incomparable numbers
+ ++ With the embedding fixed, a second problem surfaced. Delphi fans a + query out across vector, BM25, symbol, path, and trigram branches, + then fuses the results. Each branch normalised its own scores by + dividing by that branch's top score. So the best hit of every + branch got pinned to exactly 1.0, however bad it was. +
+ +
+ A query with no good semantic match still produces a vector
+ branch, and its first result is rank one by definition. Rescaled,
+ that least-bad hit became a perfect 1.0, and at weight 0.5 it
+ outranked chunks that three branches independently agreed on. The
+ fingerprint was a suspiciously round 0.5000 at the
+ top of result lists: weight times a manufactured perfect score.
+
+ Reciprocal rank fusion exists for this reason. Ranks compare
+ across branches. Raw scores do not. A cosine and a{" "}
+ ts_rank_cd were never the same unit. Fusion now
+ scores position instead of magnitude.
+
The path nobody indexed
+ +
+ The third problem was the most mundane. BM25 indexes chunk
+ content. Nothing indexed file_path. A query naming a
+ file could not retrieve that file's neighbours lexically at
+ all. Searching for etcd_grpcproxy_test returned{" "}
+ fileutil.go, because the filename existed nowhere in
+ the searchable text.
+
+ This matters because agents anchor on paths constantly: “what
+ tests cover grpc_proxy.go”, “why did{" "}
+ tokens.py change”. Delphi now has a path-affinity
+ branch that matches on separator-stripped lowercase, so an anchor
+ of grpc_proxy reaches{" "}
+ etcd_grpcproxy_test.go, which underscore-sensitive
+ comparison misses. Results are capped per
+ directory, because a stem like grpc_proxy matches
+ thirty sibling files at a perfect score and would otherwise fill
+ the branch before the one test in tests/e2e/ ever
+ appeared.
+
The reranker that never ran
+ +
+ Delphi had a cross-encoder reranker. It had never once executed
+ during the evaluation. The model loaded lazily on first query, the
+ load was a multi-hundred-megabyte download, and the call site
+ wrapped it in a try/except that fell back silently to
+ fused ranking. Every query took the fallback.
+
+ Warming it at startup and reporting readiness on the health + endpoint fixed the availability problem and produced a genuinely + surprising result once we could measure it. +
+ +
+ The 22M-parameter ms-marco-MiniLM model beats
+ 278M-parameter bge-reranker-base on every metric while
+ running about seven times faster. Preferring the larger,
+ code-aware model, which is what the default did, cost latency and
+ quality at the same time. Reranking depth behaves the same way:
+ at a window of 100 with the cross-encoder deciding the order
+ outright, Recall@20 collapses to 0.444, because the model
+ confidently promotes plausible-looking files from the tail.
+ Blending it over the fused score, shallowly, is what makes it
+ useful.
+
What each fix was worth
+ ++ The first row is what the previous evaluation actually measured. + Almost the entire improvement comes from the embedding fix; rank + fusion and the path branch add a little on top; the cross-encoder + buys the top of the list. It would be more flattering to present + this as four clever retrieval improvements. It was one + configuration bug and three modest engineering fixes, and the + honest version is more useful to anyone running a similar stack. +
+ +What we can claim
+ ++ On {BENCHMARK.name}, {BENCHMARK.cases} development cases: Delphi + leads every published baseline on MRR and Recall@5, and trails + grep on Recall@20. On the held-out split, all {HELD_OUT.scored}{" "} + positive cases with every corpus provisioned and{" "} + {HELD_OUT.failures} failed queries, it scores{" "} + {HELD_OUT.mrr.toFixed(3)} MRR, {HELD_OUT.recall5.toFixed(3)}{" "} + Recall@5 and {HELD_OUT.recall20.toFixed(3)} Recall@20 at{" "} + {(HELD_OUT.latencyMsMean / 1000).toFixed(2)}s mean latency. Those + come out slightly ahead of the split the pipeline was tuned on, + which is the direction you want: no sign of having fit the + tuning set. +
+ ++ The per-workflow spread says more than the average does. + Retrieval is close to solved when the query names things that + exist in the code, and barely works when it does not. +
+ ++ A failure trace hands the retriever real symbols, real file + names, real stack frames, and Delphi finds the root-cause file in + the top five 76% of the time. A review comment hands it English, + something like “this should probably be extracted”, and + the same engine manages 17%. The gap between those two rows is not a + ranking problem. It is the difference between a query that + contains evidence and one that does not, and no amount of fusion + tuning closes it. +
+ ++ What does close some of it is giving the embedding something in + its own vocabulary to match. A code index is written in code; the + question is written in English. Asking a small model to draft the + code it thinks the answer looks like, and embedding that + alongside the question, moves every metric on the held-out split: +
+ ++ The snippet does not have to be right. It has to be written the + way the corpus is written, which is enough to land the query + vector in the right neighbourhood. Only the vector branch sees + it. BM25, symbol, and path still get the caller's literal + words, because inventing terms for an exact-match branch + manufactures precision that is not there. Queries already full of + identifiers are skipped entirely, and their scores are unchanged, + which is the shape you would expect if the mechanism works for + the reason claimed. +
+ ++ What we cannot claim is that Delphi is state of the art at the + top of the list. On this benchmark it is not: the hosted + comparator ranks better at MRR and Recall@5, and we publish that + alongside the metrics we do lead. The claim we can defend is + narrower. Delphi finds more of the answer set than the comparator, + beats every published baseline on early precision, and does it + much faster, on your own hardware. +
+ ++ The broader lesson is not about any one engine. A retrieval system + can be completely broken and still return ranked, plausible, + confident results, and every metric downstream of it will move + smoothly and mean nothing. If you run a vector index, embed a + chunk's own content and check it against its stored vector. + It takes one query and it is the cheapest assertion in the stack. +
+ +Open the evidence
+ ++ Every number above comes from queries that were recorded in full. + Below are real traces from the benchmarked build, two per + workflow, taken by position in the split rather than picked for + how they turned out. Expand one to see the exact query the engine + received, the ranked files it returned, which retrieval branch + found each one, and the raw response record. +
+ +
+ They are worth reading for the failures as much as the hits. Three
+ of the eight miss the gold file entirely. The first{" "}
+ code2test trace puts five changelog files above the
+ regression test it was asked for: the query mentions a version
+ bump, changelogs are dense with version strings, and BM25 has no
+ way to know that a changelog can never be an answer to “which test
+ covers this?”. The gold file appears at rank 9, found by the path
+ branch. That is a live weakness, not a rounding error.
+
What comes next
+ ++ The comparison above rests on {HEAD_TO_HEAD.cases} cases because + that is how many the hosted comparator completed. The run against + all {HELD_OUT.scored} failed every query on an HTTP error, so the + larger paired sample does not exist yet. Re-indexing those corpora + into the comparator and running it again is the single thing that + would settle MRR, and until it happens we are not going to + describe that metric as won or lost. +
+ ++ On our own side the open problem is rank one. Across the + held-out split the gold file is inside the top twenty + far more often than it is inside the top five, so the candidate + is usually retrieved and then not promoted. That is a judgement + problem in the reranking stage rather than a coverage problem in + retrieval, and widening the pool makes it worse rather than + better. Two directions we have not tried: fitting a reranker to + this task instead of using an off-the-shelf one, and routing by + query shape so that a question asking “which test covers + this” is ranked by a different rule than one asking where + something is implemented. +
+ ++ Every artifact behind these numbers is in the repository: the + corpus lock file, the per-query records, the summaries, and the + measured-and-rejected list of everything that did not work. +
+ +Footnotes
+-
+
- + 1. Intervals are a paired bootstrap over the per-case + difference, 4000 resamples, 95% percentile interval. We treat + a difference as real only when the interval excludes zero. An + earlier version of this page reported the comparison at 135 + cases with a {BENCHMARK.name} configuration that had query + expansion and listwise reranking disabled, which are the two + stages that order the head of the list. Those numbers + described a build we do not ship and have been replaced. + +
References
+-
+
-
+ 1.{" "}
+
+ Agent Retrieval Bench
+
+ , commit
{BENCHMARK.commit.slice(0, 12)}. +
+ - + 2. Cormack, Clarke, and Buettcher, “Reciprocal Rank Fusion + Outperforms Condorcet and Individual Rank Learning Methods”, + SIGIR 2009. + +
- + 3. Synthetic Sciences,{" "} + + Delphi source and implementation documentation + + . + +