Follow on to corpus stats cache

README.md

@@ -215,6 +215,8 @@ SELECT * FROM pgedge_vectorizer.hybrid_search_simple(
 | `pgedge_vectorizer.enable_hybrid` | bool | `false` | — | Enable BM25 sparse vectors alongside dense embeddings |
 | `pgedge_vectorizer.bm25_k1` | real | `1.2` | `0.0–3.0` | Term frequency saturation parameter |
 | `pgedge_vectorizer.bm25_b` | real | `0.75` | `0.0–1.0` | Document length normalization parameter |
+| `pgedge_vectorizer.corpus_stats_cache_ttl` | integer | `60s` | `0–3600` | Seconds a backend may reuse a chunk table's corpus size and mean document length before reading them again. Set to 0 to read them on every call |
+| `pgedge_vectorizer.corpus_stats_cache_max_uses_pct` | integer | `5` | `0–100` | Percent of the cached corpus size that may be spent as uses of those figures before they are re-read, whichever comes first with the TTL. Each use counts as one chunk that may have been added since: a proxy rather than a measurement, so searches spend it too. Set to 0 to bound by time alone |
 
 ## Configuration Parameters
 

docs/changelog.md

@@ -56,6 +56,33 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
   vectorized tables were created under a build predating those triggers.
 - `pgedge_vectorizer.vectorizers` now records the document identifier column and
   its type, so cleanup triggers can be recreated without re-detecting it.
+- BM25 now caches a chunk table's corpus size and mean document length per
+  backend rather than deriving them on every call
+  ([#53](https://github.com/pgEdge/pgedge-vectorizer/issues/53)). Both come
+  from an aggregate that cannot use an index, so it was a full scan of the
+  chunk table for every search and for every queue item the worker processed;
+  a batch of ten scanned the same table ten times under a snapshot in which
+  the answer could not have changed. Two settings bound how stale a cached
+  reading may become, and whichever is reached first triggers a re-read:
+    - `pgedge_vectorizer.corpus_stats_cache_ttl` (default 60 seconds) bounds it
+      in time. Setting it to 0 disables the cache entirely, restoring the
+      previous behaviour exactly.
+    - `pgedge_vectorizer.corpus_stats_cache_max_uses_pct` (default 5) bounds it
+      in proportion to the corpus, which is how staleness actually harms
+      ranking: a thousand chunks added to a million barely move the weights,
+      while the same thousand added to two hundred change them several fold.
+      Each use of a cached reading counts as one chunk that may have been added
+      since — a proxy rather than a measurement, so searches spend the budget
+      too. Setting it to 0 bounds by time alone.
+
+  The figures feed a ranking heuristic rather than an account that has to
+  balance, and during ingest they are a moving target in any case, so holding
+  one briefly costs a little precision in a number that was never precise. They
+  are not maintained incrementally: chunk rows are inserted and deleted from
+  many places across the SQL — chunking, content updates, `recreate_chunks()`,
+  row deletion, truncation and disabling — and counters left wrong in any of
+  them would skew ranking silently and permanently, whereas a cache that
+  expires cannot drift for longer than its bounds and recovers by itself.
 
 ### Changed
 

docs/index.md

@@ -174,6 +174,7 @@ different use cases.
 | `pgedge_vectorizer.bm25_k1` | `1.2` | `0.0-3.0` | BM25 term-frequency saturation |
 | `pgedge_vectorizer.bm25_b` | `0.75` | `0.0-1.0` | BM25 document-length normalization |
 | `pgedge_vectorizer.corpus_stats_cache_ttl` | `60s` | `0-3600` | Seconds a backend reuses a chunk table's corpus size and mean document length before reading them again. Set to 0 to read them on every call |
+| `pgedge_vectorizer.corpus_stats_cache_max_uses_pct` | `5` | `0-100` | Percent of the cached corpus size that may be spent as uses of those figures before they are re-read, applied alongside the TTL so that whichever is reached first wins. Bounds staleness in proportion rather than in time, which matters most while a corpus is small and growing. Each use counts as one chunk that may have been added since: a proxy rather than a measurement, since the figures are read once per queue item and once per search, so a read-only workload spends the budget too. Set to 0 to bound by time alone |
 
 For more information or to download Vectorizer visit:
 

src/bm25.c

@@ -241,10 +241,110 @@ typedef struct
 	int64		total_docs;
 	float8		avg_doc_len;
 	TimestampTz	read_at;
+	int64		uses_since_read;
 } CorpusStatsEntry;
 
 static HTAB *corpus_stats_cache = NULL;
 
+/*
+ * corpus_stats_uses_spent — has this entry been leaned on long enough?
+ *
+ * A wall-clock TTL bounds staleness in time, but the harm is proportional:
+ * a thousand chunks added to a million barely move the weights, while the
+ * same thousand added to two hundred change them several fold.  Worse, N is
+ * cached while doc_freq is read fresh, so a corpus that outgrows a stale N
+ * gives ln((N+1)/(df+0.5)) < 0 for a common term, and a negative score is
+ * dropped from the stored vector altogether.
+ *
+ * Each use of a cached entry is counted as one chunk that may have been
+ * added since, and the entry is re-read once that count reaches the given
+ * percentage of N.  That is a proxy, not a measurement: the worker does not
+ * insert chunk rows itself, and a search does not grow the corpus at all.
+ * It is deliberately pessimistic in the direction that matters, and it
+ * scales itself — a million-row corpus tolerates fifty thousand uses, a
+ * two-hundred-row one ten, and re-reading a two-hundred-row table is free.
+ *
+ * The call that performs the read is not itself counted, so one read covers
+ * budget + 1 calls.  Counting it would mean no cached use at all wherever the
+ * budget works out to one, which is every corpus below 20 rows at the default
+ * 5%, and the small corpus is the case this bound exists to keep honest.
+ */
+static bool
+corpus_stats_uses_spent(const CorpusStatsEntry *entry)
+{
+	int64		budget;
+
+	if (pgedge_vectorizer_corpus_stats_cache_max_uses_pct <= 0)
+		return false;			/* use bound disabled; TTL only */
+
+	budget = entry->total_docs
+		* pgedge_vectorizer_corpus_stats_cache_max_uses_pct / 100;
+
+	return entry->uses_since_read >= Max(budget, 1);
+}
+
+static int64 bm25_corpus_stats(const char *chunk_table, float8 *avg_doc_len);
+
+/*
+ * bm25_corpus_stats_recheck — the data contradicts the cached corpus size.
+ *
+ * A term cannot appear in more documents than exist, so doc_freq above N is
+ * proof that the cached N is stale: doc_freq is read fresh on every call while
+ * N may be up to a TTL old.  Clamping the weight alone would leave the stale
+ * entry in place, so every later chunk in the same window would be scored, and
+ * have persisted into sparse_embedding, against the same wrong corpus.  Drop
+ * the entry and read again, which repairs this document and every one after it.
+ *
+ * If the fresh reading is still below max_df then the statistics are
+ * inconsistent rather than stale — a doc_freq left too high by an earlier
+ * failure, say — and re-reading cannot fix that.  Adopt max_df as this entry's
+ * corpus size in that case: the weights stay sane, and the next call sees a
+ * cached N that the data no longer contradicts, so it does not rescan to no
+ * effect.  A natural expiry re-reads the true N and the check runs again.
+ *
+ * The reading passed in may already have been fresh, in which case the read
+ * below is redundant.  That is left alone deliberately: it costs one extra
+ * scan, only for a table whose statistics are inconsistent, and only until
+ * max_df is adopted below, after which nothing contradicts the entry and the
+ * recheck stops firing.  Reporting cache hits back to the caller to avoid it
+ * would thread a flag through for a case that resolves itself.
+ *
+ * Caller must have an active SPI connection.
+ */
+static int64
+bm25_corpus_stats_recheck(const char *chunk_table, float8 *avg_doc_len,
+						  int64 max_df)
+{
+	Oid			relid;
+	int64		total_docs;
+
+	/* Nothing is cached to be stale, so the reading was already fresh. */
+	if (pgedge_vectorizer_corpus_stats_cache_ttl <= 0 ||
+		corpus_stats_cache == NULL)
+		return max_df;
+
+	relid = RelnameGetRelid(chunk_table);
+	if (!OidIsValid(relid))
+		return max_df;
+
+	hash_search(corpus_stats_cache, &relid, HASH_REMOVE, NULL);
+
+	total_docs = bm25_corpus_stats(chunk_table, avg_doc_len);
+
+	if (max_df > total_docs)
+	{
+		CorpusStatsEntry *entry;
+
+		entry = hash_search(corpus_stats_cache, &relid, HASH_FIND, NULL);
+		if (entry != NULL)
+			entry->total_docs = max_df;
+
+		total_docs = max_df;
+	}
+
+	return total_docs;
+}
+
 /*
  * bm25_corpus_stats_uncached — read N and the mean document length.
  *
@@ -355,11 +455,14 @@ bm25_corpus_stats(const char *chunk_table, float8 *avg_doc_len)
 	now = GetCurrentTimestamp();
 	entry = hash_search(corpus_stats_cache, &relid, HASH_FIND, NULL);
 
+	/* Both bounds apply: whichever is reached first forces a re-read. */
 	if (entry != NULL &&
 		!TimestampDifferenceExceeds(entry->read_at, now,
 									pgedge_vectorizer_corpus_stats_cache_ttl
-									* 1000))
+									* 1000) &&
+		!corpus_stats_uses_spent(entry))
 	{
+		entry->uses_since_read++;
 		*avg_doc_len = entry->avg_doc_len;
 		return entry->total_docs;
 	}
@@ -377,6 +480,7 @@ bm25_corpus_stats(const char *chunk_table, float8 *avg_doc_len)
 	entry->total_docs = total_docs;
 	entry->avg_doc_len = *avg_doc_len;
 	entry->read_at = now;
+	entry->uses_since_read = 0;
 
 	return total_docs;
 }
@@ -426,6 +530,9 @@ bm25_load_idf_stats(const char *chunk_table, BM25Term *tokens, int ntokens,
 	HTAB            *htab = NULL;
 	HASHCTL          ctl;
 	int64            total_docs = 1;
+	int64            max_df = 0;
+	IdfStat         *rows = NULL;
+	int              nrows = 0;
 	Datum            terms;
 	Oid              argtype = TEXTARRAYOID;
 
@@ -544,9 +651,39 @@ bm25_load_idf_stats(const char *chunk_table, BM25Term *tokens, int ntokens,
 					   &ctl,
 					   HASH_ELEM | HASH_STRINGS | HASH_CONTEXT);
 
-	for (int i = 0; i < (int) SPI_processed; i++)
+	/*
+	 * Take the rows out of SPI_tuptable before anything else runs a query.
+	 * The recheck below reads the corpus again, and SPI_execute() replaces
+	 * SPI_tuptable, so a loop still reading from it would decode that result
+	 * as (term, doc_freq).  read_idf_row() already copies the term, so the
+	 * array outlives the tuptable.
+	 *
+	 * The largest doc_freq is wanted before any weight is computed.  A term
+	 * cannot appear in more documents than exist, so a doc_freq above the
+	 * corpus size means the corpus size is wrong: it may be a cached reading
+	 * while every doc_freq here was read fresh.  Taking one maximum also
+	 * means one corpus size covers the whole document -- weighting each term
+	 * against its own would leave two terms of one chunk scored against
+	 * different corpora, which BM25 does not contemplate.
+	 */
+	nrows = (int) SPI_processed;
+	rows  = (IdfStat *) palloc(nrows * sizeof(IdfStat));
+
+	for (int i = 0; i < nrows; i++)
 	{
-		IdfStat       row   = read_idf_row(SPI_tuptable, i);
+		rows[i] = read_idf_row(SPI_tuptable, i);
+
+		if ((int64) rows[i].doc_freq > max_df)
+			max_df = (int64) rows[i].doc_freq;
+	}
+
+	if (max_df > total_docs)
+		total_docs = bm25_corpus_stats_recheck(chunk_table, avg_doc_len,
+											   max_df);
+
+	for (int i = 0; i < nrows; i++)
+	{
+		IdfStat       row   = rows[i];
 		bool          found;
 		IdfHashEntry *entry;
 
@@ -567,6 +704,8 @@ bm25_load_idf_stats(const char *chunk_table, BM25Term *tokens, int ntokens,
 		/* Duplicate terms: keep the first weight encountered */
 	}
 
+	pfree(rows);
+
 	return htab;
 }
 

src/guc.c

@@ -52,6 +52,7 @@ bool   pgedge_vectorizer_enable_hybrid = false;
 double pgedge_vectorizer_bm25_k1       = 1.2;
 double pgedge_vectorizer_bm25_b        = 0.75;
 int    pgedge_vectorizer_corpus_stats_cache_ttl = 60;
+int    pgedge_vectorizer_corpus_stats_cache_max_uses_pct = 5;
 
 /*
  * Initialize all GUC variables
@@ -330,5 +331,27 @@ pgedge_vectorizer_init_guc(void)
 		GUC_UNIT_S,
 		NULL, NULL, NULL);
 
+	DefineCustomIntVariable(
+		"pgedge_vectorizer.corpus_stats_cache_max_uses_pct",
+		"Percent of the corpus size that may be spent as uses of the cached "
+		"figures before they are re-read",
+		"Bounds staleness in proportion rather than in time, which is how it "
+		"harms ranking: a thousand chunks added to a million barely move the "
+		"weights, the same thousand added to two hundred change them several "
+		"fold. Each use of a cached reading counts as one chunk that may have "
+		"been added since. That is a proxy rather than a measurement of the "
+		"corpus, since the figures are read once per queue item and once per "
+		"search, so a read-only workload spends the budget too and a static "
+		"corpus is re-read periodically for no gain. Applied alongside "
+		"corpus_stats_cache_ttl, whichever is reached first. Set to 0 to "
+		"bound by time alone.",
+		&pgedge_vectorizer_corpus_stats_cache_max_uses_pct,
+		5,      /* default */
+		0,      /* min: 0 disables the use bound */
+		100,    /* max */
+		PGC_USERSET,
+		0,
+		NULL, NULL, NULL);
+
 	elog(DEBUG1, "pgedge_vectorizer GUC variables initialized");
 }

src/pgedge_vectorizer.h

@@ -73,6 +73,7 @@ extern bool   pgedge_vectorizer_enable_hybrid;
 extern double pgedge_vectorizer_bm25_k1;
 extern double pgedge_vectorizer_bm25_b;
 extern int    pgedge_vectorizer_corpus_stats_cache_ttl;
+extern int    pgedge_vectorizer_corpus_stats_cache_max_uses_pct;
 
 /*
  * Chunking strategy enumeration