diff --git a/docs/decisions/0038-unrated-no-trend-no-model-narrative.md b/docs/decisions/0038-unrated-no-trend-no-model-narrative.md new file mode 100644 index 0000000..114bb24 --- /dev/null +++ b/docs/decisions/0038-unrated-no-trend-no-model-narrative.md @@ -0,0 +1,64 @@ +# ADR-0038 — Unrated content carries no trend either, and demoted categories get no model narrative + +- **Status:** Accepted (2026-07-15) +- **Deciders:** Mikko +- **Completes:** [ADR-0032](0032-unrated-nonsubstantive-categories.md) (demoted = + unrated: no severity/sentiment), [ADR-0033](0033-operational-alerts-moderation-view.md) + (non-substantive content in a collapsed moderation view) + +## Context + +ADR-0032 made demoted categories (retail's `rasismi`, `asiaton`) **unrated** — no +severity badge, no sentiment badge, dropped from those aggregates. But two +severity-derived signals still leaked onto the moderation view: + +1. **The standard report path** (`/report`, the management view and the persisted + offline snapshot) ran a full LLM **synthesis** over demoted groups too. The + synthesis data block feeds the model each item's severity (the distribution and + the per-excerpt `(critical)` tag), so the model could editorialize a rating back + into the moderation card's title/narrative — `"'' (korkea)"` — the exact + thing ADR-0032 removed from the badges. (The live-summary/desk per-group + narratives were already deterministic. The whole-window **Yhteenveto** — the + live-summary `Overall` — is a *separate* surface, kept rated-only by its + companion change, the ADR-0033 §3 amendment; **this** ADR covers the + per-category moderation themes and the trend.) +2. **The trend/direction** (`kasvava`/`paheneva`/…) was computed and displayed for + demoted themes on the desk, management, and snapshot cards. `paheneva` + (worsening) is **severity-derived** (it requires a rising average severity), so a + moderation card could show a severity judgment on hostile content. + +Both contradict ADR-0032's meaning: on non-substantive content **the category is +the signal** — no good/bad/how-severe/which-way read. + +## Decision + +**Unrated (demoted) content carries no trend, and no model-authored narrative.** + +- A demoted category is built as a **count-only moderation theme** (`ReportService. + BuildModerationTheme`), in **both** the standard and live-summary paths: direction + is the neutral `stable` key with an **empty** label, and the narrative is a + deterministic count line (`ReportText.ModerationNarrative`) — the model is never + run over a demoted category's **per-group narrative**, so it cannot editorialize a + severity into the moderation card. This also returns the LLM budget the demoted + synthesis used to spend. (The whole-window Yhteenveto is kept off demoted content + by its companion rated-only `Overall` change, ADR-0033 §3.) +- **The views suppress the trend for unrated themes**, symmetric with how they + already suppress severity/sentiment: desk, management (`index.html`), and the + offline snapshot (`SnapshotHtml`) render only the count on a moderation card. +- Deterministic, no model dependency. Recognition is untouched (ADR-0027): the item + stays classified, keeps its `⚑` tag, and is counted in "Moderoitava sisältö (N)". + +## Consequences + +- A moderation card now shows the category, its count, and its content behind a + click — no severity, no sentiment, **no trend**, no model prose. The minimal + moderation view ADR-0033 intended. +- The standard `/report` path makes **one fewer LLM call per demoted category**, and + can no longer produce an ungrounded/action-bearing narrative over hostile content. +- `stable` + empty label is a deliberate "no trend," not a measured "stable": demoted + volume changes are not surfaced as a trend, because a trend is an editorial read we + do not put on non-substantive content. If a demoted-volume signal is ever wanted + (e.g. a brigading spike), it would be a separate, explicitly volume-only indicator, + not the severity-capable `direction`. +- Scope: unrated now means **no severity, no sentiment, no trend, no model + narrative** — the full "the category is the signal" contract. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index c7d5a4d..be66059 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -48,3 +48,4 @@ these ADRs relocate that reasoning into durable records. | [0035](0035-categorization-discipline-muu-single-category-hints.md) | Categorization discipline: the catch-all is one category (retires 0026 emergent topics), tighter disambiguating hints, the desk correction loop as backstop | Accepted | | [0036](0036-deterministic-category-keyword-override.md) | Deterministic category-keyword override: a grocery-core lexicon forces product departments (produce → hevi) with cross-category exclusions, sibling to the alert lexicon but raising no alert | Accepted | | [0037](0037-category-keywords-service-premises.md) | Extend the category-keyword override to service/premises (kassa_palvelu, tilat_siisteys) declared last, so products always win and service is the fallback | Accepted | +| [0038](0038-unrated-no-trend-no-model-narrative.md) | Unrated content carries no trend either, and demoted categories get a deterministic count-only moderation theme (no model narrative) | Accepted | diff --git a/src/FeedbackIntelligence.Api/Analysis/ReportService.cs b/src/FeedbackIntelligence.Api/Analysis/ReportService.cs index 66e2ea1..f4079c9 100644 --- a/src/FeedbackIntelligence.Api/Analysis/ReportService.cs +++ b/src/FeedbackIntelligence.Api/Analysis/ReportService.cs @@ -212,6 +212,16 @@ private async Task GenerateCoreAsync( .ThenBy(g => g.Key, StringComparer.Ordinal)) { var groupItems = group.ToList(); + // Demoted (unrated) categories carry NO trend, severity, or model + // narrative (ADR-0032/0038): running synthesis over them feeds the + // model their severities and lets it editorialize a rating into the + // moderation card ("'' (korkea)"). A deterministic count-only + // moderation theme instead — and it saves the LLM budget. + if (demotedCats.Contains(group.Key)) + { + themes.Add(BuildModerationTheme(group.Key, groupItems)); + continue; + } var direction = ComputeDirection(groupItems, fromIso, toIso, opts.MinItemsForTrend, opts.TrendSignificanceZ); var directionLabel = ReportText.DirectionLabel(direction, lang); var narrative = await SynthesizeThemeAsync(group.Key, groupItems, directionLabel, state, ct); @@ -250,6 +260,15 @@ int DemotedRank(string category) .ThenBy(g => g.Key, StringComparer.Ordinal)) { var groupItems = group.ToList(); + // Demoted (unrated) categories carry no trend or severity signal + // (ADR-0032/0038) — a deterministic count-only moderation theme, so no + // view renders a 'paheneva' (worsening, severity-derived) trend on + // hostile content. + if (demoted.Contains(group.Key)) + { + themes.Add(BuildModerationTheme(group.Key, groupItems)); + continue; + } var direction = ComputeDirection(groupItems, fromIso, toIso, opts.MinItemsForTrend, opts.TrendSignificanceZ); var directionLabel = ReportText.DirectionLabel(direction, lang); themes.Add(BuildTheme(group.Key, FallbackTitle(group.Key, groupItems), @@ -423,6 +442,21 @@ private ReportTheme BuildTheme( groupItems.Count(i => i.NeedsReview), SentimentCounts(groupItems), Unrated: activeDomain.Descriptor.DemotedCategories.Contains(category)); + /// A demoted / non-substantive category (retail's rasismi, asiaton) as a + /// theme: count + content only. Unrated content carries NO severity, sentiment, OR + /// trend, and gets NO model narrative (ADR-0032/0038) — feeding synthesis its + /// severities would let the model editorialize a rating onto hostile content. The + /// direction is the neutral "stable" key with an EMPTY label, so no view renders a + /// trend on it; the narrative is a deterministic count line. Sources are still + /// embedded so the collapsed moderation view shows the evidence behind a click. + private ReportTheme BuildModerationTheme(string category, List groupItems) => + new(category, FallbackTitle(category, groupItems), + ReportText.ModerationNarrative(groupItems.Count, activeDomain.Descriptor.Language), + groupItems.Count, "stable", "", + groupItems.Select(i => i.Id).ToList(), false, BuildSources(groupItems), + groupItems.Count(i => i.NeedsReview), SentimentCounts(groupItems), + Unrated: true); + private async Task<(string Title, string Narrative)?> SynthesizeThemeAsync( string category, IReadOnlyList groupItems, string directionLabel, GenState state, CancellationToken ct) { diff --git a/src/FeedbackIntelligence.Api/Analysis/ReportText.cs b/src/FeedbackIntelligence.Api/Analysis/ReportText.cs index e221b33..757370d 100644 --- a/src/FeedbackIntelligence.Api/Analysis/ReportText.cs +++ b/src/FeedbackIntelligence.Api/Analysis/ReportText.cs @@ -40,6 +40,15 @@ public static string FallbackNarrative(int count, string topThemes, string direc public static string WholeWindowScope(string language) => language == "fi" ? "kaikki" : "all"; + /// The narrative for a demoted / non-substantive (moderation) theme: a + /// count line only — no trend, no severity, nothing to editorialize on unrated + /// content (ADR-0032/0038). Deterministic, so the model never authors prose over + /// hostile content. + public static string ModerationNarrative(int count, string language) => + language == "fi" + ? $"{count} moderoitavaa palautetta aikavälillä." + : $"{count} item(s) to moderate in the window."; + /// Generic reason shown for an LLM-screened safety alert when the /// nomination pass returns no specific one — the per-item yes/no screen has /// already decided it IS an alert, so the alert still shows. diff --git a/src/FeedbackIntelligence.Api/Analysis/SnapshotHtml.cs b/src/FeedbackIntelligence.Api/Analysis/SnapshotHtml.cs index f447364..1a29a60 100644 --- a/src/FeedbackIntelligence.Api/Analysis/SnapshotHtml.cs +++ b/src/FeedbackIntelligence.Api/Analysis/SnapshotHtml.cs @@ -81,7 +81,10 @@ private static void AppendThemeCard( StringBuilder sb, ReportTheme theme, ReportText.SnapshotLabels t, IReadOnlyDictionary labels) { sb.Append($"
{E(theme.Title)}"); - sb.Append($"
{E(theme.Category)} · {theme.Count} {E(t.ItemsWord)} · {E(t.TrendWord)}: {E(theme.DirectionLabel)}
"); + // Unrated (demoted) themes carry no trend either (ADR-0032/0038) — only the + // count, so the moderation card editorializes nothing on hostile content. + var trend = theme.Unrated ? "" : $" · {E(t.TrendWord)}: {E(theme.DirectionLabel)}"; + sb.Append($"
{E(theme.Category)} · {theme.Count} {E(t.ItemsWord)}{trend}
"); var themePills = SentPills(theme.SentimentCounts, labels); if (themePills.Length > 0) sb.Append($"
{themePills}
"); diff --git a/src/FeedbackIntelligence.Api/wwwroot/desk.html b/src/FeedbackIntelligence.Api/wwwroot/desk.html index f950730..8a8cadc 100644 --- a/src/FeedbackIntelligence.Api/wwwroot/desk.html +++ b/src/FeedbackIntelligence.Api/wwwroot/desk.html @@ -561,7 +561,8 @@

label: catLabel(theme.category), count: theme.count, flagged: theme.flaggedCount, - meta: theme.directionLabel, + // Unrated (demoted) themes carry no trend either (ADR-0032/0038) — only the count. + meta: theme.unrated ? "" : theme.directionLabel, items: theme.sources ?? [], sentimentCounts: theme.sentimentCounts, unrated: theme.unrated, diff --git a/src/FeedbackIntelligence.Api/wwwroot/index.html b/src/FeedbackIntelligence.Api/wwwroot/index.html index e8d7778..97d98e3 100644 --- a/src/FeedbackIntelligence.Api/wwwroot/index.html +++ b/src/FeedbackIntelligence.Api/wwwroot/index.html @@ -310,8 +310,12 @@

function themeCard(theme) { const card = node("div", "card", [ node("strong", null, [theme.title]), - node("div", "muted", [`${catLabel(theme.category)} · ${theme.count} ${L.items} · ${L.trend}: ${theme.directionLabel ?? theme.direction}` + - (theme.narrativeFromLlm ? "" : ` · ${L.autoSummary}`)]), + node("div", "muted", [`${catLabel(theme.category)} · ${theme.count} ${L.items}` + + // Unrated (demoted) themes carry no trend, and no auto-summary tag either + // (ADR-0032/0038) — a moderation card is just the count, consistent with the + // desk and the snapshot, which never tag it. + (theme.unrated ? "" : ` · ${L.trend}: ${theme.directionLabel ?? theme.direction}`) + + (theme.unrated || theme.narrativeFromLlm ? "" : ` · ${L.autoSummary}`)]), node("p", null, [theme.narrative]), ]); // Per-theme sentiment mix (ADR-0030). diff --git a/tests/FeedbackIntelligence.Api.Tests/ReportServiceTests.cs b/tests/FeedbackIntelligence.Api.Tests/ReportServiceTests.cs index e540c35..7014bba 100644 --- a/tests/FeedbackIntelligence.Api.Tests/ReportServiceTests.cs +++ b/tests/FeedbackIntelligence.Api.Tests/ReportServiceTests.cs @@ -622,6 +622,42 @@ public async Task Alerts_AreOperationalOnly_RacismAlertsSuppressed_SafetyKept() Assert.Contains("mixed-1", alertIds); // also racist, but has a safety hazard → kept } + [Fact] + public async Task StandardMode_DemotedCategory_IsCountOnlyModerationTheme_NoModelNarrative_NoTrend() + { + // ADR-0032/0038: the standard path used to run a full LLM synthesis over demoted + // (rasismi/asiaton) groups, feeding the model their severities and letting it + // editorialize a rating into the moderation card, and demoted cards showed a + // (severity-derived) trend. Now a demoted group is a deterministic count-only + // moderation theme: no model call, no trend, no severity signal. + await _store.InsertAsync(ItemIn("maito-1", "2026-06-19T10:00:00.0000000+00:00", "maito_kylma", "tuoreus", "high"), CancellationToken.None); + // Two demoted items rated "critical" — a later, more-severe half would compute + // "worsening" if a trend were run over them. + await _store.InsertAsync(ItemIn("asia-1", "2026-06-20T10:00:00.0000000+00:00", "asiaton", "loukkaus", "critical"), CancellationToken.None); + await _store.InsertAsync(ItemIn("asia-2", "2026-06-29T10:00:00.0000000+00:00", "asiaton", "loukkaus", "critical"), CancellationToken.None); + + var llm = new CountingScriptedChatClient( + """{"title": "Maito", "narrative": "Tuoreusongelmia.", "citedIds": ["maito-1"]}"""); + var report = await CreateService(llm).GenerateAsync(WindowFrom, WindowTo, CancellationToken.None); + + var asiaton = report.Themes.Single(t => t.Category == "asiaton"); + Assert.True(asiaton.Unrated); + Assert.False(asiaton.NarrativeFromLlm); // no model narrative on demoted content + Assert.Equal("stable", asiaton.Direction); // no trend signal ... + Assert.Equal("", asiaton.DirectionLabel); // ... and nothing for a view to render + Assert.DoesNotContain("kriittinen", asiaton.Narrative); + Assert.DoesNotContain("critical", asiaton.Narrative); + + // Exactly ONE synthesis call — the rated maito group. The demoted group never + // reached the model, so its "critical" severity could not be editorialized. + Assert.Equal(1, llm.Calls); + + // The rated maito theme still gets its model narrative and a real (rendered) trend. + var maito = report.Themes.Single(t => t.Category == "maito_kylma"); + Assert.True(maito.NarrativeFromLlm); + Assert.NotEqual("", maito.DirectionLabel); + } + private sealed class ScriptedChatClient(params string[] responses) : IChatClient { private int _next; diff --git a/tools/FeedbackIntelligence.Ctl/Commands.cs b/tools/FeedbackIntelligence.Ctl/Commands.cs index efb69dc..980c8da 100644 --- a/tools/FeedbackIntelligence.Ctl/Commands.cs +++ b/tools/FeedbackIntelligence.Ctl/Commands.cs @@ -498,8 +498,14 @@ public static async Task ReportAsync(int days) Console.WriteLine($" {Term.C("▲ ALERT", "31")} [{a.GetProperty("feedbackId").GetString()}] {reason}"); } foreach (var t in rep.GetProperty("themes").EnumerateArray().Take(6)) + { + // Unrated (demoted) themes carry no trend (ADR-0032/0038) — print the count + // only, like the management views, not a "stable" token on moderation content. + var unrated = t.TryGetProperty("unrated", out var u) && u.ValueKind == JsonValueKind.True; + var trend = unrated ? "" : $", {t.GetProperty("direction").GetString()}"; Console.WriteLine($" {Term.C("●", "32")} {t.GetProperty("category").GetString(),-22} " + - $"({t.GetProperty("count").GetInt32()}, {t.GetProperty("direction").GetString()}) {t.GetProperty("title").GetString()}"); + $"({t.GetProperty("count").GetInt32()}{trend}) {t.GetProperty("title").GetString()}"); + } Console.WriteLine(); return 0; }