feat(retail): graded, density-gated vulgarity forces asiaton (ADR-0039)

docs/decisions/0039-graded-vulgarity-density-asiaton.md

@@ -0,0 +1,75 @@
+# ADR-0039 — Graded vulgarity: dense, non-substantive profanity is forced to `asiaton`
+
+- **Status:** Accepted (2026-07-15)
+- **Deciders:** Mikko
+- **Follows:** [ADR-0027](0027-racism-recognition-alert-lexicon.md) (a lexicon forces a
+  conduct category on a single hit — for ethnic slurs → `rasismi`),
+  [ADR-0032](0032-unrated-nonsubstantive-categories.md) (demoted = unrated),
+  [ADR-0036](0036-deterministic-category-keyword-override.md) (deterministic
+  category-keyword override architecture)
+
+## Context
+
+A vulgar pile-up — `Paskapillupersepornolehtipaviaani` — was stored `category=muu`
+(the rated catch-all), `severity=high`, `theme=offensive_language`. Poro *recognized*
+the offensiveness (the theme) but filed it under the rated catch-all, not `asiaton`,
+and rated nonsense `high`. Poro ignoring the `asiaton` affordance is the exact ADR-0032
+limitation ("the model ignores the affordance"); the demoted-category suppression
+(ADR-0032/0033/0038) can't help, because the item was never in a demoted category. The
+only deterministic conduct-forcing rule today is the rasismi ethnic-slur lexicon
+(ADR-0027) — this word is not ethnic, so nothing fired.
+
+The owner's requirement, and its crucial nuance: **a lone swear inside real feedback**
+(`Möivät paskaa` — "they sold crap", a crude but genuine quality complaint) **must stay
+rated**; a **pile-up of distinct vulgar stems mashed into nonsense** is not feedback and
+belongs in the moderation view. ADR-0032 gave up on *generic* garbage detection because
+Poro routed keyboard-mash unpredictably — but dense vulgarity has a **deterministic
+signal** (the stems themselves), so it is catchable where generic gibberish is not.
+
+## Decision
+
+**A deterministic, DENSITY-gated graded vulgarity scorer forces the demoted conduct
+category (`asiaton`).**
+
+1. **Domain lexicon** (`domains/<active>/vulgarity-lexicon.json`, optional like the
+   category-keyword lexicon): Finnish profanity **stems in tiers** (`mild`, `strong`),
+   plus the demote thresholds. Owner-authored (register is cultural), matched as
+   case-insensitive invariant substrings — the same contract as the alert lexicon.
+2. **Graded level 0–3** per message (`VulgarityScorer`):
+   - **0** none · **1** incidental (a mild stem) · **2** notable (a strong stem) —
+     Levels 1–2 stay **rated** (a Phase-2 `⚑` recognition tag will read them; see below).
+   - **3** dominant → **force `asiaton`** (demoted → unrated, count-only, moderation view).
+3. **Density gate for Level 3** — demote **iff** the vulgar-character share of the message
+   is `>= demoteRatio` **AND** there are `>= demoteMinDistinctStems` **distinct** stems.
+   Both conditions matter:
+   - *distinct-count* (not raw occurrences): repeating one swear (`paska`… `paska`) does
+     not inflate it — a lone stem stays rated (the owner's "singletons shouldn't flag").
+   - *ratio*: a furious-but-real complaint carrying three swears in a substantive sentence
+     is mostly real words (low ratio) → **stays rated**; a nonsense pile-up is mostly
+     vulgar stems (high ratio) → demoted. The two conditions together separate *angry real
+     feedback* from *vulgar nonsense*, which neither alone can.
+4. **Placement** — in `CategoryOverrideResolver`, **after** the rasismi alert override and
+   an already-demoted model choice, **before** the product category-keyword override:
+   racism (extreme, single-hit) → dense vulgarity → product hints → model. Ethnic slurs
+   stay in the alert lexicon (single hit → `rasismi`); this scorer never touches them.
+5. **Deterministic, config-driven, no model dependency.** Runs at ingest **and**
+   restructure, exactly like the other overrides; thresholds are validated config.
+
+## Consequences
+
+- `Paskapillupersepornolehtipaviaani` (`paska`+`pillu`+`perse` = 3 distinct, ~0.45 vulgar
+  share) → **`asiaton`**: unrated, no severity, count-only in "Moderoitava sisältö".
+  `Möivät paskaa.` (1 distinct, ~0.38 share — the *distinct* gate, not the ratio, is what
+  keeps it rated) → **stays rated** in its real category.
+- **Thresholds are empirical and MUST be measured on the owner's real examples**, not
+  guessed — the defaults here pass the two known cases and are config-tunable; they will
+  mis-fire on cases outside the fixture until tuned. This ADR does not claim the gate is
+  precise, only that it is deterministic and honest about its inputs. No invented Finnish
+  corpus is used for tuning (the eval cases are the owner's).
+- **Deferred to a follow-up (Phase 2):** the visible `⚑ kiroilu` **recognition tag** for
+  rated Level 1–2 items ("flag cursing without demoting"). It needs a structure/schema
+  field and view work, and is kept out of the operational alert channel (ADR-0033); it is
+  a separate, single-concern change. The graded *scorer* lands now; only its lower-tier
+  *presentation* is deferred.
+- A domain that ships no `vulgarity-lexicon.json` forces nothing (empty lexicon) — game
+  and any future domain are unaffected.

docs/decisions/README.md

@@ -49,3 +49,4 @@ these ADRs relocate that reasoning into durable records.
 | [0036](0036-deterministic-category-keyword-override.md) | Deterministic category-keyword override: a grocery-core lexicon forces product departments (produce → hevi) with cross-category exclusions, sibling to the alert lexicon but raising no alert | Accepted |
 | [0037](0037-category-keywords-service-premises.md) | Extend the category-keyword override to service/premises (kassa_palvelu, tilat_siisteys) declared last, so products always win and service is the fallback | Accepted |
 | [0038](0038-unrated-no-trend-no-model-narrative.md) | Unrated content carries no trend either, and demoted categories get a deterministic count-only moderation theme (no model narrative) | Accepted |
+| [0039](0039-graded-vulgarity-density-asiaton.md) | Graded vulgarity: a density-gated deterministic scorer forces dense, non-substantive profanity to `asiaton` (a lone swear in real feedback stays rated) | Accepted |

domains/retail/vulgarity-lexicon.json

@@ -0,0 +1,11 @@
+{
+  "matching": "case-insensitive invariant substring over the RAW feedback text; patterns are Finnish profanity STEMS (so inflected forms match). DENSITY-gated (ADR-0039): a message is demoted to 'asiaton' ONLY when the vulgar-character share is >= demoteRatio AND there are >= demoteMinDistinctStems DISTINCT stems — a lone swear in real feedback stays rated, a nonsense pile-up is demoted. Ethnic slurs are NOT here: they live in alert-keywords.json ('rasismi'), which forces the category on a SINGLE hit (ADR-0027). Owner-authored — register is cultural; edit freely.",
+  "tiers": {
+    "mild": ["paska", "perse", "hitto", "helvetti", "jumalauta"],
+    "strong": ["vittu", "kyrpä", "mulkku", "pillu", "runkkari"]
+  },
+  "demoteToCategory": "asiaton",
+  "demoteRatio": 0.4,
+  "demoteMinDistinctStems": 2,
+  "notes": "Thresholds are tuning knobs — measure against real desk examples before trusting them (ADR-0039). demoteRatio is the vulgar-character share of the message; demoteMinDistinctStems counts DISTINCT stems, not occurrences. Defaults chosen so 'Paskapillupersepornolehtipaviaani' (3 distinct, ~0.45 share) demotes while 'Möivät paskaa.' (1 distinct; the distinct gate keeps it rated) and a furious-but-substantive complaint (low share) stay rated."
+}

src/FeedbackIntelligence.Api/Ingest/IngestService.cs

@@ -21,6 +21,7 @@ public sealed class IngestService(
     LlmGate llmGate,
     AlertKeywordSet keywords,
     CategoryKeywordSet categoryKeywords,
+    VulgarityLexiconSet vulgarityLexicon,
     Core.Domain.IActiveDomain activeDomain,
     Analysis.ReportCache reportCache,
     ILogger<IngestService> logger)
@@ -62,7 +63,7 @@ public async Task<StoredFeedback> IngestAsync(FeedbackRequest request, Cancellat
         // structure: /interpret already previews the forced category, so a mismatch here
         // means it was edited away — the rule re-asserts it.
         var overrideCategory = CategoryOverrideResolver.Resolve(
-            alerts, request.Text, structure, activeDomain.Descriptor, categoryKeywords.Rules);
+            alerts, request.Text, structure, activeDomain.Descriptor, categoryKeywords.Rules, vulgarityLexicon.Lexicon);
         var corrections = request.Corrections;
         var forcedStructure = AlertMatcher.ApplyCategoryOverride(structure, overrideCategory);
         if (!ReferenceEquals(forcedStructure, structure))
@@ -173,7 +174,7 @@ public async Task<StoredFeedback> IngestAsync(FeedbackRequest request, Cancellat
                     // store update clears corrections like any restructure, since they
                     // audited a categorization that no longer stands.
                     var forced = AlertMatcher.ApplyCategoryOverride(item.Structure,
-                        CategoryOverrideResolver.Resolve(currentAlerts, item.Text, item.Structure, domain, categoryKeywords.Rules));
+                        CategoryOverrideResolver.Resolve(currentAlerts, item.Text, item.Structure, domain, categoryKeywords.Rules, vulgarityLexicon.Lexicon));
                     if (!ReferenceEquals(forced, item.Structure))
                     {
                         var forcedRows = await store.UpdateStructureAsync(
@@ -190,7 +191,7 @@ public async Task<StoredFeedback> IngestAsync(FeedbackRequest request, Cancellat
 
                 var (structure, itemFailed, notes) = await StructureViaGateAsync(item.Text, $"restructure of {item.Id}", ct);
                 structure = AlertMatcher.ApplyCategoryOverride(structure,
-                    CategoryOverrideResolver.Resolve(currentAlerts, item.Text, structure, domain, categoryKeywords.Rules));
+                    CategoryOverrideResolver.Resolve(currentAlerts, item.Text, structure, domain, categoryKeywords.Rules, vulgarityLexicon.Lexicon));
                 var reviewFlags = BuildReviewFlags(item.Text, structure);
                 var updatedRows = await store.UpdateStructureAsync(
                     item.Id, structure, itemFailed, notes, reviewFlags.Count > 0, reviewFlags, ct);

src/FeedbackIntelligence.Api/Program.cs

@@ -38,6 +38,15 @@
     var domain = sp.GetRequiredService<IActiveDomain>();
     return CategoryKeywordSet.LoadFrom(domain.CategoryKeywordsPath, domain.Descriptor.Categories);
 });
+// The graded vulgarity lexicon (ADR-0039) is domain data too — optional, and validated
+// against the active domain's declared + demoted categories so a typo'd or non-conduct
+// demoteToCategory fails the boot rather than re-rating real feedback at the first request.
+builder.Services.AddSingleton(sp =>
+{
+    var domain = sp.GetRequiredService<IActiveDomain>();
+    return VulgarityLexiconSet.LoadFrom(
+        domain.VulgarityLexiconPath, domain.Descriptor.Categories, domain.Descriptor.DemotedCategories);
+});
 builder.Services.AddSingleton<FeedbackStore>();
 builder.Services.AddSingleton<LlmGate>();
 builder.Services.AddSingleton<IngestService>();
@@ -64,6 +73,7 @@
     sp.GetRequiredService<LlmGate>(),
     sp.GetRequiredService<AlertKeywordSet>(),
     sp.GetRequiredService<CategoryKeywordSet>(),
+    sp.GetRequiredService<VulgarityLexiconSet>(),
     sp.GetRequiredService<IActiveDomain>(),
     sp.GetRequiredKeyedService<ReportCache>(Channels.Live),
     sp.GetRequiredService<ILogger<IngestService>>()));
@@ -117,6 +127,7 @@
 var reportOptions = app.Services.GetRequiredService<IOptions<ReportOptions>>().Value;
 _ = app.Services.GetRequiredService<AlertKeywordSet>();
 _ = app.Services.GetRequiredService<CategoryKeywordSet>(); // ADR-0036: a typo'd category-keywords.json fails the boot, not the first request
+_ = app.Services.GetRequiredService<VulgarityLexiconSet>(); // ADR-0039: a bad vulgarity-lexicon.json fails the boot too
 
 // The report resolves its prompts from the active domain at generation time.
 // Validate those roles (and their files) HERE so a domain that omits/misspells a
@@ -217,6 +228,7 @@
     IOptions<IngestOptions> options,
     AlertKeywordSet keywords,
     CategoryKeywordSet categoryKeywords,
+    VulgarityLexiconSet vulgarityLexicon,
     IActiveDomain domain,
     ILoggerFactory loggerFactory,
     CancellationToken ct) =>
@@ -233,7 +245,7 @@
         var alerts = AlertMatcher.Match(request.Text, keywords.Categories);
         var structure = AlertMatcher.ApplyCategoryOverride(
             result.Structure,
-            CategoryOverrideResolver.Resolve(alerts, request.Text, result.Structure, domain.Descriptor, categoryKeywords.Rules));
+            CategoryOverrideResolver.Resolve(alerts, request.Text, result.Structure, domain.Descriptor, categoryKeywords.Rules, vulgarityLexicon.Lexicon));
         return Results.Ok(new
         {
             structure,

src/FeedbackIntelligence.Api/Structuring/VulgarityLexiconSet.cs

@@ -0,0 +1,93 @@
+using System.Text.Json;
+using FeedbackIntelligence.Core.Structuring;
+
+namespace FeedbackIntelligence.Api.Structuring;
+
+/// <summary>The active domain's OPTIONAL graded vulgarity lexicon
+/// (domains/&lt;active&gt;/vulgarity-lexicon.json, ADR-0039): tiered Finnish profanity stems
+/// and the density thresholds that force the demoted conduct category (retail: asiaton).
+/// Optional by design — a domain without the file demotes nothing (empty lexicon), so the
+/// game domain and any future domain keep working. When the file IS present it is validated
+/// at startup: a missing/blank <c>demoteToCategory</c>, one that is not a declared DEMOTED
+/// category, or a tiers object with no stems fails the boot rather than silently disabling
+/// the layer.</summary>
+public sealed class VulgarityLexiconSet
+{
+    public required VulgarityLexicon Lexicon { get; init; }
+
+    public static readonly VulgarityLexiconSet Empty = new() { Lexicon = VulgarityLexicon.Empty };
+
+    public static VulgarityLexiconSet LoadFrom(
+        string path, IReadOnlySet<string> declaredCategories, IReadOnlyList<string> demotedCategories)
+    {
+        var resolved = FeedbackIntelligence.Llm.AppPathResolver.Resolve(path);
+        if (!File.Exists(resolved))
+            return Empty; // optional layer: no file → demote nothing
+
+        using var doc = JsonDocument.Parse(File.ReadAllText(resolved));
+        var root = doc.RootElement;
+
+        if (!root.TryGetProperty("demoteToCategory", out var demoteEl)
+            || demoteEl.ValueKind != JsonValueKind.String
+            || string.IsNullOrWhiteSpace(demoteEl.GetString()))
+            throw new InvalidOperationException($"'{resolved}' must set a non-empty 'demoteToCategory'.");
+        var demoteTo = demoteEl.GetString()!;
+        // Forcing an undeclared or non-demoted category would either write an invalid value
+        // or (worse) re-rate real feedback — fail the boot, like the category-keyword checks.
+        if (!declaredCategories.Contains(demoteTo))
+            throw new InvalidOperationException(
+                $"'demoteToCategory' \"{demoteTo}\" in '{resolved}' is not a declared category.");
+        if (!demotedCategories.Contains(demoteTo))
+            throw new InvalidOperationException(
+                $"'demoteToCategory' \"{demoteTo}\" in '{resolved}' is not a DEMOTED category — vulgarity may only "
+                + "force a conduct/moderation category (ADR-0039).");
+
+        var mild = ReadTier(root, "mild");
+        var strong = ReadTier(root, "strong");
+        if (mild.Count == 0 && strong.Count == 0)
+            throw new InvalidOperationException($"'{resolved}' has no vulgarity stems under 'tiers.mild' / 'tiers.strong'.");
+
+        var ratio = ReadDouble(root, "demoteRatio", VulgarityLexicon.Empty.DemoteRatio, resolved);
+        var minDistinct = ReadInt(root, "demoteMinDistinctStems", VulgarityLexicon.Empty.DemoteMinDistinctStems, resolved);
+        if (ratio is <= 0 or > 1)
+            throw new InvalidOperationException($"'demoteRatio' in '{resolved}' must be in (0, 1].");
+        if (minDistinct < 1)
+            throw new InvalidOperationException($"'demoteMinDistinctStems' in '{resolved}' must be >= 1.");
+
+        return new VulgarityLexiconSet
+        {
+            Lexicon = new VulgarityLexicon(mild, strong, demoteTo, ratio, minDistinct),
+        };
+    }
+
+    private static IReadOnlyList<string> ReadTier(JsonElement root, string tier) =>
+        root.TryGetProperty("tiers", out var tiers) && tiers.ValueKind == JsonValueKind.Object
+        && tiers.TryGetProperty(tier, out var arr) && arr.ValueKind == JsonValueKind.Array
+            ? arr.EnumerateArray()
+                .Where(e => e.ValueKind == JsonValueKind.String)
+                .Select(e => e.GetString()!)
+                .Where(s => !string.IsNullOrWhiteSpace(s))
+                .ToList()
+            : [];
+
+    // Absent → the documented default; present but not a number → fail the boot (a quoted
+    // "0.6" or a null was meant to tune the gate and must not be silently ignored, matching
+    // the strict demoteToCategory validation above).
+    private static double ReadDouble(JsonElement root, string prop, double fallback, string resolved)
+    {
+        if (!root.TryGetProperty(prop, out var e))
+            return fallback;
+        if (e.ValueKind != JsonValueKind.Number)
+            throw new InvalidOperationException($"'{prop}' in '{resolved}' must be a number.");
+        return e.GetDouble();
+    }
+
+    private static int ReadInt(JsonElement root, string prop, int fallback, string resolved)
+    {
+        if (!root.TryGetProperty(prop, out var e))
+            return fallback;
+        if (e.ValueKind != JsonValueKind.Number)
+            throw new InvalidOperationException($"'{prop}' in '{resolved}' must be a number.");
+        return e.GetInt32();
+    }
+}

src/FeedbackIntelligence.Core/Domain/ActiveDomain.cs

@@ -16,6 +16,12 @@
     /// (category-keywords.json, ADR-0036) — term lists that force a category. The file
     /// need not exist; a domain without it forces nothing.</summary>
     string CategoryKeywordsPath { get; }
+
+    /// <summary>Absolute path of the domain's OPTIONAL graded vulgarity lexicon
+    /// (vulgarity-lexicon.json, ADR-0039) — tiered profanity stems + density thresholds
+    /// that force the demoted conduct category. The file need not exist; a domain without
+    /// it demotes nothing.</summary>
+    string VulgarityLexiconPath { get; }
     string StoriesPath { get; }
 
     /// <summary>Domain-owned prompt files by role (e.g. "synthesis",
@@ -34,6 +40,7 @@
     public string Name => Descriptor.Name;
     public string AlertKeywordsPath { get; }
     public string CategoryKeywordsPath { get; }
+    public string VulgarityLexiconPath { get; }
     public string StoriesPath { get; }
     public IReadOnlyDictionary<string, string> PromptPaths { get; }
 
@@ -53,6 +60,7 @@
         Descriptor = BuildDescriptor(root, o.Active, domainJson);
         AlertKeywordsPath = Path.Combine(dir, "alert-keywords.json");
         CategoryKeywordsPath = Path.Combine(dir, "category-keywords.json");
+        VulgarityLexiconPath = Path.Combine(dir, "vulgarity-lexicon.json");
         StoriesPath = Path.Combine(dir, "stories.json");
         PromptPaths = BuildPromptPaths(root, dir);
     }