diff --git a/CHANGELOG.md b/CHANGELOG.md index f2dbeee..714b07d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Added + +- Turkish quality corpus fixtures under `tests/Fixtures/tr/` (60 cases: 30 clean, 10 profane, 10 obfuscated, 10 edge) +- `CorpusLoader` and `CorpusQualityReport` support classes for fixture validation and quality metrics +- `tests/TurkishCorpusQualityTest.php` with aggregate FP/FN/coverage gate +- `docs/dictionary-expansion-policy.md` — criteria for future Turkish dictionary entry PRs + +### Changed + +- `CONTRIBUTING.md` — corpus authoring rules and dictionary expansion policy link + ## [0.2.0] - 2026-07-03 Dictionary infrastructure release — author-only dictionary rows with build-time normalized key generation. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 71eb6a6..843fe5b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -60,6 +60,37 @@ ci: add PHP 8.3 to test matrix - False-positive regression tests are as important as true-positive tests - Prefer simple, readable code over abstractions +## Turkish Dictionary Quality Corpus + +VerbaGuard maintains a fixture-based Turkish quality corpus under `tests/Fixtures/tr/`. + +- **Clean corpus** — false-positive candidates (`expect.detected: false`) +- **Profane / obfuscated corpus** — detection cases (`expect.detected: true`) +- **Edge cases** — boundary behavior with explicit expectations + +Quality gate (CI): false positives = 0, false negatives = 0, detection coverage = 100%. + +### Adding corpus cases + +1. Use stable `id` values (`category.slug`) unique across all fixture files. +2. Required fields: `id`, `text`, `expect.detected`. +3. Set `expect.terms` to seed dictionary terms only when asserting canonical matches. +4. Empty `text` is allowed only in `edge_cases.php`. +5. Run `composer test` — `TurkishCorpusQualityTest` must pass. + +### Dictionary expansion PRs + +Before adding Turkish dictionary entries, read [docs/dictionary-expansion-policy.md](docs/dictionary-expansion-policy.md). + +New TR entry PRs **must** include: + +- Clean neighbor cases (false-positive documentation) +- Profane and obfuscated positive cases +- A risk assessment in the PR description +- Passing aggregate corpus gate (FP = 0) + +Do not add dictionary entries without corresponding corpus coverage. + ## Reporting Bugs Use the [bug report issue template](.github/ISSUE_TEMPLATE/bug_report.yml). Include: diff --git a/docs/dictionary-expansion-policy.md b/docs/dictionary-expansion-policy.md new file mode 100644 index 0000000..9192b15 --- /dev/null +++ b/docs/dictionary-expansion-policy.md @@ -0,0 +1,120 @@ +# Turkish Dictionary Expansion Policy + +This document defines the quality gate for adding or changing Turkish dictionary entries in VerbaGuard. + +For corpus format and test infrastructure, see [`tests/Fixtures/tr/README.md`](../tests/Fixtures/tr/README.md). + +--- + +## Principles + +1. **False positives are worse than false negatives** — see [`FOUNDATION.md`](../FOUNDATION.md). +2. **Corpus before expansion** — measurable quality ground must exist before growing the dictionary. +3. **Short terms need extra care** — tokens of three characters or fewer have higher false-positive risk in real text. + +--- + +## Pre-merge checklist for new TR entries + +Before merging a pull request that adds or changes Turkish dictionary rows: + +1. Add **clean neighbor cases** documenting false-positive risk. +2. Add **profane detection cases** for the new term (direct usage). +3. Add **obfuscated detection cases** where evasion is realistic (leetspeak, separators, Turkish folding). +4. Ensure the aggregate corpus gate passes: + - false positives = **0** + - false negatives = **0** + - detection coverage = **100%** (profane + obfuscated corpus) +5. Include a **risk assessment** in the PR description (template below). + +Do not merge dictionary expansion PRs that increase false positives without explicit project approval. + +--- + +## Clean neighbor cases (required) + +Every new entry must include clean corpus cases showing the term does **not** match innocuous text. + +| Term length | Minimum clean neighbor cases | +|-------------|------------------------------| +| ≤2 characters | 8 | +| 3 characters | 5 | +| ≥4 characters | 3 | + +Neighbor cases should cover: + +- Substring collisions (term embedded in longer innocent words) +- Similar spellings and Turkish character variants +- Realistic sentence context where the innocent word appears + +Add cases to [`tests/Fixtures/tr/clean.php`](../tests/Fixtures/tr/clean.php) with `tags` and optional `risk`. + +--- + +## Profane and obfuscated positive cases (required) + +For each new entry, add: + +- At least **3** cases to [`tests/Fixtures/tr/profane.php`](../tests/Fixtures/tr/profane.php) (direct token and sentence usage) +- At least **3** cases to [`tests/Fixtures/tr/obfuscated.php`](../tests/Fixtures/tr/obfuscated.php) (realistic evasion) + +Each positive case should set `expect.terms` to the canonical dictionary term. + +--- + +## Short-term evaluation (≤3 characters) + +Short dictionary terms (e.g. `aq`, `mal`, `amk`) require: + +- Explicit `risk` annotation on related clean cases +- Extra clean neighbor coverage per table above +- Documented trade-off in the PR risk assessment +- Reviewer acknowledgment that standalone-token matching is intentional + +Short-term false positives are a **dictionary design** concern, not a matcher defect. + +--- + +## False-positive gate + +The CI quality gate enforced by [`tests/TurkishCorpusQualityTest.php`](../tests/TurkishCorpusQualityTest.php) requires: + +``` +falsePositiveCount() === 0 +falseNegativeCount() === 0 +detectionCoverage() === 100.0 // profane + obfuscated corpus +``` + +Any PR that increases false positives must either: + +- Revise or reject the dictionary entry, or +- Add clean corpus cases proving the new behavior is acceptable (rare; requires explicit approval) + +--- + +## Risk assessment template + +Include this block in dictionary expansion PRs: + +```markdown +## Dictionary entry risk assessment + +- Term: `example` +- Length: 3 +- Category / severity: profanity / medium +- Clean neighbors added: clean.example.* (5 cases) +- Profane cases added: profane.example.* (3 cases) +- Obfuscated cases added: obfuscated.example.* (3 cases) +- Known FP risk: [none | documented in corpus notes] +- Short-term trade-off: [N/A | standalone token may match innocuous text in edge contexts] +``` + +--- + +## Related documents + +| Document | Purpose | +|----------|---------| +| [`FOUNDATION.md`](../FOUNDATION.md) | Correctness policy | +| [`docs/specification.md`](specification.md) | Dictionary author format | +| [`CONTRIBUTING.md`](../CONTRIBUTING.md) | Contribution workflow | diff --git a/tests/Fixtures/tr/README.md b/tests/Fixtures/tr/README.md new file mode 100644 index 0000000..70412eb --- /dev/null +++ b/tests/Fixtures/tr/README.md @@ -0,0 +1,58 @@ +# Turkish Quality Corpus + +Test-only fixture corpus for measuring Turkish dictionary quality against the seed dictionary in `data/tr.php`. + +## Files + +| File | Purpose | Expected detection | +|------|---------|-------------------| +| `clean.php` | Innocuous text; false-positive candidates | `false` | +| `profane.php` | Direct profanity and insult detection | `true` | +| `obfuscated.php` | Leetspeak, separators, folding, repeats | `true` | +| `edge_cases.php` | Boundary behavior (mixed expectations) | per case | + +## Case format + +```php +[ + 'id' => 'clean.malzeme.embedded', + 'text' => 'malzeme listesi hazır', + 'expect' => [ + 'detected' => false, + ], + 'tags' => ['substring', 'mal-risk'], + 'note' => 'mal must not match inside malzeme', +] +``` + +### Required fields + +- `id` — stable identifier (`category.slug`) +- `text` — UTF-8 input +- `expect.detected` — boolean + +### Optional fields + +- `expect.terms` — expected canonical dictionary terms (seed terms only) +- `expect.match_count` — expected number of matches +- `expect.severity` — expected highest severity +- `expect.categories` — expected match categories +- `tags` — filtering labels +- `note` — human-readable rationale +- `risk` — dictionary expansion risk hint (`low`, `medium`, `high`) + +## Rules + +- Empty `text` is allowed **only** in `edge_cases.php`. +- `expect.terms` must reference seed terms: `amk`, `aq`, `siktir`, `orospu`, `mal`. +- IDs must be unique across all fixture files. + +## Quality gate + +`TurkishCorpusQualityTest` enforces: + +- false positives = 0 +- false negatives = 0 +- detection coverage = 100% (profane + obfuscated corpus) + +See [docs/dictionary-expansion-policy.md](../../docs/dictionary-expansion-policy.md) before adding dictionary entries. diff --git a/tests/Fixtures/tr/clean.php b/tests/Fixtures/tr/clean.php new file mode 100644 index 0000000..8182ab5 --- /dev/null +++ b/tests/Fixtures/tr/clean.php @@ -0,0 +1,195 @@ + 'clean.normal.word', + 'text' => 'normal', + 'expect' => ['detected' => false], + 'tags' => ['substring', 'mal-risk'], + 'note' => 'normal must not match mal via substring', + ], + [ + 'id' => 'clean.animal.word', + 'text' => 'animal', + 'expect' => ['detected' => false], + 'tags' => ['substring', 'mal-risk'], + ], + [ + 'id' => 'clean.formal.word', + 'text' => 'formal', + 'expect' => ['detected' => false], + 'tags' => ['substring', 'mal-risk'], + ], + [ + 'id' => 'clean.small.word', + 'text' => 'small', + 'expect' => ['detected' => false], + 'tags' => ['substring', 'mal-risk'], + ], + [ + 'id' => 'clean.malzeme.embedded', + 'text' => 'malzeme', + 'expect' => ['detected' => false], + 'tags' => ['substring', 'mal-risk'], + 'note' => 'mal must not match inside malzeme', + ], + [ + 'id' => 'clean.malzeme.phrase', + 'text' => 'küçük malzeme', + 'expect' => ['detected' => false], + 'tags' => ['substring', 'mal-risk'], + ], + [ + 'id' => 'clean.kamk.neighbor', + 'text' => 'kamk', + 'expect' => ['detected' => false], + 'tags' => ['substring', 'amk-risk'], + 'risk' => 'high', + ], + [ + 'id' => 'clean.samk.neighbor', + 'text' => 'şamk', + 'expect' => ['detected' => false], + 'tags' => ['substring', 'amk-risk'], + 'risk' => 'high', + ], + [ + 'id' => 'clean.greeting.merhaba', + 'text' => 'Merhaba dünya', + 'expect' => ['detected' => false], + 'tags' => ['sentence'], + ], + [ + 'id' => 'clean.weather.sentence', + 'text' => 'Bugün hava güzel', + 'expect' => ['detected' => false], + 'tags' => ['sentence'], + ], + [ + 'id' => 'clean.meeting.sentence', + 'text' => 'Toplantı saat üçte başlayacak', + 'expect' => ['detected' => false], + 'tags' => ['sentence'], + ], + [ + 'id' => 'clean.delivery.sentence', + 'text' => 'Projeyi yarın teslim edeceğiz', + 'expect' => ['detected' => false], + 'tags' => ['sentence'], + ], + [ + 'id' => 'clean.anormal.word', + 'text' => 'Anormal değerler görüldü', + 'expect' => ['detected' => false], + 'tags' => ['substring', 'mal-risk'], + ], + [ + 'id' => 'clean.small.business', + 'text' => 'Small business model', + 'expect' => ['detected' => false], + 'tags' => ['substring', 'mal-risk'], + ], + [ + 'id' => 'clean.formal.outfit', + 'text' => 'Formal giysi giydi', + 'expect' => ['detected' => false], + 'tags' => ['substring', 'mal-risk'], + ], + [ + 'id' => 'clean.animal.documentary', + 'text' => 'Animal kingdom documentary', + 'expect' => ['detected' => false], + 'tags' => ['substring', 'mal-risk'], + ], + [ + 'id' => 'clean.trafo.sentence', + 'text' => 'Trafo merkezi bakımda', + 'expect' => ['detected' => false], + 'tags' => ['sentence'], + ], + [ + 'id' => 'clean.kapak.malzeme', + 'text' => 'Kapak malzemesi geldi', + 'expect' => ['detected' => false], + 'tags' => ['substring', 'mal-risk'], + ], + [ + 'id' => 'clean.portakal.sentence', + 'text' => 'Portakal suyu içtim', + 'expect' => ['detected' => false], + 'tags' => ['sentence'], + ], + [ + 'id' => 'clean.academic.sentence', + 'text' => 'Akademik çalışma tamamlandı', + 'expect' => ['detected' => false], + 'tags' => ['sentence'], + ], + [ + 'id' => 'clean.engineering.sentence', + 'text' => 'Mühendislik fakültesi', + 'expect' => ['detected' => false], + 'tags' => ['sentence'], + ], + [ + 'id' => 'clean.software.sentence', + 'text' => 'Yazılım güncellendi', + 'expect' => ['detected' => false], + 'tags' => ['sentence'], + ], + [ + 'id' => 'clean.database.sentence', + 'text' => 'Veritabanı yedeklendi', + 'expect' => ['detected' => false], + 'tags' => ['sentence'], + ], + [ + 'id' => 'clean.login.sentence', + 'text' => 'Kullanıcı giriş yaptı', + 'expect' => ['detected' => false], + 'tags' => ['sentence'], + ], + [ + 'id' => 'clean.order.sentence', + 'text' => 'Sipariş onaylandı', + 'expect' => ['detected' => false], + 'tags' => ['sentence'], + ], + [ + 'id' => 'clean.campaign.sentence', + 'text' => 'Kampanya başladı', + 'expect' => ['detected' => false], + 'tags' => ['sentence'], + ], + [ + 'id' => 'clean.map.sentence', + 'text' => 'Harita güncellendi', + 'expect' => ['detected' => false], + 'tags' => ['sentence'], + ], + [ + 'id' => 'clean.calendar.sentence', + 'text' => 'Takvim etkinliği eklendi', + 'expect' => ['detected' => false], + 'tags' => ['sentence'], + ], + [ + 'id' => 'clean.autoreply.sentence', + 'text' => 'Otomatik yanıt açık', + 'expect' => ['detected' => false], + 'tags' => ['sentence'], + ], + [ + 'id' => 'clean.programmatic.sentence', + 'text' => 'Programatik erişim kapalı', + 'expect' => ['detected' => false], + 'tags' => ['sentence'], + ], +]; diff --git a/tests/Fixtures/tr/edge_cases.php b/tests/Fixtures/tr/edge_cases.php new file mode 100644 index 0000000..b0f1145 --- /dev/null +++ b/tests/Fixtures/tr/edge_cases.php @@ -0,0 +1,85 @@ + 'edge.spelled-chain.normal.spaces', + 'text' => 'n o r m a l', + 'expect' => ['detected' => false], + 'tags' => ['separator-chain', 'false-positive-risk'], + 'note' => 'Full chain normalizes to normal, not mal', + ], + [ + 'id' => 'edge.spelled-chain.normal.dots', + 'text' => 'n.o.r.m.a.l', + 'expect' => ['detected' => false], + 'tags' => ['separator-chain', 'false-positive-risk'], + ], + [ + 'id' => 'edge.spelled-chain.aq-neighbor', + 'text' => 'x a q y', + 'expect' => ['detected' => false], + 'tags' => ['separator-chain', 'aq-risk'], + ], + [ + 'id' => 'edge.spelled-chain.mal.spaces', + 'text' => 'm a l', + 'expect' => [ + 'detected' => true, + 'terms' => ['mal'], + ], + 'tags' => ['separator-chain', 'short-term'], + ], + [ + 'id' => 'edge.empty.string', + 'text' => '', + 'expect' => ['detected' => false], + 'tags' => ['empty-input'], + ], + [ + 'id' => 'edge.whitespace.only', + 'text' => ' ', + 'expect' => ['detected' => false], + 'tags' => ['empty-input'], + ], + [ + 'id' => 'edge.malzeme.boundary', + 'text' => 'malzeme', + 'expect' => ['detected' => false], + 'tags' => ['substring', 'mal-risk'], + 'note' => 'Boundary check: mal inside malzeme must not match', + ], + [ + 'id' => 'edge.mal.standalone-in-sentence', + 'text' => 'bu mal bir şey', + 'expect' => [ + 'detected' => true, + 'terms' => ['mal'], + ], + 'tags' => ['sentence', 'short-term'], + ], + [ + 'id' => 'edge.siktir.repeated-letters', + 'text' => 'siiiktir', + 'expect' => [ + 'detected' => true, + 'terms' => ['siktir'], + ], + 'tags' => ['repeated-letters'], + ], + [ + 'id' => 'edge.siktir.turkish-variant', + 'text' => 'sıktır', + 'expect' => [ + 'detected' => true, + 'terms' => ['siktir'], + ], + 'tags' => ['turkish-chars'], + ], +]; diff --git a/tests/Fixtures/tr/obfuscated.php b/tests/Fixtures/tr/obfuscated.php new file mode 100644 index 0000000..502efab --- /dev/null +++ b/tests/Fixtures/tr/obfuscated.php @@ -0,0 +1,101 @@ + 'obfuscated.amk.leetspeak-4', + 'text' => '4mk', + 'expect' => [ + 'detected' => true, + 'terms' => ['amk'], + ], + 'tags' => ['leetspeak'], + ], + [ + 'id' => 'obfuscated.amk.leetspeak-at', + 'text' => '@mk', + 'expect' => [ + 'detected' => true, + 'terms' => ['amk'], + ], + 'tags' => ['leetspeak'], + ], + [ + 'id' => 'obfuscated.siktir.leetspeak', + 'text' => 's1kt1r', + 'expect' => [ + 'detected' => true, + 'terms' => ['siktir'], + ], + 'tags' => ['leetspeak'], + ], + [ + 'id' => 'obfuscated.siktir.dots', + 'text' => 's.i.k.t.i.r', + 'expect' => [ + 'detected' => true, + 'terms' => ['siktir'], + ], + 'tags' => ['separator-chain'], + ], + [ + 'id' => 'obfuscated.siktir.spaces', + 'text' => 's i k t i r', + 'expect' => [ + 'detected' => true, + 'terms' => ['siktir'], + ], + 'tags' => ['separator-chain'], + ], + [ + 'id' => 'obfuscated.amk.spaced', + 'text' => 'a m k', + 'expect' => [ + 'detected' => true, + 'terms' => ['amk'], + ], + 'tags' => ['separator-chain'], + ], + [ + 'id' => 'obfuscated.amk.partial-space', + 'text' => 'am k', + 'expect' => [ + 'detected' => true, + 'terms' => ['amk'], + ], + 'tags' => ['separator-chain'], + ], + [ + 'id' => 'obfuscated.aq.dots', + 'text' => 'a.q', + 'expect' => [ + 'detected' => true, + 'terms' => ['aq'], + ], + 'tags' => ['separator-chain'], + ], + [ + 'id' => 'obfuscated.siktir.turkish-fold', + 'text' => 'şiktir', + 'expect' => [ + 'detected' => true, + 'terms' => ['siktir'], + ], + 'tags' => ['turkish-chars'], + ], + [ + 'id' => 'obfuscated.amk.repeated-letters', + 'text' => 'ammmmK', + 'expect' => [ + 'detected' => true, + 'terms' => ['amk'], + ], + 'tags' => ['repeated-letters'], + ], +]; diff --git a/tests/Fixtures/tr/profane.php b/tests/Fixtures/tr/profane.php new file mode 100644 index 0000000..6f240c0 --- /dev/null +++ b/tests/Fixtures/tr/profane.php @@ -0,0 +1,102 @@ + 'profane.amk.direct', + 'text' => 'amk', + 'expect' => [ + 'detected' => true, + 'terms' => ['amk'], + ], + 'tags' => ['direct'], + ], + [ + 'id' => 'profane.aq.direct', + 'text' => 'aq', + 'expect' => [ + 'detected' => true, + 'terms' => ['aq'], + ], + 'tags' => ['direct'], + ], + [ + 'id' => 'profane.siktir.direct', + 'text' => 'siktir', + 'expect' => [ + 'detected' => true, + 'terms' => ['siktir'], + ], + 'tags' => ['direct'], + ], + [ + 'id' => 'profane.orospu.direct', + 'text' => 'orospu', + 'expect' => [ + 'detected' => true, + 'terms' => ['orospu'], + ], + 'tags' => ['direct'], + ], + [ + 'id' => 'profane.mal.standalone', + 'text' => 'mal', + 'expect' => [ + 'detected' => true, + 'terms' => ['mal'], + ], + 'tags' => ['direct', 'short-term'], + ], + [ + 'id' => 'profane.mal.in-sentence', + 'text' => 'bu mal bir şey', + 'expect' => [ + 'detected' => true, + 'terms' => ['mal'], + ], + 'tags' => ['sentence', 'short-term'], + ], + [ + 'id' => 'profane.amk.uppercase', + 'text' => 'AMK', + 'expect' => [ + 'detected' => true, + 'terms' => ['amk'], + ], + 'tags' => ['case-variant'], + ], + [ + 'id' => 'profane.siktir.sentence', + 'text' => 'git siktir', + 'expect' => [ + 'detected' => true, + 'terms' => ['siktir'], + ], + 'tags' => ['sentence'], + ], + [ + 'id' => 'profane.multi.amk-aq', + 'text' => 'amk ve aq', + 'expect' => [ + 'detected' => true, + 'terms' => ['amk', 'aq'], + 'match_count' => 2, + ], + 'tags' => ['multi-match'], + ], + [ + 'id' => 'profane.siktir.turkish-case', + 'text' => 'SİKTİR', + 'expect' => [ + 'detected' => true, + 'terms' => ['siktir'], + ], + 'tags' => ['case-variant', 'turkish-chars'], + ], +]; diff --git a/tests/Support/CorpusLoader.php b/tests/Support/CorpusLoader.php new file mode 100644 index 0000000..35e248c --- /dev/null +++ b/tests/Support/CorpusLoader.php @@ -0,0 +1,266 @@ +|null */ + private static ?array $seedTerms = null; + + /** @var list */ + private const CASE_FIELDS = ['id', 'text', 'expect', 'tags', 'note', 'risk']; + + /** @var list */ + private const EXPECT_FIELDS = ['detected', 'terms', 'match_count', 'severity', 'categories']; + + /** @var list */ + private const FIXTURE_FILES = [ + 'clean', + 'profane', + 'obfuscated', + 'edge_cases', + ]; + + /** + * @return list> + */ + public static function load(string $name): array + { + $path = self::fixturePath($name); + /** @var list> $rows */ + $rows = require $path; + + return self::validateCases($rows, $name); + } + + /** + * @return list> + */ + public static function all(): array + { + $cases = []; + $seenIds = []; + + foreach (self::FIXTURE_FILES as $name) { + foreach (self::load($name) as $case) { + $id = $case['id']; + + if (isset($seenIds[$id])) { + throw new InvalidArgumentException( + sprintf('Duplicate corpus id "%s" across fixtures.', $id), + ); + } + + $seenIds[$id] = true; + $cases[] = $case; + } + } + + return $cases; + } + + /** + * @return list> + */ + public static function loadClean(): array + { + return self::load('clean'); + } + + /** + * @return list> + */ + public static function loadProfane(): array + { + return self::load('profane'); + } + + /** + * @return list> + */ + public static function loadObfuscated(): array + { + return self::load('obfuscated'); + } + + /** + * @return list> + */ + public static function loadEdge(): array + { + return self::load('edge_cases'); + } + + private static function fixturePath(string $name): string + { + return dirname(__DIR__) . '/Fixtures/tr/' . $name . '.php'; + } + + /** + * @param list> $rows + * + * @return list> + */ + private static function validateCases(array $rows, string $fixtureName): array + { + $seenIds = []; + $allowEmptyText = $fixtureName === 'edge_cases'; + + foreach ($rows as $index => $case) { + if (! is_array($case)) { + throw new InvalidArgumentException( + sprintf('Corpus case at index %d in %s must be an array.', $index, $fixtureName), + ); + } + + self::assertKnownCaseFields($case, $fixtureName, $index); + self::assertRequiredFields($case, $fixtureName, $index); + self::assertTextRule($case, $fixtureName, $index, $allowEmptyText); + self::assertExpectShape($case, $fixtureName, $index); + + $id = $case['id']; + + if (! is_string($id) || $id === '') { + throw new InvalidArgumentException( + sprintf('Corpus case id must be a non-empty string at index %d in %s.', $index, $fixtureName), + ); + } + + if (isset($seenIds[$id])) { + throw new InvalidArgumentException( + sprintf('Duplicate corpus id "%s" in %s.', $id, $fixtureName), + ); + } + + $seenIds[$id] = true; + } + + return $rows; + } + + /** + * @param array $case + */ + private static function assertKnownCaseFields(array $case, string $fixtureName, int $index): void + { + foreach (array_keys($case) as $field) { + if (! in_array($field, self::CASE_FIELDS, true)) { + throw new InvalidArgumentException( + sprintf('Unknown corpus field "%s" at index %d in %s.', $field, $index, $fixtureName), + ); + } + } + } + + /** + * @param array $case + */ + private static function assertRequiredFields(array $case, string $fixtureName, int $index): void + { + foreach (['id', 'text', 'expect'] as $field) { + if (! array_key_exists($field, $case)) { + throw new InvalidArgumentException( + sprintf('Missing required corpus field "%s" at index %d in %s.', $field, $index, $fixtureName), + ); + } + } + } + + /** + * @param array $case + */ + private static function assertTextRule(array $case, string $fixtureName, int $index, bool $allowEmptyText): void + { + if (! is_string($case['text'])) { + throw new InvalidArgumentException( + sprintf('Corpus field "text" must be a string at index %d in %s.', $index, $fixtureName), + ); + } + + if ($case['text'] === '' && ! $allowEmptyText) { + throw new InvalidArgumentException( + sprintf('Corpus field "text" must not be empty at index %d in %s.', $index, $fixtureName), + ); + } + } + + /** + * @param array $case + */ + private static function assertExpectShape(array $case, string $fixtureName, int $index): void + { + if (! is_array($case['expect'])) { + throw new InvalidArgumentException( + sprintf('Corpus field "expect" must be an array at index %d in %s.', $index, $fixtureName), + ); + } + + foreach (array_keys($case['expect']) as $field) { + if (! in_array($field, self::EXPECT_FIELDS, true)) { + throw new InvalidArgumentException( + sprintf('Unknown expect field "%s" at index %d in %s.', $field, $index, $fixtureName), + ); + } + } + + if (! array_key_exists('detected', $case['expect'])) { + throw new InvalidArgumentException( + sprintf('Missing expect.detected at index %d in %s.', $index, $fixtureName), + ); + } + + if (! is_bool($case['expect']['detected'])) { + throw new InvalidArgumentException( + sprintf('expect.detected must be a boolean at index %d in %s.', $index, $fixtureName), + ); + } + + if (array_key_exists('terms', $case['expect'])) { + if (! is_array($case['expect']['terms'])) { + throw new InvalidArgumentException( + sprintf('expect.terms must be an array at index %d in %s.', $index, $fixtureName), + ); + } + + foreach ($case['expect']['terms'] as $term) { + if (! is_string($term) || ! in_array($term, self::seedTerms(), true)) { + throw new InvalidArgumentException( + sprintf('Invalid expect.terms entry at index %d in %s.', $index, $fixtureName), + ); + } + } + } + } + + /** + * @return list + */ + private static function seedTerms(): array + { + if (self::$seedTerms !== null) { + return self::$seedTerms; + } + + /** @var list $rows */ + $rows = require dirname(__DIR__, 2) . '/data/tr.php'; + $terms = []; + + foreach ($rows as $index => $row) { + if (! is_array($row) || ! array_key_exists('term', $row) || ! is_string($row['term']) || $row['term'] === '') { + throw new InvalidArgumentException( + sprintf('Invalid seed dictionary row at index %d in data/tr.php.', $index), + ); + } + + $terms[] = $row['term']; + } + + self::$seedTerms = $terms; + + return self::$seedTerms; + } +} diff --git a/tests/Support/CorpusQualityReport.php b/tests/Support/CorpusQualityReport.php new file mode 100644 index 0000000..808dabf --- /dev/null +++ b/tests/Support/CorpusQualityReport.php @@ -0,0 +1,112 @@ + */ + private array $falsePositiveCases = []; + + /** @var list */ + private array $falseNegativeCases = []; + + private int $positiveExpectationCount = 0; + + private int $positiveDetectionCount = 0; + + /** + * @param list> $cases + */ + public function __construct( + private readonly VerbaGuard $guard, + array $cases, + ) { + foreach ($cases as $case) { + $this->evaluateCase($case); + } + } + + /** + * @param list> $cases + */ + public static function fromCases(VerbaGuard $guard, array $cases): self + { + return new self($guard, $cases); + } + + public static function fromAllFixtures(VerbaGuard $guard): self + { + return new self($guard, CorpusLoader::all()); + } + + public static function fromDetectionCorpus(VerbaGuard $guard): self + { + return new self($guard, [ + ...CorpusLoader::loadProfane(), + ...CorpusLoader::loadObfuscated(), + ]); + } + + public function falsePositiveCount(): int + { + return count($this->falsePositiveCases); + } + + public function falseNegativeCount(): int + { + return count($this->falseNegativeCases); + } + + public function detectionCoverage(): float + { + if ($this->positiveExpectationCount === 0) { + return 100.0; + } + + return ($this->positiveDetectionCount / $this->positiveExpectationCount) * 100; + } + + /** + * @return list + */ + public function falsePositiveCases(): array + { + return $this->falsePositiveCases; + } + + /** + * @return list + */ + public function falseNegativeCases(): array + { + return $this->falseNegativeCases; + } + + /** + * @param array $case + */ + private function evaluateCase(array $case): void + { + /** @var array{detected: bool} $expect */ + $expect = $case['expect']; + $id = $case['id']; + $text = $case['text']; + $detected = $this->guard->contains($text); + + if ($expect['detected']) { + $this->positiveExpectationCount++; + + if ($detected) { + $this->positiveDetectionCount++; + } else { + $this->falseNegativeCases[] = $id; + } + } elseif ($detected) { + $this->falsePositiveCases[] = $id; + } + } +} diff --git a/tests/TurkishCorpusQualityTest.php b/tests/TurkishCorpusQualityTest.php new file mode 100644 index 0000000..f250eec --- /dev/null +++ b/tests/TurkishCorpusQualityTest.php @@ -0,0 +1,106 @@ + $case + */ +function assertCorpusExpectations(VerbaGuard $guard, array $case): void +{ + /** @var array{detected: bool, terms?: list, match_count?: int, severity?: string, categories?: list} $expect */ + $expect = $case['expect']; + $result = $guard->analyze($case['text']); + + expect($result->hasProfanity())->toBe($expect['detected']); + + if (! $expect['detected']) { + return; + } + + if (isset($expect['match_count'])) { + expect($result->matches())->toHaveCount($expect['match_count']); + } + + if (isset($expect['terms'])) { + $matchedTerms = array_map( + static fn ($match): string => $match->term(), + $result->matches(), + ); + + foreach ($expect['terms'] as $term) { + expect($matchedTerms)->toContain($term); + } + } + + if (isset($expect['severity'])) { + expect($result->severity())->toBe($expect['severity']); + } + + if (isset($expect['categories'])) { + $matchedCategories = array_map( + static fn ($match): string => $match->category(), + $result->matches(), + ); + + foreach ($expect['categories'] as $category) { + expect($matchedCategories)->toContain($category); + } + } +} + +test('clean corpus produces no false positives', function (array $case) { + assertCorpusExpectations(turkishGuard(), $case); +})->with(fn (): array => array_map( + static fn (array $case): array => [$case], + CorpusLoader::loadClean(), +)); + +test('profane corpus is detected', function (array $case) { + assertCorpusExpectations(turkishGuard(), $case); +})->with(fn (): array => array_map( + static fn (array $case): array => [$case], + CorpusLoader::loadProfane(), +)); + +test('obfuscated corpus is detected', function (array $case) { + assertCorpusExpectations(turkishGuard(), $case); +})->with(fn (): array => array_map( + static fn (array $case): array => [$case], + CorpusLoader::loadObfuscated(), +)); + +test('edge cases match expectations', function (array $case) { + assertCorpusExpectations(turkishGuard(), $case); +})->with(fn (): array => array_map( + static fn (array $case): array => [$case], + CorpusLoader::loadEdge(), +)); + +test('aggregate report has zero false positives', function () { + $report = CorpusQualityReport::fromAllFixtures(turkishGuard()); + + expect($report->falsePositiveCount())->toBe(0) + ->and($report->falsePositiveCases())->toBe([]); +}); + +test('aggregate report has zero false negatives', function () { + $report = CorpusQualityReport::fromAllFixtures(turkishGuard()); + + expect($report->falseNegativeCount())->toBe(0) + ->and($report->falseNegativeCases())->toBe([]); +}); + +test('aggregate detection coverage is 100', function () { + $report = CorpusQualityReport::fromDetectionCorpus(turkishGuard()); + + expect($report->detectionCoverage())->toBe(100.0); +});