diff --git a/README.md b/README.md index 7f3fdf6..0c86f29 100644 --- a/README.md +++ b/README.md @@ -31,11 +31,12 @@ PersonalityProtect keeps the corpus on disk, measures your cadence, retrieves sh ## How you get voice 1. **Ingest** your LinkedIn export and/or local notes (stays on disk). -2. **`index-voice`** builds a local retrieval index. -3. **`build-style-profile`** measures cadence (sentence length, short lines, post length band, banned filler). -4. **`write --topic --points`** drafts from the brief only; retrieved pieces are rhythm reference. +2. **`select`** gates the corpus by length (`--min-words`, default 50) and an optional year cap (`--through-year`, default: current year). +3. **`index-voice`** builds a local retrieval index. +4. **`build-style-profile`** measures cadence from the selection (sentence length, short lines, post length band, banned filler). +5. **`write --topic --points`** drafts from the brief only; retrieved pieces are rhythm reference. -Two channels come out of step 4: +Two channels come out of step 5: - **`--channel post`** (default) targets your long-post band, up to the LinkedIn ~3000-character limit (~550 words). - **`--channel article`** runs outline → sections → stitch, and needs at least five `linkedin_article` pieces in the corpus. @@ -61,6 +62,7 @@ personality-protect download --format mlx # ~6 GB, once personality-protect ingest --linkedin ~/path/to/linkedin-export personality-protect ingest --path ~/path/to/notes --source note +personality-protect select personality-protect index-voice personality-protect build-style-profile @@ -94,18 +96,23 @@ pip install -e ".[cuda]" # NVIDIA path (optional) Public docs use **synthetic Contoso / synergy-slop text only**. No personal corpus. -| `write` (post + article) | `status` | Mark | +| `write` (post + article) | `status` | Setup | | --- | --- | --- | -| write drafting a Contoso post with adapter=none | status output for the synthetic demo profile | Telivity CLI logo | +| write drafting a Contoso post with adapter=none | status output for the synthetic demo profile | personality-protect setup / logo | ```bash +personality-protect select personality-protect index-voice personality-protect build-style-profile personality-protect write --topic "Contoso pricing" --points "Name one owner." personality-protect status ``` -Optional smoke tour (no model download; synthetic only): +Optional smoke tour — runs the write path with a stubbed model call (no download; synthetic Contoso only): + +| Smoke tour (`demo`) | Mark | +| --- | --- | +| personality-protect demo smoke tour of the write path | Telivity CLI logo | ```bash personality-protect demo @@ -160,13 +167,20 @@ personality-protect ingest --linkedin ~/path/to/linkedin-export.zip personality-protect ingest --path ~/path/to/notes --source note ``` -### Index and style +### Select, index and style + +`select` is required before `build-style-profile` (the style card reads `selection.json`). It is a length gate plus an optional year cap: ```bash +personality-protect select +personality-protect select --min-words 75 --include-undated +personality-protect select --through-year 2024 # deliberate narrowing only personality-protect index-voice personality-protect build-style-profile ``` +Defaults: **≥50 words**, dates through the **current year**. Use `--through-year` when you intentionally want an older slice. Corpus gates: **warn** below 50 selected pieces; **block** below 20 unless `--force`. Holding pieces back from retrieval is separate — `index-voice --holdout-id`, scored by `eval-write-holdout`. + Post length targets come from `linkedin_post` pieces (p75/p90), clamped to the LinkedIn ~3000-character band (~550 words). ### Write @@ -197,12 +211,13 @@ Global flags (most commands): `--profile`, `--home`, `--json`, plus branding `-- | `init` | Create profile under `~/.personality-protect/` | | `download` | Prefetch quantized MLX or GGUF base | | `ingest` | Index LinkedIn export and/or local paths | +| `select` | Gate corpus by length / year — required before `build-style-profile` | | `index-voice` | Build local voice retrieval index | | `build-style-profile` | Build cadence / length / banned-filler style card | | `write` | Draft a post or article (`--channel post\|article`) | | `eval-write-holdout` | Score write quality on held-out pieces (local receipt) | | `status` | Show profile state | -| `demo` | Optional synthetic smoke tour (no download) | +| `demo` | Optional synthetic smoke tour of the write path (no download) | | `api` | Loopback HTTP stub | | `logo` | Print Telivity CLI mark | | `build-writer-sft`, `train` | Optional LoRA experiments — see [Advanced](#advanced-optional) | @@ -247,7 +262,7 @@ Keep an adapter only if `eval-write-holdout` shows it beating RAG-alone on held- ### Other experiment commands -`select`, `filter`, `compare`, `eval`, and the translator-pair commands remain available. They score or rewrite existing text and are not part of the drafting path above. +`filter`, `compare`, `eval`, and the translator-pair commands remain available. They score or rewrite existing text and are not part of the drafting path above. (`select` *is* part of the drafting path — see [Select, index and style](#select-index-and-style).) ### Operator script @@ -268,11 +283,26 @@ Operator checklist: [docs/LAUNCH.md](docs/LAUNCH.md). ```bash pip install -e ".[dev]" pytest -ruff check src tests +ruff check src tests scripts ``` CI (`.github/workflows/ci.yml`) required checks: `lint`, `test (3.11)`, `test (3.12)`, `sanitize`, `cli-smoke`. +### Regenerating screenshots + +`scripts/shot.py` renders captured ANSI terminal bytes to PNG on a fixed character grid (so Rich box-drawing lines up). Needs `pillow`. + +```bash +export PERSONALITY_PROTECT_HOME=/tmp/shots COLUMNS=94 TERM=xterm-256color +pip install pillow +script -qec "personality-protect --logo off demo" /dev/null \ + | python3 scripts/shot.py docs/images/cli-demo.png "personality-protect demo" +script -qec "personality-protect --logo off status" /dev/null \ + | python3 scripts/shot.py docs/images/cli-status.png "personality-protect status" +``` + +Do not regenerate `docs/images/cli-shipped.png` off Apple Silicon — it shows a real `write` against MLX weights. + --- ## License diff --git a/docs/images/cli-demo.png b/docs/images/cli-demo.png index 08a211b..d9fe7ed 100644 Binary files a/docs/images/cli-demo.png and b/docs/images/cli-demo.png differ diff --git a/docs/images/cli-setup.png b/docs/images/cli-setup.png new file mode 100644 index 0000000..872c736 Binary files /dev/null and b/docs/images/cli-setup.png differ diff --git a/docs/images/cli-status.png b/docs/images/cli-status.png index 2335e4a..faf7c8b 100644 Binary files a/docs/images/cli-status.png and b/docs/images/cli-status.png differ diff --git a/scripts/shot.py b/scripts/shot.py new file mode 100755 index 0000000..e29c880 --- /dev/null +++ b/scripts/shot.py @@ -0,0 +1,272 @@ +#!/usr/bin/env python3 +"""Render captured ANSI terminal bytes to a PNG on a fixed character grid. + +Usage: + script -qec "personality-protect --logo off demo" /dev/null \\ + | python3 scripts/shot.py docs/images/cli-demo.png "personality-protect demo" + +Needs pillow. Fixed cell size keeps Rich box-drawing aligned. +""" + +from __future__ import annotations + +import re +import sys +from pathlib import Path + +from PIL import Image, ImageDraw, ImageFont + +# Fixed grid — match existing docs shots (≈10×20 cell, macOS chrome). +CELL_W = 10 +CELL_H = 20 +PAD_X = 14 +PAD_Y = 12 +TITLE_H = 36 +COLS = 94 + +BG = (18, 20, 24) +TITLE_BG = (40, 42, 48) +TITLE_EDGE = (48, 50, 56) +FG = (230, 232, 236) +DIM = (140, 146, 156) +BOLD = (255, 255, 255) +CYAN = (120, 210, 220) +GREEN = (120, 200, 140) +YELLOW = (220, 190, 100) +RED = (230, 120, 110) +MAGENTA = (200, 140, 200) +BLUE = (120, 160, 230) + +# CSI / OSC / charset noise from `script` / Rich +_ANSI_OSC = re.compile(r"\x1b\][^\x07\x1b]*(?:\x07|\x1b\\)") +_ANSI_CSI = re.compile(r"\x1b\[[0-9;?]*[ -/]*[@-~]") +_ANSI_OTHER = re.compile(r"\x1b[@-Z\\-_]") +_ANSI_CHARSET = re.compile(r"\x1b[()][0-9A-Za-z]") + + +def _font(size: int = 15) -> ImageFont.FreeTypeFont | ImageFont.ImageFont: + candidates = [ + "/usr/share/fonts/truetype/jetbrains-mono/JetBrainsMono-Regular.ttf", + "/usr/share/fonts/truetype/dejavu/DejaVuSansMono.ttf", + "/System/Library/Fonts/Menlo.ttc", + ] + for path in candidates: + if Path(path).is_file(): + return ImageFont.truetype(path, size=size) + return ImageFont.load_default() + + +def _sgr_color(code: int, fg: bool) -> tuple[int, int, int] | None: + table = { + 30: (60, 64, 72), + 31: RED, + 32: GREEN, + 33: YELLOW, + 34: BLUE, + 35: MAGENTA, + 36: CYAN, + 37: FG, + 90: DIM, + 91: RED, + 92: GREEN, + 93: YELLOW, + 94: BLUE, + 95: MAGENTA, + 96: CYAN, + 97: BOLD, + } + if not fg: + return None + return table.get(code) + + +def parse_ansi(data: bytes) -> list[list[tuple[str, tuple[int, int, int], bool]]]: + """Return rows of (char, fg, bold) cells.""" + text = data.decode("utf-8", errors="replace") + text = text.replace("\r\n", "\n").replace("\r", "\n") + text = _ANSI_OSC.sub("", text) + text = _ANSI_CHARSET.sub("", text) + + rows: list[list[tuple[str, tuple[int, int, int], bool]]] = [[]] + fg = FG + bold = False + i = 0 + while i < len(text): + ch = text[i] + if ch == "\x1b": + m = _ANSI_CSI.match(text, i) + if m: + seq = m.group(0) + i = m.end() + if seq.endswith("m"): + body = seq[2:-1] + parts = [int(x) for x in body.split(";") if x.isdigit()] if body else [0] + j = 0 + while j < len(parts): + code = parts[j] + if code == 0: + fg, bold = FG, False + elif code == 1: + bold = True + if fg == FG: + fg = BOLD + elif code == 2: + fg = DIM + elif code == 22: + bold = False + if fg == BOLD: + fg = FG + elif 30 <= code <= 37 or 90 <= code <= 97: + color = _sgr_color(code, True) + if color: + fg = color + elif code == 39: + fg = BOLD if bold else FG + elif code == 38 and j + 1 < len(parts): + mode = parts[j + 1] + if mode == 2 and j + 4 < len(parts): + fg = (parts[j + 2], parts[j + 3], parts[j + 4]) + j += 4 + elif mode == 5 and j + 2 < len(parts): + # 256-color: approximate via grayscale/primary buckets + n = parts[j + 2] + if n < 16: + basic = [ + (0, 0, 0), + (205, 0, 0), + (0, 205, 0), + (205, 205, 0), + (0, 0, 238), + (205, 0, 205), + (0, 205, 205), + (229, 229, 229), + (127, 127, 127), + RED, + GREEN, + YELLOW, + BLUE, + MAGENTA, + CYAN, + BOLD, + ] + fg = basic[n] + elif n < 232: + c = n - 16 + r = (c // 36) * 51 + g = ((c // 6) % 6) * 51 + b = (c % 6) * 51 + fg = (r, g, b) + else: + v = 8 + (n - 232) * 10 + fg = (v, v, v) + j += 2 + else: + j += 1 + j += 1 + continue + m2 = _ANSI_OTHER.match(text, i) + if m2: + i = m2.end() + continue + i += 1 + continue + if ch == "\n": + rows.append([]) + i += 1 + continue + if ch == "\t": + spaces = 4 - (len(rows[-1]) % 4) + rows[-1].extend([(" ", fg, bold)] * spaces) + i += 1 + continue + if ch == "\x08": + if rows[-1]: + rows[-1].pop() + i += 1 + continue + if ord(ch) < 32: + i += 1 + continue + rows[-1].append((ch, fg, bold)) + i += 1 + + # Drop trailing empty rows from script(1) noise + while rows and not rows[-1]: + rows.pop() + return rows + + +def _strip_script_noise(rows: list[list[tuple[str, tuple[int, int, int], bool]]]): + """Drop typescript headers / bare prompts that `script` sometimes emits.""" + cleaned = [] + for row in rows: + line = "".join(c for c, _, _ in row).strip() + if line.startswith("Script started") or line.startswith("Script done"): + continue + cleaned.append(row) + return cleaned + + +def render( + rows: list[list[tuple[str, tuple[int, int, int], bool]]], + title: str, + cols: int = COLS, +) -> Image.Image: + rows = _strip_script_noise(rows) + # Prepend a synthetic prompt line matching existing docs shots + prompt = [ + (">", CYAN, True), + (" ", FG, False), + ] + prompt.extend((c, BOLD if c != " " else FG, c != " ") for c in title) + # If the capture already starts with the command, don't double it + first = "".join(c for c, _, _ in rows[0]).strip() if rows else "" + if not first.startswith(">") and title not in first: + rows = [prompt] + rows + + width = PAD_X * 2 + cols * CELL_W + height = TITLE_H + PAD_Y * 2 + max(1, len(rows)) * CELL_H + img = Image.new("RGB", (width, height), BG) + draw = ImageDraw.Draw(img) + + # Title bar + draw.rectangle([0, 0, width, TITLE_H], fill=TITLE_BG) + draw.line([(0, TITLE_H), (width, TITLE_H)], fill=TITLE_EDGE) + for x, color in ((18, (255, 95, 86)), (38, (255, 189, 46)), (58, (39, 201, 63))): + draw.ellipse([x, 12, x + 12, 24], fill=color) + font_title = _font(13) + tw = draw.textlength(title, font=font_title) + draw.text(((width - tw) / 2, 10), title, fill=DIM, font=font_title) + + font = _font(15) + y = TITLE_H + PAD_Y + for row in rows: + x = PAD_X + for ch, color, _bold in row[:cols]: + draw.text((x, y - 1), ch, fill=color, font=font) + x += CELL_W + y += CELL_H + return img + + +def main(argv: list[str]) -> int: + if len(argv) < 2: + print( + "usage: shot.py OUT.png [TITLE]\n" + " reads ANSI bytes from stdin", + file=sys.stderr, + ) + return 2 + out = Path(argv[1]) + title = argv[2] if len(argv) > 2 else out.stem + data = sys.stdin.buffer.read() + rows = parse_ansi(data) + img = render(rows, title=title) + out.parent.mkdir(parents=True, exist_ok=True) + img.save(out, format="PNG", optimize=True) + print(f"wrote {out} ({img.size[0]}x{img.size[1]})") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main(sys.argv)) diff --git a/src/personality_protect/cli.py b/src/personality_protect/cli.py index 0910dec..d149265 100644 --- a/src/personality_protect/cli.py +++ b/src/personality_protect/cli.py @@ -459,7 +459,8 @@ def select_cmd( console.print(f"[red]{gate_error}[/red]") raise typer.Exit(2) console.print(f"Saved: {paths.selection_path}") - console.print("Next: personality-protect train") + console.print("Next: personality-protect index-voice") + console.print("Then: personality-protect build-style-profile") @app.command("build-style-profile") @@ -1104,8 +1105,8 @@ def on_progress(info: dict) -> None: if result.notes: console.print(result.notes) if result.status == "ok": - console.print("Next: personality-protect filter --text '…'") - console.print("Or: personality-protect compare --synthetic slop_branding") + console.print("Next: personality-protect eval-write-holdout --out receipt.json") + console.print("[dim]Keep this adapter only if it beats RAG-alone on the holdout.[/dim]") @app.command("filter") @@ -1414,20 +1415,42 @@ def demo_cmd( ), as_json: bool = typer.Option(False, "--json"), ) -> None: - """Optional synthetic mock tour (not the shipped mlx/llama path).""" + """Optional synthetic smoke tour of the write path (no model download).""" _banner_from_ctx(ctx, json_mode=as_json) result = run_demo(home=home, draft=draft) if as_json: typer.echo(json.dumps(result, indent=2, ensure_ascii=False)) return console.print( - "[bold]Mock tour complete[/bold] " - "(synthetic data + mock adapter only — not the shipped product path)." + "[bold]Smoke tour complete[/bold] " + "(synthetic Contoso corpus; the model call inside write is stubbed)." ) console.print( - "[dim]Shipped path: train --backend mlx|cuda → filter/compare with llama|mlx.[/dim]" + "[dim]Product path: ingest → select → index-voice → " + "build-style-profile → write.[/dim]" ) - console.print(f"Ingested: {result['ingested']} Selected: {result['selected']}") + console.print( + "[dim]For a real draft: download --format mlx, then " + "write --topic … --points … on Apple Silicon.[/dim]" + ) + console.print( + f"Ingested: {result['ingested']} Selected: {result['selected']} " + f"Indexed: {result['indexed']}" + ) + console.print( + f"Style card: {result['style_pieces']} pieces, " + f"{result['banned_ai_filler']} banned filler phrases" + ) + console.print("") + exemplars = result.get("write_exemplars") or [] + console.print( + f"write channel={result['write_channel']} " + f"adapter={result['write_adapter']} model=stub " + f"exemplars={len(exemplars)}" + ) + console.print(result["written"]) + console.print("") + console.print("[dim]Legacy mock train/filter (not the drafting path):[/dim]") console.print(f"Train: {result['train_status']} via {result['train_backend']}") console.print("") console.print("[bold]Draft[/bold]") diff --git a/src/personality_protect/config.py b/src/personality_protect/config.py index b4f6560..4d22ded 100644 --- a/src/personality_protect/config.py +++ b/src/personality_protect/config.py @@ -5,6 +5,7 @@ import json import os from dataclasses import asdict, dataclass, field +from datetime import datetime, timezone from pathlib import Path from typing import Any @@ -26,7 +27,14 @@ DEFAULT_BASE_MODEL = DEFAULT_MLX_MODEL DEFAULT_MIN_WORDS = 50 -DEFAULT_THROUGH_YEAR = 2024 + +# `select` caps the corpus at this year. It must NOT exclude current writing: +# a fixed past year silently selects zero pieces for anyone whose corpus is +# recent, and the style profile is built from the selection. The writer SFT +# path already bypasses the year gate for exactly this reason +# (see run_build_writer_sft). Use --through-year to narrow deliberately. +# Holdouts are a separate, explicit mechanism (--holdout-id). +DEFAULT_THROUGH_YEAR = datetime.now(timezone.utc).year DEFAULT_PROFILE = "default" DEFAULT_VOICE_MODE = "rag" DEFAULT_WRITE_ADAPTER: str | None = None diff --git a/src/personality_protect/demo.py b/src/personality_protect/demo.py index c9f0f22..38de785 100644 --- a/src/personality_protect/demo.py +++ b/src/personality_protect/demo.py @@ -9,10 +9,31 @@ from personality_protect.filter import filter_draft from personality_protect.ingest import run_ingest from personality_protect.select import run_select +from personality_protect.style_profile import run_build_style_profile from personality_protect.train import run_train +from personality_protect.voice_index import build_voice_index +from personality_protect.write import run_write DEMO_PROFILE = "demo" +# Synthetic brief for the demo's write step. Contoso only — never personal text. +DEMO_TOPIC = "Contoso Ledger exceptions queue" +DEMO_POINTS = "- Name one owner\n- Keep the rollout boring" + +# The demo never downloads weights, so the model call is stubbed. Everything +# around it — retrieval, style card, trim, guards — is the real write path. +_DEMO_STUB_DRAFT = ( + "The Contoso Ledger exceptions queue is not a tooling problem.\n\n" + "Name one owner.\n\n" + "Keep the rollout boring. Boring is what survives contact with a Monday." +) + + +def _demo_generate(_messages, **_kwargs) -> str: + """Canned generator for the demo — no model, no download.""" + return _DEMO_STUB_DRAFT + + # Inline fallback if package data missing _FALLBACK_DOCS = { "post_2023_craft.md": """Date: 2023-06-12 @@ -98,7 +119,7 @@ def run_demo( home: Path | None = None, draft: str | None = None, ) -> dict: - """Full synthetic pipeline: init → ingest → select → mock train → filter.""" + """Full synthetic pipeline: init → ingest → select → write path → mock train/filter.""" paths, config, _ = init_profile(DEMO_PROFILE, home=home, force=True) corpus = ensure_demo_corpus(paths.cache_dir / "demo_corpus") added, _ = run_ingest(paths, local=[corpus], source_hint="demo") @@ -108,6 +129,17 @@ def run_demo( through_year=2024, include_undated=True, ) + voice = build_voice_index(paths) + style, _style_path = run_build_style_profile(paths) + written = run_write( + DEMO_TOPIC, + DEMO_POINTS, + paths, + k=2, + channel="post", + use_adapter=False, + generate_fn=_demo_generate, + ) train = run_train( paths, backend="mock", @@ -129,6 +161,15 @@ def run_demo( "ingested": added, "selected": len(selected), "selection_summary": selection.summary, + "indexed": voice["indexed"], + "style_pieces": style["stats"]["pieces"], + "banned_ai_filler": len(style["banned_ai_filler"]), + "write_topic": DEMO_TOPIC, + "write_channel": written["channel"], + "write_adapter": written["adapter"], + "write_exemplars": written["exemplar_ids"], + "write_stubbed_model": True, + "written": written["text"], "train_status": train.status, "train_backend": train.backend, "adapter_dir": train.adapter_dir, diff --git a/tests/test_demo_pipeline.py b/tests/test_demo_pipeline.py index 903821a..aa873d2 100644 --- a/tests/test_demo_pipeline.py +++ b/tests/test_demo_pipeline.py @@ -106,6 +106,58 @@ def test_detect_backend_mock(): assert detect_backend("mock") == "mock" +def test_demo_runs_write_path(tmp_path: Path): + """Demo must exercise index-voice → style profile → write, not just train/filter.""" + result = run_demo(home=tmp_path) + assert result["indexed"] >= 1 + assert result["style_pieces"] >= 1 + assert result["banned_ai_filler"] >= 1 + assert result["write_channel"] == "post" + assert result["write_adapter"] == "none" + assert result["write_stubbed_model"] is True + # Guards must have passed — run_write raises/flags rather than return junk. + assert "Contoso" in result["written"] + + +def test_style_profile_requires_selection(tmp_path: Path): + """README quick start must keep `select` before `build-style-profile`.""" + import pytest + + from personality_protect.config import get_paths, init_profile + from personality_protect.style_profile import run_build_style_profile + + init_profile("t", home=tmp_path) + with pytest.raises(FileNotFoundError): + run_build_style_profile(get_paths("t", home=tmp_path)) + + +def test_select_default_includes_current_year(tmp_path: Path): + """A corpus written this year must not select to zero under bare `select`.""" + from datetime import datetime, timezone + + from personality_protect.config import DEFAULT_THROUGH_YEAR + from personality_protect.models import Piece + from personality_protect.select import filter_pieces + + this_year = datetime.now(timezone.utc).year + assert DEFAULT_THROUGH_YEAR >= this_year + + recent = Piece( + id="r1", + source="linkedin_post", + text="word " * 200, + year=this_year, + word_count=200, + ) + kept = filter_pieces( + [recent], + min_words=50, + through_year=DEFAULT_THROUGH_YEAR, + include_undated=False, + ) + assert [p.id for p in kept] == ["r1"] + + def test_api_health(tmp_path: Path): import threading import urllib.request