Skip to content

Non-Mac write backend: draft via llama.cpp GGUF (and/or CUDA) without MLX #57

Description

@telivity-otaip

Why

write currently requires Apple Silicon + MLX. That caps adoption hard. filter already has a GGUF/llama.cpp path and train has optional CUDA — contributors who bring write to Linux/Windows NVIDIA (or CPU GGUF) unlock a much larger audience and star/fork traction.

Goal

Make personality-protect write work when MLX is unavailable by selecting an alternate local backend:

  1. GGUF / llama.cpp (preferred first milestone — already partially supported via [gguf] extra)
  2. Optional follow-up: CUDA / transformers path for NVIDIA boxes

Still local-only. No cloud fallback.

Acceptance criteria

  • On non-Apple hosts with GGUF weights present, write --channel post produces a draft
  • Backend selection: auto prefers MLX on Apple Silicon, else GGUF if installed, else clear error
  • Same prompt / style-card / retrieval contract as the MLX path (adapter=none default)
  • Article channel either works or errors with an explicit "not yet" message (document either way)
  • README Hardware section updated (what works where)
  • Tests mock the llama.cpp binding so CI stays download-free
  • Memory/timeouts documented for 16 GB machines

Pointers

  • src/personality_protect/write.py
  • src/personality_protect/filter.py (existing GGUF usage)
  • src/personality_protect/download.py
  • pyproject.toml optional extras gguf, cuda, mlx

Out of scope

  • Hosted APIs / OpenAI-compatible remote endpoints
  • Shipping weights in git

Difficulty

Hard — but the highest-leverage platform expansion in the repo. Issue farmers with llama.cpp experience welcome.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesthelp wantedExtra attention is needed

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions