Skip to content

Ingest Substack, Medium, and Ghost exports as first-class writing sources #56

Description

@telivity-otaip

Why

LinkedIn export + local notes cover a lot, but many writers keep their longform on Substack / Medium / Ghost. Broader ingest = richer article-channel voice indexes and a bigger potential user base discovering the repo.

Scope

Extend ingest so these sources land as proper Piece rows (prefer linkedin_article-equivalent longform or a new source label that write --channel article can use).

Suggested sources

Source Input shape Notes
Substack export ZIP or posts/ markdown Dates + titles from front matter
Medium export ZIP (posts/ HTML) Strip Medium chrome/CSS (see corpus_text.py patterns)
Ghost JSON export or markdown Map lexical/html → clean text

CLI sketch:

personality-protect ingest --substack ~/Exports/substack.zip
personality-protect ingest --medium ~/Exports/medium.zip
personality-protect ingest --ghost ~/Exports/ghost.json

Acceptance criteria

  • At least two of the three sources fully supported (third can be follow-up)
  • Pieces get stable ids, dates/years when present, and a clear source field
  • HTML/CSS chrome stripped so style measurement sees real prose
  • Synthetic Contoso fixtures + unit tests (no real author exports in the repo)
  • README ingest section updated
  • Article channel can retrieve these as longform rhythm refs when labeled appropriately

Pointers

  • src/personality_protect/ingest.py
  • src/personality_protect/corpus_text.py (Medium/Ghost CSS stripping already exists)
  • src/personality_protect/models.py (Piece.source)

Privacy

Never commit real exports. Ship only synthetic fixtures under tests/fixtures/.

Difficulty

Medium — parsing + tests; high user value.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesthelp wantedExtra attention is needed

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions