feat(local-paper-index): index raw markdown corpora, not just PDFs

[gas2own]

What

Generalize local-paper-index's chunk-and-index.py so it can index a corpus that is already markdown — skipping PDF extraction entirely — while staying fully backward compatible for the existing PDF workflow.

Adds a --source-ext flag (default pdf):

# index a tree of authored .md docs directly
chunk-and-index.py my-notes /data/notes --index-path /data/notes/index.yaml --source-ext md

• Canonical extra.source_file is always written; legacy extra.source_pdf is still written for --source-ext pdf, so existing consumers and indexes are unaffected.
• Secondary tag becomes <ext>-index (md-index, txt-index, …); pdf-index unchanged for the default.
• Resumability keys on source_file with a source_pdf fallback.
• SKILL.md gains an "Indexing raw markdown (skipping PDF extraction)" section.

Why

This is the enabling change for allenai/gas2own#150 — @rodneykinney recommended using asta-documents / local-paper-index for retrieval over gas2own's own investigation record (a corpus of authored .md), noting the skill "could easily be refactored to index raw .md documents." This is that refactor. The downstream gas2own retriever PR will consume it.

Verification

• ruff check + ruff format --check clean on the changed file
• validate-skills.py: all 15 SKILL.md valid
• Smoke-indexed a 2-file markdown corpus → md-index tag, source_file metadata, relative URLs
• Confirmed PDF default still emits source_pdf + pdf-index

🤖 Generated with Claude Code

[rodneykinney]

The url field already points to the source .md field in any case, so the extra.source_file field is redundant. It can be omitted.

For the case when the .md is downstream of a .pdf the only difference should be the addition of an extra field pointing to the upstream PDF. That can be the purpose of extra.source_pdf. If it is absent, then there is no upstream document, and url points to the original source.

Instead of a --source_ext flag, introduce a flag that points to the upstream directory of PDFs. The script can iterate over PDFs in that directory to find the PDF corresponding to each .md file instead of building a name based on assumptions. This would be an improvement over the existing logic for the pdf-to-md case.

[gas2own]

Thanks @rodneykinney — reworked per your feedback (pushed in 2e6f087, on top of a merge of main which had relocated the skill to plugins/asta-preview/skills/):

1. Dropped extra.source_file — the url already points at the source .md, so it was redundant. The indexed document is the markdown.
2. extra.source_pdf is now optional upstream provenance — present only when the .md is downstream of a PDF, holding a real pointer (relative / file:// URL) to it. Absent ⇒ no upstream document, url is the original source.
3. Replaced --source-ext with --pdf-dir — the script iterates that directory and matches each .md to the PDF actually on disk (by basename), instead of synthesizing <stem>.pdf. A .md with no match is indexed without source_pdf and warned about. This is the improvement to the pdf→md path you described.

Secondary tag follows the same signal: pdf-index for PDF-derived markdown (unchanged for the existing workflow), md-index for raw markdown. Resumability now keys on url. SKILL.md Step 3 + the raw-markdown section are updated to match.

Merge conflict is resolved and all CI jobs are now running/green (they couldn't run before — see below).

[robe-ai2]

@rodneykinney could you take another look at gas2own's latest on this?

[rodneykinney]

Looks good!