chore: clarify/resolve ts-node vs dist worker mismatch (stale dist hazard)

[kapral18]

Problem

Currently, npm run index runs the main CLI via ts-node, but full + incremental indexing spawn worker threads from dist/utils/producer_worker.js.

This creates two user-facing issues:

1. README / Quick Start ambiguity

   • A user can reasonably expect npm install + npm run index ... to work.
   • In practice, indexing fails unless dist/ exists (i.e., you must run npm run build first), because worker threads are hard-coded to load from dist/.

2. Stale artifact / mixed-version hazard

   • If dist/ exists but is stale, a single run can execute new code in the main process (ts-node) while executing old code in the worker (dist JS).
   • This is subtle and hard to diagnose.

Evidence in code (paths are repo-relative):

• src/commands/full_index_producer.ts and src/commands/incremental_index_command.ts build worker path via:
  • path.join(process.cwd(), 'dist', 'utils', 'producer_worker.js')

Why this matters

• Consumers want deterministic behavior: one run should not mix sources of truth.
• Docs shouldn’t imply npm run build is optional if runtime depends on dist/.

Proposed direction: explicit execution modes (avoid auto-detect)

Recommend enforcing a strict invariant: within one run, main + workers come from the same source.

Option 1 (consumer-first / production default): dist-only

• Make the default consumer entrypoint run compiled output end-to-end.
• Change scripts so npm run index runs node dist/index.js index ... (or a wrapper that requires npm run build).
• Workers continue to load from dist/.
• Add a separate dev command for contributors.

Option 2 (dev convenience): explicit dev/source mode

• Add npm run index:dev (or similar) that runs main via ts-node AND runs workers via TS as well.
• Crucially: dev mode should never load from dist/ (even if it exists), to avoid the stale artifact hazard.

Suggested script split

• npm run build -> tsc
• npm run index -> node dist/index.js index (requires build)
• npm run index:dev -> ts-node src/index.ts index (and worker threads also use TS)

Doc follow-up

• Update README Quick Start to reflect the chosen default (likely dist-only), and optionally mention the dev command for contributors.

Acceptance criteria

• Running the tool from source does not accidentally consume stale dist/.
• Running the tool in production (Docker/systemd/cron) uses compiled JS consistently.
• README Quick Start matches actual runtime requirements.