fix: harden location extraction and docs automation safety

.github/copilot-instructions.md

@@ -13,4 +13,8 @@ When writing analytical functions, categorize them strictly into these four tier
 1.  **Descriptive:** Word glossaries, frequency counters, and extraction of hard dates/locations (What happened?).
 2.  **Diagnostic:** Correlation maps linking specific word usage rates to specific dates or locations (Why did it happen?).
 3.  **Predictive:** Trend forecasting based on historical file timestamps, projecting future keyword frequencies or location clusters (What is likely to happen?).
-4.  **Prescriptive:** Actionable output generation, such as recommending folder restructuring based on topic clusters or flagging missing data points (What should we do?).
+4.  **Prescriptive:** Actionable output generation, such as recommending folder restructuring based on topic clusters or flagging missing data points (What should we do?).
+
+## Current Objectives
+* When suggesting new features, refer to `docs/ROADMAP.md` to align with the current development stage.
+* The immediate priority is Stage 1: NLP Integration.

.github/prompts/stage1-nlp.prompt.md

@@ -0,0 +1,17 @@
+---
+description: "Implement NLP Extraction in file-ingestion.js"
+name: "Execute Roadmap Stage 1"
+agent: "agent"
+---
+Implement Stage 1 from `docs/ROADMAP.md` into `src/ingestion/file-ingestion.js`.
+
+Use this workflow:
+1. Inspect `src/ingestion/file-ingestion.js`. Notice the regex implementations for `extractDates` and `extractLocations`.
+2. Replace these functions using the `compromise` NLP library.
+3. Keep the function signatures the same. They must return an array of strings.
+4. Maintain the non-destructive, read-only architectural rules.
+5. Do not alter the async stream processing logic, only the text extraction logic.
+
+Output requirements:
+- Edit `src/ingestion/file-ingestion.js` directly.
+- Ensure the changes pass the existing tests in `test/pipeline.test.js`.

.github/prompts/update-readme.prompt.md

@@ -1,25 +1,29 @@
 ---
-description: "Update README.md to reflect current implementation and usage in UAP AnalyticsBot"
+description: "Update README.md and related docs to reflect the current implementation in UAP AnalyticsBot"
 name: "Update README"
-argument-hint: "What changed that README should include?"
+argument-hint: "What changed that the active docs should include?"
 agent: "agent"
 ---
-Update [README.md](../../README.md) to match the current state of this repository.
+Update the active documentation set to match the current state of this repository.
 
 Use this workflow:
 1. Inspect the code and docs to identify behavior, CLI usage, dependencies, and architecture status.
 2. Revise only the sections that are now inaccurate or incomplete.
 3. Preserve the core domain framing: Ingest -> Analyze -> Report.
 4. Keep the analytics tiers explicitly categorized as Descriptive, Diagnostic, Predictive, and Prescriptive.
 5. Preserve the non-destructive guarantee: source folder operations are read-only.
+6. Keep [docs/architecture.md](../../docs/architecture.md) hand-authored and aligned with current-vs-planned system boundaries.
+7. Keep Python prototype details in [docs/legacy-prototype.md](../../docs/legacy-prototype.md), not in the main README.
+8. If command reference, supported file types, or repo layout metadata changed, update [docs/docs-source.json](../../docs/docs-source.json) and regenerate README via `npm run docs:generate`.
 
 Output requirements:
 - Edit [README.md](../../README.md) directly.
+- Update related docs when needed.
 - Keep language concise, technically accurate, and contributor-friendly.
 - Prefer incremental edits over full rewrites unless requested.
-- Keep setup and usage examples runnable on Windows PowerShell.
+- Keep narrative docs hand-authored and generated sections bounded.
 
 Input from user:
 - {{$ARGUMENTS}}
 
-If user input is empty, infer updates from repository changes and refresh the most likely outdated sections (implementation status, usage, supported types, and architecture notes).
+If user input is empty, infer updates from repository changes and refresh the most likely outdated sections (implementation status, usage, supported types, architecture notes, and legacy Python references).

.github/pull_request_template.md

@@ -0,0 +1,10 @@
+## Summary
+
+- Describe the change.
+
+## Checklist
+
+- [ ] I reviewed whether this change affects README, architecture docs, or legacy docs.
+- [ ] If commands, supported file types, or layout metadata changed, I ran `npm run docs:generate`.
+- [ ] I ran `npm run docs:check`.
+- [ ] If historical Python prototype guidance changed, I updated `docs/legacy-prototype.md`.

.github/workflows/docs.yml

@@ -0,0 +1,29 @@
+name: Documentation
+
+on:
+  pull_request:
+  release:
+    types:
+      - published
+
+permissions:
+  contents: read
+
+jobs:
+  docs:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Check out repository
+        uses: actions/checkout@v4
+
+      - name: Set up Node.js
+        uses: actions/setup-node@v4
+        with:
+          node-version: 22
+
+      - name: Install npm dependencies
+        run: npm install
+
+      - name: Validate documentation
+        run: npm run docs:check

README.md

@@ -19,140 +19,92 @@ The analysis stage is intentionally split into four tiers:
 3. **Predictive**: What is likely to happen? (trend forecasting from historical timestamps)
 4. **Prescriptive**: What should we do? (actionable recommendations and data-quality flags)
 
-## Current Implementation Status
+## Current Implementation
 
-The repository currently includes the **Ingestion Engine** as the executable entry point (Node.js):
+The active implementation is a Node.js CLI that:
 
-- Walks a source directory in read-only mode
-- Filters supported file types
-- Produces descriptive, diagnostic, predictive, and prescriptive analytics in JSON
-- Enforces non-destructive, read-only behavior on source data
+- resolves the source directory from the first CLI argument, or defaults to the current working directory
+- recursively scans supported text files in read-only mode
+- extracts words, dates, locations, and file metadata
+- builds descriptive, diagnostic, predictive, and prescriptive analytics
+- emits a formatted JSON report to standard output
 
-The current Node.js pipeline already executes the descriptive, diagnostic, predictive, and prescriptive analytics stages and emits a JSON report. Still planned are more advanced extraction capabilities, NLP/entity recognition (NER), and additional reporting outputs beyond the current JSON format; see `docs/architecture.md`.
+See [docs/architecture.md](docs/architecture.md) for the current-vs-planned architecture view. Historical Python prototype details live in [docs/legacy-prototype.md](docs/legacy-prototype.md).
 
-## Past Implementations
+## CLI Runtime Behavior
 
-### Python CLI Ingestion Engine
+On a successful run, the CLI prints a JSON report to stdout that includes:
 
-An earlier Python-based implementation provided:
+- `sourceDirectory`
+- `descriptive`
+- `diagnostic`
+- `predictive`
+- `prescriptive`
 
-- Recursive directory scanning with asynchronous generators
-- Supported file type filtering (`.pdf`, `.mp4`, `.jpg`, `.jpeg`, `.png`)
-- Command-line interface for manual trigger and folder path configuration
-- File discovery and emission for downstream processing
-- Read-only behavioral guarantees
+If an error occurs, the CLI prints the error message to stderr and exits with a non-zero status.
 
-This approach laid the foundation for the current Node.js analytics-focused implementation.
+## Command Reference
 
-## Supported File Types
-
-Current Node.js ingestion scan targets:
-
-- `.txt`
-- `.md`
-- `.json`
-- `.csv`
-- `.log`
-
-(The Python `ingestion.py` prototype scans `.pdf`, `.mp4`, `.jpg`, `.jpeg`, `.png`.)
-## Setup
-
-### 1) Create and activate a virtual environment
-
-Windows PowerShell:
+<!-- GENERATED:commands:START -->
+| Command | Purpose |
+| --- | --- |
+| `npm start -- /absolute/path/to/source-folder` | Run the active Node CLI and emit a JSON analytics report. |
+| `npm test` | Run the current Node test suite. |
+| `npm run docs:generate` | Refresh autogenerated documentation sections. |
+| `npm run docs:check` | Verify autogenerated docs are current and that required documentation references remain valid. |
+<!-- GENERATED:commands:END -->
 
-```powershell
-python -m venv .venv
-.\.venv\Scripts\Activate.ps1
-```
-
-### 2) Install dependencies
-
-```powershell
-pip install -r requirements.txt
-```
-
-## Usage
+## Supported File Types
 
-Run ingestion against a folder path:
+The current Node ingestion pipeline only analyzes text-oriented files.
 
-```powershell
-python ingestion.py "C:\path\to\source_folder"
-```
+<!-- GENERATED:supported-file-types:START -->
+| Extension | Status |
+| --- | --- |
+| `.txt` | Ingested by the active Node pipeline |
+| `.md` | Ingested by the active Node pipeline |
+| `.json` | Ingested by the active Node pipeline |
+| `.csv` | Ingested by the active Node pipeline |
+| `.log` | Ingested by the active Node pipeline |
+<!-- GENERATED:supported-file-types:END -->
 
-You can also run with a relative path:
+## Repository Layout
 
-```powershell
-python ingestion.py .\data
-```
+<!-- GENERATED:repo-layout:START -->
+- `src/index.js` — Node CLI entry point.
+- `src/pipeline.js` — Pipeline coordinator that assembles all analytics tiers.
+- `src/ingestion/file-ingestion.js` — Read-only recursive file ingestion for supported text files.
+- `src/analytics/` — Descriptive, diagnostic, predictive, and prescriptive analytics modules.
+- `test/pipeline.test.js` — Node test coverage for core pipeline behavior.
+- `docs/architecture.md` — Hand-authored architecture overview for current and planned system design.
+- `docs/legacy-prototype.md` — Historical Python prototype reference.
+<!-- GENERATED:repo-layout:END -->
 
-Show CLI help:
+## Testing
 
-```powershell
-python ingestion.py -h
-```
+The current Node test suite verifies that:
 
-## Expected Runtime Behavior
+- the full analytics report is produced for supported text fixtures
+- dates and locations are extracted into analytics outputs
+- prescriptive recommendations flag files with missing metadata
 
-On a successful run, the CLI:
+Run `npm test` to execute the existing suite.
 
-1. Prints a scan banner
-2. Prints a separator line
-3. Lists each matching file as `[FOUND] <name>`
-4. If no matching files are found, prints a "no supported files" message with the supported extension list
-5. Exits with code `0`
+## Documentation Workflow
 
-If the folder does not exist, it prints an error and exits with code `1`.
+- Narrative documentation stays hand-authored.
+- Command reference, supported file types, and repository layout are generated from repository metadata.
+- Run `npm run docs:generate` after changing documented commands, supported file types, or layout metadata.
+- Run `npm run docs:check` before submitting changes; the repository is configured to enforce this on pull requests and releases.
 
 ## Non-Destructive Guarantee
 
 The bot must never modify, move, or delete ingested source files. Ingestion is read-only by design.
 
-## Repository Layout
-
-    .
-    |- ingestion.py
-    |- requirements.txt
-    |- package.json
-    |- README.md
-    |- src/
-    |- test/
-    `- docs/
-        `- architecture.md
-
-## Architecture Reference
-
-See `docs/architecture.md` for the planned full pipeline:
-
-1. Ingestion Engine (currently implemented as a manual CLI trigger)
-2. Extraction Node
-3. NLP + Entity Recognition
-4. Analytics Engine (Descriptive, Diagnostic, Predictive, Prescriptive)
-5. Output Layer (JSON/dashboard)
-
 ## Notes for Contributors and Copilot
 
 - Keep ingestion logic modular and separate from analytics logic.
 - Prefer asynchronous and streaming patterns for large datasets.
 - Preserve strict read-only behavior for source directories.
 - When adding analytics, classify behavior under one of the four analytics tiers.
-
-## Initial implementation
-
-The repository now contains a minimal Node.js implementation that:
-
-- walks a source directory in read-only mode
-- streams supported text files for ingestion
-- produces descriptive, diagnostic, predictive, and prescriptive analytics in JSON
-
-### Usage
-
-```bash
-npm start -- /absolute/path/to/source-folder
-```
-
-### Testing
-
-```bash
-npm test
-```
+- Update [docs/architecture.md](docs/architecture.md) when implementation changes affect current-vs-planned system boundaries.

docs/ROADMAP.md

@@ -0,0 +1,30 @@
+# Development Roadmap: UAP AnalyticsBot
+
+This document tracks the active expansion phases for the AnalyticsBot pipeline.
+
+## Stage 1: Extraction Node Upgrade (NLP Integration)
+**Goal:** Replace hardcoded Regex date/location extraction with Named Entity Recognition (NER).
+- [ ] Install NLP library (e.g., `npm install compromise`).
+- [ ] Refactor `extractDates` in `src/ingestion/file-ingestion.js` to use NLP `#Date` tagging.
+- [ ] Refactor `extractLocations` in `src/ingestion/file-ingestion.js` to use NLP `#Place` tagging.
+- [ ] Ensure the output schema matches the existing `dates: []` and `locations: []` arrays so downstream analytics engines do not break.
+
+## Stage 2: Multimedia Ingestion Routing
+**Goal:** Reintroduce legacy Python capabilities to parse PDFs and Images via the Node pipeline.
+- [ ] Install extraction dependencies (`npm install pdf-parse tesseract.js`).
+- [ ] Update `TEXT_EXTENSIONS` set to include `.pdf`, `.png`, `.jpg`.
+- [ ] Abstract `readTextFile` into a routing function (`readFileData`).
+- [ ] Implement PDF buffer streaming and OCR image processing before passing text strings to the normalizer.
+
+## Stage 3: Alternate Delivery Surfaces
+**Goal:** Abstract the output layer to support CSV and Markdown reports instead of just stdout JSON.
+- [ ] Create `src/delivery/` directory.
+- [ ] Implement `src/delivery/markdown-generator.js`.
+- [ ] Update `src/index.js` to accept CLI flags (e.g., `--format=md`).
+- [ ] Route generated reports to a local ignored folder (e.g., `/data_exports/`).
+
+## Stage 4: Continuous Event Loop
+**Goal:** Allow the bot to monitor a directory and update analytics dynamically.
+- [ ] Install filesystem watcher (`npm install chokidar`).
+- [ ] Implement a `--watch` flag in the CLI orchestrator.
+- [ ] Hook `chokidar` file events (add, change, unlink) into the `pipeline.js` to update the corpus index without full application restarts.