feat: alpha upload V2 pipeline with hardened telemetry contract

AGENTS.md

@@ -28,7 +28,7 @@ The CLI (`cli/selftune/`) is the **agent's API**. The skill definition (`skill/S
 ```text
 selftune/
 ├── cli/selftune/            # TypeScript package — the CLI
-│   ├── index.ts             # CLI entry point
+│   ├── index.ts             # CLI entry point (status, doctor, alpha upload, etc.)
 │   ├── init.ts              # Agent identity bootstrap + config init
 │   ├── sync.ts              # Source-truth sync orchestration
 │   ├── orchestrate.ts       # Autonomy-first loop: sync → evolve → watch
@@ -61,11 +61,11 @@ selftune/
 │   │   └── openclaw-ingest.ts # OpenClaw session importer (experimental)
 │   ├── routes/              # HTTP route handlers (extracted from dashboard-server)
 │   ├── repair/              # Rebuild repaired skill-usage overlays
-│   ├── localdb/             # SQLite schema, direct-write, queries, materialization
+│   ├── localdb/             # SQLite schema, direct-write, queries, materialization, canonical_upload_staging
 │   │   ├── db.ts            # Database lifecycle + singleton
 │   │   ├── direct-write.ts  # Fail-open insert functions for all tables
 │   │   ├── queries.ts       # Read queries for dashboard + CLI consumers
-│   │   ├── schema.ts        # Table DDL + indexes
+│   │   ├── schema.ts        # Table DDL + indexes (includes canonical_upload_staging)
 │   │   └── materialize.ts   # JSONL → SQLite rebuild (startup/backfill only)
 │   ├── cron/                # Optional OpenClaw-specific scheduler adapter
 │   ├── memory/              # Evolution memory persistence
@@ -84,11 +84,20 @@ selftune/
 │   │   └── stopping-criteria.ts  # Stopping criteria evaluator
 │   ├── monitoring/          # Post-deploy monitoring (M4)
 │   │   └── watch.ts
+│   ├── alpha-identity.ts    # Alpha user identity (UUID, consent, persistence)
+│   ├── alpha-upload-contract.ts # Upload queue infrastructure types + PushUploadResult
+│   ├── alpha-upload/        # Alpha remote data pipeline (V2 canonical push to cloud API)
+│   │   ├── index.ts         # Upload orchestration (prepareUploads, runUploadCycle)
+│   │   ├── queue.ts         # Local upload queue + watermark tracking
+│   │   ├── stage-canonical.ts # JSONL + SQLite → canonical_upload_staging writer
+│   │   ├── build-payloads.ts # Staging table → V2 canonical push payload builders
+│   │   ├── client.ts        # HTTP upload client with Bearer auth (never throws)
+│   │   └── flush.ts         # Queue flush with exponential backoff (409=success, 401/403=non-retryable)
 │   ├── contribute/          # Opt-in anonymized data export (M7)
 │   │   ├── bundle.ts        # Bundle assembler
 │   │   ├── sanitize.ts      # Privacy sanitization (conservative/aggressive)
 │   │   └── contribute.ts    # CLI entry point + GitHub submission
-│   ├── observability.ts     # Health checks, log integrity
+│   ├── observability.ts     # Health checks, log integrity, alpha queue health
 │   ├── status.ts            # Skill health summary (M6)
 │   ├── last.ts              # Last session insight (M6)
 │   └── workflows/           # Workflow discovery and persistence
@@ -98,9 +107,15 @@ selftune/
 │   └── src/hooks/           # Data-fetching hooks against dashboard-server
 ├── bin/                     # npm/node CLI entry point
 │   └── selftune.cjs
-├── skill/                   # Agent-facing selftune skill
-│   ├── SKILL.md             # Skill definition
+├── skill/                   # Agent-facing selftune skill (self-contained)
+│   ├── SKILL.md             # Skill definition + routing
 │   ├── settings_snippet.json
+│   ├── agents/              # Specialized subagents (bundled, copied to ~/.claude/agents/ on init)
+│   │   ├── diagnosis-analyst.md
+│   │   ├── evolution-reviewer.md
+│   │   ├── integration-guide.md
+│   │   └── pattern-analyst.md
+│   ├── assets/              # Config templates (activation rules, settings)
 │   ├── Workflows/           # Skill workflow routing docs
 │   │   ├── Contribute.md
 │   │   ├── Cron.md
@@ -120,6 +135,7 @@ selftune/
 │   │   └── Watch.md
 │   └── references/
 │       ├── grading-methodology.md
+│       ├── interactive-config.md
 │       ├── invocation-taxonomy.md
 │       └── logs.md
 ├── tests/                   # Test suite (bun test)
@@ -174,7 +190,7 @@ This prevents stale docs and broken contracts.
 | Dashboard contract (`dashboard-contract.ts`) | `apps/local-dashboard/src/types.ts`, dashboard components that consume the changed fields |
 | Hook behavior (`hooks/*.ts`) | `skill/Workflows/Initialize.md` hook table, `skill/settings_snippet.json` |
 | Orchestrate behavior | `skill/Workflows/Orchestrate.md`, `ARCHITECTURE.md` operating modes |
-| Agent files (`.claude/agents/*.md`) | `skill/SKILL.md` Specialized Agents table |
+| Agent files (`skill/agents/*.md`) | `skill/SKILL.md` Specialized Agents table |
 | New workflow file | `skill/SKILL.md` Workflow Routing table + Resource Index |
 | Evolution pipeline changes | `skill/Workflows/Evolve.md`, `docs/design-docs/evolution-pipeline.md` |
 | Platform adapter (ingestor) changes | `skill/Workflows/Ingest.md`, `README.md` Platforms section |

ARCHITECTURE.md

@@ -1,4 +1,4 @@
-<!-- Verified: 2026-03-15 -->
+<!-- Verified: 2026-03-19 -->
 
 # Architecture — selftune
 
@@ -50,6 +50,12 @@ flowchart LR
   SQLite -. WAL watch .-> API
   API -. SSE push .-> SPA[apps/local-dashboard]
   API --> CLI[status / last / badge]
+
+  SQLite -. alpha enrolled .-> AlphaUpload[alpha-upload pipeline]
+  AlphaUpload --> Queue[(upload_queue table)]
+  Queue --> Flush[flush + retry]
+  Flush --> CloudAPI[cloud API — POST /api/v1/push]
+  CloudAPI --> Postgres[(Neon Postgres — canonical tables)]
 ```
 
 ## Operating Rules
@@ -58,6 +64,7 @@ flowchart LR
 - **Shared local evidence.** Downstream modules communicate through SQLite (primary operational store), append-only JSONL audit trails, and repaired overlays.
 - **Autonomy with safeguards.** Low-risk description evolution can deploy automatically, but validation, watch, and rollback remain mandatory.
 - **Local-first product surfaces.** `status`, `last`, and the dashboard read from local evidence, not external services.
+- **Alpha data pipeline.** Opted-in users upload V2 canonical push payloads to the cloud API via `alpha-upload/`. Uploads are fail-open and never block the orchestrate loop.
 - **Generic scheduling first.** `selftune cron setup` is the main automation path (auto-detects platform). `selftune schedule` is a backward-compatible alias.
 
 ## Domain Map
@@ -78,6 +85,7 @@ flowchart LR
 | Local DB | `cli/selftune/localdb/` | SQLite materialization and payload-oriented queries | B |
 | Dashboard | `cli/selftune/dashboard.ts`, `cli/selftune/dashboard-server.ts`, `apps/local-dashboard/` | Local SPA shell, v2 API with SSE live updates, overview/report/status UI | B |
 | Observability CLI | `cli/selftune/status.ts`, `cli/selftune/last.ts`, `cli/selftune/badge/` | Fast local readouts of health, recent activity, and badge state | B |
+| Alpha Upload | `cli/selftune/alpha-upload/`, `cli/selftune/alpha-identity.ts` | Alpha data pipeline: queue, V2 payload build, flush, HTTP transport with API key auth | B |
 | Contribute | `cli/selftune/contribute/` | Opt-in anonymized export for community signal pooling | C |
 | Skill | `skill/` | Agent-facing routing table, workflows, and references | B |
 
@@ -164,16 +172,17 @@ don't need agent intelligence or user interaction.
 
 ## Data Architecture
 
-SQLite is the operational database for all reads. Hooks and sync write
+SQLite is the operational database for local runtime reads. Hooks and sync write
 directly to SQLite via `localdb/direct-write.ts`. JSONL files are retained
-as an append-only audit trail and can be used to rebuild SQLite on demand.
+as append-only capture/audit material and can still be used for rebuild/export
+paths while the migration is being closed.
 
 ```text
 Primary Store: SQLite (~/.selftune/selftune.db)
 ├── Hooks write directly via localdb/direct-write.ts (primary write path)
 ├── Sync writes directly via localdb/direct-write.ts
 ├── All reads (orchestrate, evolve, grade, status, dashboard) query SQLite
-└── WAL-mode watch powers SSE live updates
+└── Target freshness model: WAL-mode watch powers SSE live updates
 
 Audit Trail: JSONL files (~/.claude/*.jsonl)
 ├── session_telemetry_log.jsonl    Session telemetry records
@@ -192,12 +201,23 @@ Core Loop: reads SQLite
 Rebuild Paths:
 ├── materialize.ts  — runs once on startup for historical JSONL backfill
 └── selftune export — generates JSONL from SQLite on demand
+
+Alpha Upload Path (opted-in users only):
+├── stage-canonical.ts  — reads canonical JSONL + evolution evidence + orchestrate_runs into canonical_upload_staging table
+├── build-payloads.ts   — reads staging table via single monotonic cursor, produces V2 canonical push payloads
+├── flush.ts            — POSTs to cloud API (POST /api/v1/push) with Bearer auth, handles 409/401/403
+└── Cloud storage: Neon Postgres (raw_pushes for lossless ingest → canonical tables for analysis)
 ```
 
-Hooks and sync write to both SQLite (primary) and JSONL (audit trail) in
-parallel. All reads go through SQLite. The materializer runs once on startup
-to backfill any historical JSONL data not yet in the database. `selftune export`
-can regenerate JSONL from SQLite when needed for portability or debugging.
+Hooks and sync currently write to both SQLite (primary local runtime store)
+and JSONL (capture/audit trail) in parallel. All local product reads go
+through SQLite. The materializer runs once on startup to backfill any
+historical JSONL data not yet in the database. `selftune export` can
+regenerate JSONL from SQLite when needed for portability or debugging.
+
+Current freshness caveat: the shipped dashboard still uses legacy JSONL file
+watchers for SSE invalidation in `dashboard-server.ts`. WAL-only invalidation
+is the intended end-state, but it is not the sole live-refresh path yet.
 
 ## Repository Shape
 

CONTRIBUTING.md

@@ -147,7 +147,7 @@ When modifying JSONL log schemas or adding new fields, update all of these to ke
 | Symptom | Fix |
 |---------|-----|
 | Dashboard shows stale data | `selftune sync --force` |
-| SQLite schema mismatch after code change | `rm ~/.selftune/selftune.db && selftune sync --force` (materializer rebuilds from JSONL) |
+| SQLite schema mismatch after code change | `selftune export` first, then `rm ~/.selftune/selftune.db && selftune sync --force` |
 | Missing invocations after hook changes | Verify `~/.claude/settings.json` matchers, then `selftune doctor` |
 | Need to backfill from transcripts | `selftune ingest claude --force` |
 

README.md

@@ -125,6 +125,7 @@ Your agent runs these — you just say what you want ("improve my skills", "show
 | **auto** | `selftune cron setup` | Install OS-level scheduling (cron/launchd/systemd) |
 | | `selftune watch --skill <name>` | Monitor after deploy. Auto-rollback on regression. |
 | **other** | `selftune telemetry` | Manage anonymous usage analytics (status, enable, disable) |
+| | `selftune alpha upload` | Run a manual alpha upload cycle and emit a JSON send summary |
 
 Full command reference: `selftune --help`
 

apps/local-dashboard/package.json

@@ -4,7 +4,7 @@
   "version": "0.1.0",
   "type": "module",
   "scripts": {
-    "dev": "concurrently \"cd ../.. && bun --watch run cli/selftune/dashboard-server.ts --port 7888\" \"vite\"",
+    "dev": "concurrently \"cd ../.. && bun --watch run cli/selftune/dashboard-server.ts --port 7888 --runtime-mode dev-server\" \"sh -c 'echo \\\"Waiting for dashboard server on localhost:7888...\\\"; i=0; max=150; until curl -fsS http://localhost:7888/api/health >/dev/null 2>&1; do i=$((i+1)); if [ \\\"$i\\\" -ge \\\"$max\\\" ]; then echo \\\"Dashboard server did not become healthy within 30s\\\"; exit 1; fi; sleep 0.2; done; echo \\\"Dashboard server healthy; starting Vite.\\\"; vite --strictPort'\"",
     "build": "vite build",
     "preview": "vite preview",
     "typecheck": "tsc --noEmit",

apps/local-dashboard/src/App.tsx

@@ -10,6 +10,7 @@ import { Overview } from "@/pages/Overview"
 import { SkillReport } from "@/pages/SkillReport"
 import { Status } from "@/pages/Status"
 import { useOverview } from "@/hooks/useOverview"
+import { RuntimeFooter } from "@/components/runtime-footer"
 import { useSSE } from "@/hooks/useSSE"
 import type { SkillHealthStatus, SkillSummary } from "@/types"
 import { deriveStatus, sortByPassRateAndChecks } from "@selftune/ui/lib"
@@ -90,6 +91,7 @@ function DashboardShell() {
           <Route path="/status" element={<StatusWithHeader />} />
         </Routes>
       </SidebarInset>
+      <RuntimeFooter />
     </SidebarProvider>
   )
 }

apps/local-dashboard/src/components/app-sidebar.tsx

@@ -32,6 +32,7 @@ import {
   GlobeIcon,
   HeartPulseIcon,
   HelpCircleIcon,
+  LayoutDashboardIcon,
   SearchIcon,
   ServerIcon,
   XCircleIcon,
@@ -200,6 +201,24 @@ export function AppSidebar({
           </SidebarGroupContent>
         </SidebarGroup>
 
+        {/* Dashboard */}
+        <SidebarGroup>
+          <SidebarGroupContent>
+            <SidebarMenu>
+              <SidebarMenuItem>
+                <SidebarMenuButton
+                  isActive={location.pathname === "/"}
+                  tooltip="Dashboard"
+                  render={<Link to="/" />}
+                >
+                  <LayoutDashboardIcon className="size-4" />
+                  <span>Dashboard</span>
+                </SidebarMenuButton>
+              </SidebarMenuItem>
+            </SidebarMenu>
+          </SidebarGroupContent>
+        </SidebarGroup>
+
         {/* Skills */}
         <SidebarGroup className="flex-1">
           <SidebarGroupLabel>Skills</SidebarGroupLabel>

apps/local-dashboard/src/components/runtime-footer.tsx

@@ -0,0 +1,59 @@
+import { useEffect, useState } from "react"
+import type { HealthResponse } from "@/types"
+
+function isHealthResponse(value: unknown): value is HealthResponse {
+  if (typeof value !== "object" || value === null) return false
+  const record = value as Record<string, unknown>
+  return (
+    typeof record.workspace_root === "string" &&
+    typeof record.git_sha === "string" &&
+    typeof record.db_path === "string" &&
+    typeof record.process_mode === "string" &&
+    (record.watcher_mode === "jsonl" || record.watcher_mode === "none")
+  )
+}
+
+export function RuntimeFooter() {
+  const [health, setHealth] = useState<HealthResponse | null>(null)
+
+  useEffect(() => {
+    fetch("/api/health")
+      .then((res) => res.json())
+      .then((data: unknown) => {
+        if (isHealthResponse(data)) {
+          setHealth(data)
+        }
+      })
+      .catch(() => {
+        /* non-critical — footer simply stays hidden */
+      })
+  }, [])
+
+  if (!health) return null
+  const legacyWatcherMode = health.watcher_mode === "jsonl"
+
+  return (
+    <footer className="fixed bottom-0 left-0 right-0 z-10 border-t border-border/40 bg-background/80 backdrop-blur-sm px-4 py-1.5">
+      <div className="flex flex-wrap items-center gap-4 text-[11px] font-mono text-muted-foreground">
+        <span title="Workspace root">{health.workspace_root}</span>
+        <span title="Git SHA">{health.git_sha}</span>
+        <span title="Database path">{health.db_path}</span>
+        <span title="Process mode">mode: {health.process_mode}</span>
+        <span
+          title="Watcher mode"
+          className={legacyWatcherMode ? "text-amber-700 dark:text-amber-300" : undefined}
+        >
+          watcher: {health.watcher_mode}
+        </span>
+        {legacyWatcherMode && (
+          <span
+            className="rounded border border-amber-300/70 bg-amber-500/10 px-2 py-0.5 text-amber-800 dark:border-amber-800 dark:text-amber-300"
+            title="Dashboard reads SQLite, but live invalidation still comes from JSONL log watchers."
+          >
+            warning: legacy JSONL watcher invalidation
+          </span>
+        )}
+      </div>
+    </footer>
+  )
+}

apps/local-dashboard/src/components/ui/sidebar.tsx

@@ -473,7 +473,7 @@ function SidebarMenuItem({ className, ...props }: React.ComponentProps<"li">) {
 }
 
 const sidebarMenuButtonVariants = cva(
-  "peer/menu-button group/menu-button flex w-full items-center gap-2 overflow-hidden rounded-md p-2 text-left text-sm ring-sidebar-ring outline-hidden transition-[width,height,padding] group-has-data-[sidebar=menu-action]/menu-item:pr-8 group-data-[collapsible=icon]:size-8! group-data-[collapsible=icon]:p-2! hover:bg-sidebar-accent hover:text-sidebar-accent-foreground focus-visible:ring-2 active:bg-sidebar-accent active:text-sidebar-accent-foreground disabled:pointer-events-none disabled:opacity-50 aria-disabled:pointer-events-none aria-disabled:opacity-50 data-open:hover:bg-sidebar-accent data-open:hover:text-sidebar-accent-foreground data-active:bg-sidebar-accent data-active:font-medium data-active:text-sidebar-accent-foreground [&_svg]:size-4 [&_svg]:shrink-0 [&>span:last-child]:truncate",
+  "peer/menu-button group/menu-button flex w-full items-center gap-2 overflow-hidden rounded-md p-2 text-left text-sm ring-sidebar-ring outline-hidden transition-[width,height,padding] group-has-data-[sidebar=menu-action]/menu-item:pr-8 group-data-[collapsible=icon]:size-8! group-data-[collapsible=icon]:p-2! hover:bg-sidebar-accent hover:text-sidebar-accent-foreground focus-visible:ring-2 active:bg-sidebar-accent active:text-sidebar-accent-foreground disabled:pointer-events-none disabled:opacity-50 aria-disabled:pointer-events-none aria-disabled:opacity-50 data-open:hover:bg-sidebar-accent data-open:hover:text-sidebar-accent-foreground data-active:bg-sidebar-accent data-active:font-medium data-active:text-sidebar-accent-foreground border-l-2 border-transparent data-active:border-primary data-active:rounded-l-none [&_svg]:size-4 [&_svg]:shrink-0 [&>span:last-child]:truncate",
   {
     variants: {
       variant: {
@@ -677,7 +677,7 @@ function SidebarMenuSubButton({
     props: mergeProps<"a">(
       {
         className: cn(
-          "flex h-7 min-w-0 -translate-x-px items-center gap-2 overflow-hidden rounded-md px-2 text-sidebar-foreground ring-sidebar-ring outline-hidden group-data-[collapsible=icon]:hidden hover:bg-sidebar-accent hover:text-sidebar-accent-foreground focus-visible:ring-2 active:bg-sidebar-accent active:text-sidebar-accent-foreground disabled:pointer-events-none disabled:opacity-50 aria-disabled:pointer-events-none aria-disabled:opacity-50 data-[size=md]:text-sm data-[size=sm]:text-xs data-active:bg-sidebar-accent data-active:text-sidebar-accent-foreground [&>span:last-child]:truncate [&>svg]:size-4 [&>svg]:shrink-0 [&>svg]:text-sidebar-accent-foreground",
+          "flex h-7 min-w-0 -translate-x-px items-center gap-2 overflow-hidden rounded-md px-2 text-sidebar-foreground ring-sidebar-ring outline-hidden group-data-[collapsible=icon]:hidden hover:bg-sidebar-accent hover:text-sidebar-accent-foreground focus-visible:ring-2 active:bg-sidebar-accent active:text-sidebar-accent-foreground disabled:pointer-events-none disabled:opacity-50 aria-disabled:pointer-events-none aria-disabled:opacity-50 data-[size=md]:text-sm data-[size=sm]:text-xs data-active:bg-sidebar-accent data-active:font-medium data-active:text-sidebar-accent-foreground border-l-2 border-transparent data-active:border-primary data-active:rounded-l-none [&>span:last-child]:truncate [&>svg]:size-4 [&>svg]:shrink-0 [&>svg]:text-sidebar-accent-foreground",
           className
         ),
       },