feat(storage): move runtime state to SQLite

ARCHITECTURE.md

@@ -49,11 +49,11 @@ that answer, and delivers it.
 │                    │ worker queues │                                    │
 │                    └───────┬───────┘                                    │
 │                            ▼                                            │
-│  ┌──────────────┐  ┌───────────────┐  ┌───────────────────────────────┐ │
-│  │ state.json   │  │ push.db       │  │ audit.jsonl                   │ │
-│  │ cursors and  │  │ conversations │  │ redacted operational events   │ │
-│  │ sessions     │  │ jobs/delivery │  │                               │ │
-│  └──────────────┘  └───────────────┘  └───────────────────────────────┘ │
+│  ┌───────────────────────────────────────┐  ┌──────────────────────────┐ │
+│  │ push.db                               │  │ audit.jsonl              │ │
+│  │ conversations, jobs, delivery,        │  │ redacted operational     │ │
+│  │ channel cursors, backend sessions     │  │ events                   │ │
+│  └───────────────────────────────────────┘  └──────────────────────────┘ │
 └────────────────────────────────────┬─────────────────────────────────────┘
                                      │ subprocess
                                      ▼
@@ -174,14 +174,19 @@ Push uses separate stores because they have different update and query needs:
 
 | Store | Purpose |
 | --- | --- |
-| `$PUSH_HOME/state.json` | small atomically replaced map of channel cursors and backend session IDs |
-| `$PUSH_HOME/push.db` | transactional conversation, delivery, approval compatibility, and job-run history |
+| `$PUSH_HOME/state.json` | retained legacy cursor and session source for one-time migration |
+| `$PUSH_HOME/push.db` | transactional conversation, delivery, approval compatibility, job-run history, channel cursors, and channel-qualified backend sessions |
 | `$PUSH_HOME/state.json.slack-inbox.db` | Slack Socket Mode inbox committed before envelope acknowledgement |
 | `$PUSH_HOME/audit.jsonl` | append-only operational audit events |
 | `$PUSH_HOME/run/` | advisory job locks |
 | `$PUSH_HOME/cache/` | disposable agent handoff files |
 | assistant Git repository | user-owned identity, context, evals, jobs, and project resources |
 
+`state_path` remains a compatibility input for one-time migration. Push never
+uses the legacy JSON file as a live source of truth after its migration marker
+commits. The Slack inbox remains separate because its rows are an
+acknowledgement queue, not gateway cursor or session state.
+
 Runtime databases, secrets, locks, sessions, and logs must stay outside the
 assistant repository.
 
@@ -226,9 +231,11 @@ The gateway runs the required preflight checks before starting.
 
 ### Step 3: Build Shared State
 
-`GatewayGroup::new` opens one shared `Store`, one shared `History`, one runner
-per required backend, and one serialized audit-log lock. It then creates one
-`Gateway` for each enabled channel.
+`GatewayGroup::new` opens one shared transactional `Store` and one shared
+`History` connection to `push.db`, one runner per required backend, and one
+serialized audit-log lock. Opening the store also performs any one-time legacy
+`state.json` migration before polling can begin. It then creates one `Gateway`
+for each enabled channel.
 
 Each gateway owns its own:
 
@@ -517,9 +524,16 @@ configured, unattended, and evaluator modes.
 
 ## 7. Durable Data Model
 
-### `state.json`
+### Runtime State in `push.db`
 
-[`src/store.rs`](src/store.rs) owns a small JSON document:
+[`src/store.rs`](src/store.rs) owns channel cursors and backend session
+mappings in the canonical SQLite database. Cursor advancement uses a
+monotonic upsert, so concurrent or repeated writes cannot move a channel
+backwards. Session reads, backend changes, backend-owned ID updates, and
+`/clear` rotation use transactions keyed by channel and thread.
+
+The first startup after upgrade imports the legacy document configured by
+`state_path`:
 
 ```json
 {
@@ -539,9 +553,18 @@ configured, unattended, and evaluator modes.
 }
 ```
 
-`last_row_id` and the field name `uuid` remain for backward compatibility.
-Writes use a new owner-only temporary file followed by atomic rename. In-memory
-state is rolled back if persistence fails.
+`last_row_id` remains an iMessage fallback only when the JSON has no explicit
+`cursors.imessage` value. Unqualified session keys are imported as iMessage
+keys; an explicit channel-qualified key wins when both forms exist. Empty
+session IDs are preserved so the next request starts fresh.
+
+The imported cursor/session rows and a source-path migration marker commit in
+one immediate transaction. A crash or error before commit leaves no partial
+import, and startup retries from the unchanged JSON file. After commit, later
+starts ignore that source even though Push deliberately leaves it in place as
+a private recovery copy. Restoring a pre-migration database, or removing the
+database after preserving other needed data, allows the retained file to be
+imported again. Corrupt JSON stops startup with recovery guidance.
 
 ### `push.db`
 
@@ -556,6 +579,9 @@ timeout.
 | `approval_questions` | retained durable question and answer state |
 | `job_runs` | immutable job claims, bounded results, evaluation, scheduling, and delivery |
 | `gateway_control_actions` | idempotent control actions such as `/stop` targets |
+| `channel_cursors` | monotonic per-channel polling checkpoints |
+| `backend_sessions` | current backend session for each channel and thread |
+| `legacy_state_migrations` | atomic record of each imported legacy JSON source |
 
 Important constraints:
 
@@ -807,7 +833,7 @@ result before proactive delivery.
 | [`src/claude.rs`](src/claude.rs) | Claude Code CLI adapter |
 | [`src/codex.rs`](src/codex.rs) | Codex CLI adapter |
 | [`src/pi.rs`](src/pi.rs) | Pi CLI adapter |
-| [`src/store.rs`](src/store.rs) | atomic cursor and session state |
+| [`src/store.rs`](src/store.rs) | transactional SQLite cursor/session state and legacy JSON migration |
 | [`src/history.rs`](src/history.rs) | SQLite schema, conversation history, delivery, and migrations |
 | [`src/jobs.rs`](src/jobs.rs) | runbook validation, execution, evaluation, scheduler, and ledger |
 | [`src/audit.rs`](src/audit.rs) | redacted JSONL audit log |

docs/channels/imessage.md

@@ -75,12 +75,13 @@ See [configuration](../configuration.md#routing) for route precedence.
 
 On the first iMessage start, Push records the newest existing Messages row
 without running it. Send a new message after the gateway starts. Later starts
-continue after the last completed row stored in `state.json`.
+continue after the last completed row stored in `push.db`.
 
-Push stores the last completed Messages row in `state.json` and accepted
-conversation turns in `push.db`. It advances the cursor only after a row is
-ignored or completed. An earlier in-flight row prevents later completed rows
-from pushing the cursor past it.
+Push stores both the last completed Messages row and accepted conversation
+turns in `push.db`. It advances the cursor only after a row is ignored or
+completed. An earlier in-flight row prevents later completed rows from pushing
+the cursor past it. Existing `state.json` data is imported once on upgrade and
+the original file remains as a recovery copy.
 
 If a generated outbound reply was stored before a crash, restart delivers the
 stored reply without generating a different second answer. A crash before the

docs/configuration.md

@@ -256,8 +256,8 @@ requests. Review [permissions and security](security.md) before enabling jobs.
 | Setting | Default | Purpose |
 | --- | --- | --- |
 | `assistant_root` | required for new setups | Canonical root of the one assistant repository; `SOUL.md`, `context/`, `evals/`, and `jobs/` are derived |
-| `state_path` | `$PUSH_HOME/state.json` | Channel cursors and backend session IDs |
-| `database_path` | `$PUSH_HOME/push.db` | Canonical conversation, approval, and job history |
+| `state_path` | `$PUSH_HOME/state.json` | Legacy JSON source retained for one-time cursor and session migration |
+| `database_path` | `$PUSH_HOME/push.db` | Canonical history, jobs, delivery, channel cursors, and backend sessions |
 | `audit_log_path` | `$PUSH_HOME/audit.jsonl` | Structured local audit log |
 | `audit_log_content` | `false` | Include message and reply content in audit events |
 
@@ -325,6 +325,13 @@ jobs path is exactly `<assistant_dir>/jobs`. For separate legacy paths, move
 
 Explicit `state_path`, `database_path`, `audit_log_path`, and `jobs_run_dir`
 settings remain supported for the rest of the 0.x release line. Push does not
-move or rewrite data at those paths. A future removal would require an
-announced major release and migration instructions. New installations should
-omit them and let `PUSH_HOME` own the whole runtime layout.
+move or rewrite data at the database, audit, or run-lock paths. On startup,
+Push imports an existing JSON file at `state_path` into `database_path` once,
+records the import in the same SQLite transaction, and leaves the JSON
+untouched as a recovery copy. After that commit, Push reads and writes cursor
+and backend-session state only in SQLite. Do not edit the retained JSON
+expecting live state to change.
+
+A future removal of these settings would require an announced major release
+and migration instructions. New installations should omit them and let
+`PUSH_HOME` own the whole runtime layout.

docs/prd.md

@@ -161,10 +161,10 @@ user prompt.
 | `telegram.bot_token` | Telegram bot token stored in the private config. |
 | `telegram.allow_user_ids` | Allowed private-chat sender IDs. |
 | `telegram.allow_chat_ids` | Allowed private chat IDs. |
-| `state_path` | JSON state path. |
+| `state_path` | Legacy JSON cursor/session source retained for one-time migration compatibility. |
 | `audit_log_path` | Local JSONL audit log path. |
 | `audit_log_content` | Whether audit events include message and reply text. |
-| `database_path` | Canonical SQLite history path; defaults to `$PUSH_HOME/push.db`. |
+| `database_path` | Canonical SQLite history, job, delivery, cursor, and backend-session path; defaults to `$PUSH_HOME/push.db`. |
 
 ## Control Commands
 

docs/security.md

@@ -62,8 +62,8 @@ service environment using the narrowest policy that works.
 | --- | --- |
 | `$PUSH_HOME/config.toml` | allowlists, routes, paths, and possibly credentials |
 | `<assistant_root>/` | Git-versioned identity, context, evals, jobs, and optional project skills |
-| `$PUSH_HOME/state.json` | channel cursors and backend session IDs |
-| `$PUSH_HOME/push.db` | conversation history, approvals, and job runs |
+| `$PUSH_HOME/state.json` | retained legacy cursor and session recovery copy after migration |
+| `$PUSH_HOME/push.db` | conversation history, approvals, jobs, delivery, channel cursors, and backend sessions |
 | `$PUSH_HOME/state.json.slack-inbox.db` | Slack envelopes accepted before gateway processing |
 | `$PUSH_HOME/audit.jsonl` | metadata, errors, handles, and optional content |
 | `$PUSH_HOME/run/` | local job lock files |
@@ -110,6 +110,12 @@ error, and character counts. Message and reply content are omitted unless
 The redacted log is still sensitive because it can contain handles, thread
 IDs, file paths, and backend errors. Protect and rotate it like a service log.
 
+The legacy state migration repairs the JSON file to owner-only permissions and
+does not delete or rewrite it. Keep that recovery copy private until your
+normal encrypted backup process has captured the migrated `push.db`. The
+database is the live source of truth after migration, so backing up only
+`state.json` is not sufficient.
+
 ## Deployment checklist
 
 - allow only identities you control

docs/services.md

@@ -22,6 +22,11 @@ Set one absolute `PUSH_HOME` in the service definition. It defaults to
 `~/.push` for interactive commands. The service user needs:
 
 - read and write access to `PUSH_HOME`
+- access to the configured `config.toml`
+- read access and owner control of an existing legacy `state_path` for migration
+- write access to `audit_log_path`
+- write access to `database_path`
+- write access to `jobs_run_dir`
 - filesystem access to `assistant_root` as allowed by the selected agent
 - agent write access to `assistant_root/jobs/` when jobs should be created from chat
 - access to the selected `claude`, `codex`, or `pi` executable on `PATH`
@@ -35,12 +40,14 @@ Set one absolute `PUSH_HOME` in the service definition. It defaults to
   `OPENAI_API_KEY` in the service environment, plus network access to
   `api.openai.com`
 
-`$PUSH_HOME/state.json` stores independent cursors for each channel and backend
-session IDs. `$PUSH_HOME/push.db` stores the canonical conversation and job
-journal. The audit log, Slack recovery inbox, job locks, and cache are derived
-from the same root. Chat agents run from `assistant_root`. Keep `PUSH_HOME` on
-durable storage. Restarting the service resumes after the last completed row
-and reuses existing backend sessions when the backend for that thread has not
+`database_path` stores the canonical conversation journal, channel cursors, and
+backend session mappings. `state_path` is only a legacy JSON migration source
+and retained recovery copy; Push does not write live state to it. The audit
+log, Slack recovery inbox, job locks, and cache remain separate paths derived
+from `PUSH_HOME`, unless their documented compatibility settings override
+them. Chat agents run from `assistant_root`. Keep these paths on durable
+storage. Restarting the service resumes after the last completed row and
+reuses existing backend sessions when the backend for that thread has not
 changed.
 
 Keep `assistant_root` in its own Git repository. Keep config secrets, state,
@@ -209,6 +216,24 @@ pending result delivery; it does not catch up missed cron times or rerun
 interrupted agent execution. Use `push job runs` to distinguish execution state
 from delivery attempts.
 
+## Backup and state migration recovery
+
+Stop the managed service before taking a filesystem copy of `push.db`, or use
+SQLite's online backup tooling. The database contains conversation history,
+job and delivery state, channel cursors, and backend session mappings. Back up
+the audit log and assistant repository separately. Slack's durable inbox stays
+in `<state_path>.slack-inbox.db`; include it when preserving unprocessed Slack
+events.
+
+After an upgrade, Push imports an existing configured `state_path` in one
+transaction and leaves that JSON file unchanged. Keep it as a private recovery
+copy until a verified database backup exists. If migration fails, fix or
+restore the JSON and restart; Push will not poll while the import is
+incomplete. If the post-migration database is lost, restore `push.db` from
+backup. As a last resort, move the unusable database aside and restart with the
+retained JSON to recover its older cursors and sessions, understanding that
+conversation, job, and delivery records not present in JSON will be absent.
+
 ## Agent-created jobs
 
 When asked, the agent writes jobs directly under `<assistant_root>/jobs` and

docs/telegram.md

@@ -120,16 +120,17 @@ state.
 
 ## Cursor and Restart Behavior
 
-`state.json` stores independent `imessage` and `telegram` cursors. `push.db`
-stores channel-qualified canonical conversations, so Telegram and iMessage
-history cannot collide. On the first
+`push.db` stores independent `imessage` and `telegram` cursors plus
+channel-qualified canonical conversations and backend sessions, so Telegram
+and iMessage state cannot collide. On the first
 Telegram start, Push asks Telegram for the newest pending update and records its
 id without running it. This explicit backlog skip prevents old bot messages
 from unexpectedly starting agent work. Later accepted and ignored updates
 advance only the Telegram cursor. Restarts continue from the next update id.
 
 As with iMessage, a crash after delivery but before cursor persistence can
-repeat a reply. Keep `state.json` on durable storage.
+repeat a reply. Keep `push.db` on durable storage. Existing `state.json` data
+is imported once and retained unchanged as a recovery copy.
 
 ## Linux and Service Mode
 
@@ -142,8 +143,7 @@ directly in a world-readable unit file.
 Protect these files as credentials or private assistant data:
 
 - the bot token and configuration
-- `state.json` and the audit log
-- `push.db`
+- `push.db`, the retained legacy `state.json`, and the audit log
 - the private `assistant_root` repository, including identity, context, jobs,
   and optional project skills
 

src/doctor.rs

@@ -5,7 +5,7 @@ use std::path::{Path, PathBuf};
 
 use anyhow::{bail, Result};
 
-use crate::{channel::Channel, config, history, jobs};
+use crate::{channel::Channel, config, jobs};
 use config::{SLACK_APP_TOKEN_ENV, SLACK_BOT_TOKEN_ENV, TELEGRAM_BOT_TOKEN_ENV};
 
 /// Fails fast with actionable messages when the environment is not ready.
@@ -256,16 +256,20 @@ fn check_writable_dir(name: &str, field: &str, dir: &Path, checks: &mut Vec<Chec
 }
 
 fn check_history_database(cfg: &config::Config, checks: &mut Vec<Check>) {
-    match history::History::open(&cfg.paths.database) {
+    match crate::store::Store::open(&cfg.paths) {
         Ok(_) => checks.push(Check::pass(
             "conversation database",
-            format!("{} is ready", cfg.paths.database.display()),
+            format!(
+                "{} is ready for history and runtime state",
+                cfg.paths.database.display()
+            ),
         )),
         Err(error) => checks.push(Check::fail(
             "conversation database",
             format!(
-                "cannot open {}: {error}. Choose a writable database_path and repair or remove an invalid database.",
-                cfg.paths.database.display()
+                "cannot prepare {} or migrate legacy state from {}: {error:#}. Repair the database or legacy JSON before starting Push.",
+                cfg.paths.database.display(),
+                cfg.paths.state.display()
             ),
         )),
     }