Log and commit

RALPH-METHODOLOGY-ASSESSMENT.md

@@ -272,7 +272,7 @@ underlying content explicitly as `{{base_prompt}}` when a prompt template is
 used, so template-mode output includes the wrapped prompt body rather than
 silently dropping it. `render()` also applies `promptTemplate` substitution for
 `{{iteration}}`, `{{max_iterations}}`, `{{tasks}}`, `{{task_context}}`,
-`{{task_promise}}`, `{{completion_promise}}`, and `{{context}}`.
+`{{task_promise}}`, and `{{completion_promise}}`.
 `scripts/ralph-run.sh:735-843` — `create_prompt_template()` writes
 `prompt-template.md` embedding all template variables, including an explicit
 invocation-time PRD snapshot section.
@@ -628,7 +628,7 @@ rendering that injects iteration-specific values before invoking OpenCode.
 | Field | Value |
 |-------|-------|
 | Verdict | `verified` — `prompt.js` supports both inline text and file-based prompt loading, and applies full template variable substitution on every iteration. `ralph-run.sh` creates a dedicated `prompt-template.md` and passes both files to the CLI. Confirmed by unit tests for all template variables and bash tests for template construction. Confidence: **high**. |
-| Implementation evidence | `lib/mini-ralph/prompt.js:33-53` — `loadBase()` supports both `promptText` (inline) and `promptFile` (file read); `prompt.js:64-101` — `render()` applies `promptTemplate` when present, replacing `{{iteration}}`, `{{max_iterations}}`, `{{tasks}}`, `{{task_context}}`, `{{task_promise}}`, `{{completion_promise}}`, `{{context}}`; `prompt.js:110-113` — `_renderTemplate()` does `{{key}}` replacement; `scripts/ralph-run.sh:735-843` — `create_prompt_template()` writes `prompt-template.md` with all template variables; `ralph-run.sh:1007-1013` — `--prompt-file PRD.md --prompt-template prompt-template.md` passed to mini-ralph |
+| Implementation evidence | `lib/mini-ralph/prompt.js:33-53` — `loadBase()` supports both `promptText` (inline) and `promptFile` (file read); lazy-loaded only when the template contains `{{base_prompt}}`; `prompt.js:64-101` — `render()` applies `promptTemplate` when present, replacing `{{iteration}}`, `{{max_iterations}}`, `{{tasks}}`, `{{task_context}}`, `{{task_promise}}`, `{{completion_promise}}`; `prompt.js:110-113` — `_renderTemplate()` does `{{key}}` replacement; `scripts/ralph-run.sh:735-843` — `create_prompt_template()` writes `prompt-template.md` with all template variables; `ralph-run.sh:1007-1013` — `--prompt-file PRD.md --prompt-template prompt-template.md` passed to mini-ralph |
 | Test evidence | `tests/unit/javascript/mini-ralph-prompt.test.js` — `_renderTemplate()` suite (lines 30–69): `replaces a single variable` (line 31), `replaces multiple variables` (line 35), `replaces repeated occurrences of the same variable` (line 40), `handles empty string as variable value` (line 56); `render()` suite — `renders template with iteration variables` (line 110), `injects tasks content when tasksFile is present` (line 131), `includes task_promise and completion_promise in template` (line 185); `tests/unit/bash/test-create-prompt-template.bats` — `create_prompt_template: includes Ralph iteration placeholders` (line 57), `create_prompt_template: includes task list placeholder` (line 78), `create_prompt_template: includes context placeholder` (line 98), `create_prompt_template: includes promise placeholders` (line 138); `tests/unit/bash/test-execute-ralph-loop.bats` — `execute_ralph_loop: passes --prompt-template to the CLI` (line 251) |
 
 ---

README.md

@@ -456,6 +456,59 @@ Windows is not currently part of the supported runtime contract.
 
 For common issues and solutions, see [QUICKSTART.md#troubleshooting](./QUICKSTART.md#troubleshooting).
 
+## Environment Variables
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `RALPH_BASE_PROMPT_WARN_BYTES` | `4096` | Byte threshold above which `render()` emits a one-line warning to stderr when `{{base_prompt}}` resolves to a large file. Set to `0` to silence warnings entirely. Invalid values fall back to `4096` with a one-time notice per process. |
+| `RALPH_ITERATION_IDLE_TIMEOUT_MS` | `300000` | Milliseconds of silence on stdout+stderr before the per-iteration idle watchdog fires. Set to `0` to disable the watchdog entirely and restore pre-change behavior (no timeout). |
+| `RALPH_ITERATION_KILL_GRACE_MS` | `10000` | Milliseconds the runner waits after sending `SIGTERM` to a timed-out iteration child before escalating to `SIGKILL`. |
+
+### Auto-commit ignore-filter surfacing and iteration watchdog
+
+This section covers two surfacing improvements added on top of the `harden-auto-commit-against-ignored-paths` change, which is the underlying mechanism that _detects_ when `.gitignore` rules filter out loop-managed paths.
+
+**No new CLI flags are introduced by this change. No startup behavior changes. Every existing `ralph-run` invocation continues to work unchanged.**
+
+#### Loud stderr block
+
+When `_autoCommit` detects that one or more paths were filtered by `.gitignore` (anomaly types `paths_ignored_filtered` or `all_paths_ignored`), the runner emits the following block directly to `process.stderr` on **every** iteration where the anomaly fires — independently of any reporter buffering or deduplication:
+
+```
+================================================================================
+⚠ AUTO-COMMIT IGNORE FILTER FIRED  (iteration 7, type: paths_ignored_filtered)
+Paths filtered because .gitignore matches:
+  - openspec/changes/my-change/tasks.md
+  - openspec/changes/my-change/proposal.md
+Consequence: these paths are NOT in the latest commit.
+Remediation (pick one):
+  1. git add -f <path>   # one-time unblock, if you want it tracked
+  2. edit .gitignore     # narrow or remove the matching rule
+  3. pass --no-auto-commit on the ralph-run invocation
+================================================================================
+```
+
+The three remediation options mean:
+
+1. **`git add -f <path>`** — force-stage a specific file for the next commit. One-time unblock; the path stays gitignored and will be filtered again on the next auto-commit unless you also do option 2.
+2. **edit `.gitignore`** — narrow or remove the matching rule so the path is no longer excluded. The safest long-term fix when the rule is too broad.
+3. **`--no-auto-commit`** — disable auto-commit for this run entirely. Use when you want to manage commits yourself and don't want the runner touching git.
+
+#### Iteration watchdog
+
+The runner enforces a per-iteration idle timeout: if the `opencode run` subprocess produces no new bytes on stdout **or** stderr for `RALPH_ITERATION_IDLE_TIMEOUT_MS` milliseconds, the watchdog fires. It sends `SIGTERM`, waits up to `RALPH_ITERATION_KILL_GRACE_MS` for a graceful exit, then sends `SIGKILL`.
+
+The timed-out iteration is recorded in history with `failureReason: 'iteration_timeout_idle'` and three diagnostic fields: `idleMs` (how long the process was silent), `lastStdoutBytes` (last ≤200 bytes of stdout), and `lastStderrBytes` (last ≤200 bytes of stderr). These fields are absent on entries where the watchdog did not fire.
+
+The `iteration_timeout_idle` reason also appears in the `## Recent Loop Signals` block injected into each iteration's prompt, giving the agent visibility into prior timeout events.
+
+Set `RALPH_ITERATION_IDLE_TIMEOUT_MS=0` to disable the watchdog if your agent workflow runs legitimately long silent tools (e.g., large integration test suites). Example:
+
+```bash
+RALPH_ITERATION_IDLE_TIMEOUT_MS=900000 ralph-run --change my-feature   # 15-minute idle threshold
+RALPH_ITERATION_IDLE_TIMEOUT_MS=0 ralph-run --change my-feature        # watchdog disabled
+```
+
 **Quick fixes:**
 
 ```bash

lib/mini-ralph/history.js

@@ -52,6 +52,43 @@ function read(ralphDir) {
  * @param {Array}  entry.toolUsage         - Tool usage summary array
  * @param {Array}  entry.filesChanged      - Files changed in this iteration
  * @param {number} entry.exitCode          - OpenCode exit code
+ * @param {number} [entry.promptBytes]     - UTF-8 byte length of the assembled prompt
+ * @param {number} [entry.promptChars]     - Character length of the assembled prompt
+ * @param {number} [entry.promptTokens]    - Estimated token count for the prompt (chars/4, rounded)
+ * @param {number} [entry.responseBytes]   - UTF-8 byte length of the raw response
+ * @param {number} [entry.responseChars]   - Character length of the raw response
+ * @param {number} [entry.responseTokens]  - Estimated token count for the response (chars/4, rounded)
+ * @param {boolean} [entry.truncated]      - Whether the response was truncated by the invoker
+ * @param {boolean} [entry.commitAttempted]  - Whether an auto-commit was attempted this iteration
+ * @param {boolean} [entry.commitCreated]    - Whether a git commit was successfully created
+ * @param {string}  [entry.commitSha]        - The SHA of the created commit (if commitCreated is true)
+ * @param {string}  [entry.commitMessage]    - The commit message used (if commitCreated is true)
+ * @param {string}  [entry.commitAnomaly]    - Human-readable description of any commit anomaly
+ * @param {string}  [entry.commitAnomalyType] - Machine-readable anomaly type string. Known values:
+ *   - `'nothing_staged'`          - No files were staged for commit.
+ *   - `'commit_failed'`           - The git commit command failed.
+ *   - `'paths_ignored_filtered'`  - Some staged paths were gitignored and filtered out;
+ *                                   the remaining paths were committed successfully.
+ *   - `'all_paths_ignored'`       - Every staged path was gitignored; no commit was made.
+ * @param {string[]} [entry.ignoredPaths]    - Paths that were dropped by the gitignore filter.
+ *   Present only when `commitAnomalyType` is `'paths_ignored_filtered'` or `'all_paths_ignored'`.
+ *   Omitted entirely when no paths were filtered.
+ * @param {string}  [entry.failureReason]    - Human-readable reason for iteration failure.
+ *   Known values include:
+ *   - `'iteration_timeout_idle'`  - The iteration subprocess was terminated by the idle watchdog
+ *                                   because no bytes were written to stdout or stderr for longer
+ *                                   than `RALPH_ITERATION_IDLE_TIMEOUT_MS`. When this value is
+ *                                   present, `idleMs`, `lastStdoutBytes`, and `lastStderrBytes`
+ *                                   are also present on the entry.
+ *   Omitted entirely when the iteration succeeded or failed for another reason.
+ * @param {number}  [entry.idleMs]           - Observed idle duration in milliseconds when the
+ *   watchdog fired. Present only when `failureReason === 'iteration_timeout_idle'`.
+ * @param {string}  [entry.lastStdoutBytes]  - Tail of the iteration subprocess stdout at the
+ *   moment the watchdog fired, capped at 200 bytes. Present only when
+ *   `failureReason === 'iteration_timeout_idle'`.
+ * @param {string}  [entry.lastStderrBytes]  - Tail of the iteration subprocess stderr at the
+ *   moment the watchdog fired, capped at 200 bytes. Present only when
+ *   `failureReason === 'iteration_timeout_idle'`.
  */
 function append(ralphDir, entry) {
   _ensureDir(ralphDir);

lib/mini-ralph/invoker.js

@@ -75,17 +75,43 @@ async function invoke(opts) {
     signal: result.signal,
     toolUsage: _extractToolUsage(result.stdout),
     filesChanged,
+    // Pass through watchdog fields when present (task 2.1)
+    ...(result.failureReason !== undefined && {
+      failureReason: result.failureReason,
+      idleMs: result.idleMs,
+      lastStdoutBytes: result.lastStdoutBytes,
+      lastStderrBytes: result.lastStderrBytes,
+    }),
   };
 }
 
 /**
  * Spawn the opencode process and stream output to terminal while capturing.
+ * Wraps the subprocess with a per-iteration stream-idle watchdog controlled
+ * by RALPH_ITERATION_IDLE_TIMEOUT_MS (default 300000 ms; 0 = disabled) and
+ * RALPH_ITERATION_KILL_GRACE_MS (default 10000 ms).
+ *
+ * When the watchdog fires the returned result gains:
+ *   failureReason: 'iteration_timeout_idle'
+ *   idleMs: <number>          — observed idle duration in ms
+ *   lastStdoutBytes: <string> — tail of stdout, capped at 200 bytes
+ *   lastStderrBytes: <string> — tail of stderr, capped at 200 bytes
+ *
+ * When the watchdog is disabled or does not fire, the return shape is
+ * unchanged from the pre-watchdog contract.
  *
  * @param {Array<string>} args
  * @param {boolean} verbose
- * @returns {Promise<{stdout: string, stderr: string, exitCode: ?number, signal: ?string}>}
+ * @returns {Promise<{stdout: string, stderr: string, exitCode: ?number, signal: ?string, failureReason?: string, idleMs?: number, lastStdoutBytes?: string, lastStderrBytes?: string}>}
  */
 function _spawnOpenCode(args, verbose) {
+  // Parse watchdog knobs from environment. task 2.1 (surface-autocommit-ignore-warning-and-watchdog)
+  const idleTimeoutRaw = process.env.RALPH_ITERATION_IDLE_TIMEOUT_MS;
+  const killGraceRaw   = process.env.RALPH_ITERATION_KILL_GRACE_MS;
+  const idleTimeoutMs  = idleTimeoutRaw !== undefined ? Number(idleTimeoutRaw) : 300000;
+  const killGraceMs    = killGraceRaw   !== undefined ? Number(killGraceRaw)   : 10000;
+  const watchdogEnabled = idleTimeoutMs !== 0;
+
   return new Promise((resolve, reject) => {
     const child = spawn('opencode', args, {
       stdio: ['inherit', 'pipe', 'pipe'],
@@ -94,20 +120,80 @@ function _spawnOpenCode(args, verbose) {
 
     let stdout = '';
     let stderr = '';
+    let watchdogFired = false;
+    let idleTimer = null;
+    let idleStart = Date.now();
+
+    // Tail buffers — keep only the most recent 200 bytes of each stream
+    const TAIL_CAP = 200;
+    let stdoutTail = '';
+    let stderrTail = '';
+
+    function _appendTail(current, chunk) {
+      const combined = current + chunk;
+      return combined.length > TAIL_CAP ? combined.slice(-TAIL_CAP) : combined;
+    }
+
+    function _resetIdleTimer() {
+      if (!watchdogEnabled || watchdogFired) return;
+      if (idleTimer) clearTimeout(idleTimer);
+      idleStart = Date.now();
+      idleTimer = setTimeout(_onIdleTimeout, idleTimeoutMs);
+    }
+
+    function _onIdleTimeout() {
+      if (watchdogFired) return;
+      watchdogFired = true;
+      const observedIdleMs = Date.now() - idleStart;
+
+      // Send SIGTERM first; after killGraceMs escalate to SIGKILL if still alive.
+      // Use process.kill(-pid, signal) with detached:true when process group
+      // kill is needed; for a direct child process.kill() is sufficient on macOS.
+      try {
+        child.kill('SIGTERM');
+      } catch (_) { /* child may have already exited */ }
+
+      const killTimer = setTimeout(() => {
+        try {
+          child.kill('SIGKILL');
+        } catch (_) { /* child may have already exited */ }
+      }, killGraceMs);
+
+      // Prevent killTimer from keeping the event loop alive
+      if (killTimer && typeof killTimer.unref === 'function') {
+        killTimer.unref();
+      }
+
+      // Stash metadata on the child so the close handler can read it
+      child._watchdogMeta = {
+        idleMs: observedIdleMs,
+        lastStdoutBytes: stdoutTail,
+        lastStderrBytes: stderrTail,
+      };
+    }
+
+    if (watchdogEnabled) {
+      _resetIdleTimer();
+    }
 
     child.stdout.on('data', (chunk) => {
       const text = chunk.toString();
       stdout += text;
+      stdoutTail = _appendTail(stdoutTail, text);
       process.stdout.write(chunk);
+      _resetIdleTimer();
     });
 
     child.stderr.on('data', (chunk) => {
       const text = chunk.toString();
       stderr += text;
+      stderrTail = _appendTail(stderrTail, text);
       process.stderr.write(chunk);
+      _resetIdleTimer();
     });
 
     child.on('error', (err) => {
+      if (idleTimer) clearTimeout(idleTimer);
       if (err.code === 'ENOENT') {
         reject(new Error('mini-ralph invoker: opencode CLI not found. Please install opencode: npm install -g opencode-ai'));
       } else {
@@ -116,12 +202,27 @@ function _spawnOpenCode(args, verbose) {
     });
 
     child.on('close', (code, signal) => {
-      resolve({
-        stdout,
-        stderr,
-        exitCode: typeof code === 'number' ? code : null,
-        signal: signal || null,
-      });
+      if (idleTimer) clearTimeout(idleTimer);
+
+      if (watchdogFired && child._watchdogMeta) {
+        resolve({
+          stdout,
+          stderr,
+          exitCode: typeof code === 'number' ? code : null,
+          signal: signal || null,
+          failureReason: 'iteration_timeout_idle',
+          idleMs: child._watchdogMeta.idleMs,
+          lastStdoutBytes: child._watchdogMeta.lastStdoutBytes,
+          lastStderrBytes: child._watchdogMeta.lastStderrBytes,
+        });
+      } else {
+        resolve({
+          stdout,
+          stderr,
+          exitCode: typeof code === 'number' ? code : null,
+          signal: signal || null,
+        });
+      }
     });
   });
 }

lib/mini-ralph/lessons.js

@@ -0,0 +1,93 @@
+'use strict';
+
+const fs = require('fs');
+const nodePath = require('path');
+
+const LESSONS_FILENAME = 'LESSONS.md';
+const MAX_BULLET_CHARS = 120;
+const MAX_INJECT_BULLETS = 15;
+
+/**
+ * Returns the absolute path to the LESSONS.md file for the given ralphDir.
+ * @param {string} ralphDir - Path to the .ralph directory.
+ * @returns {string}
+ */
+function path(ralphDir) {
+  return nodePath.join(ralphDir, LESSONS_FILENAME);
+}
+
+/**
+ * Reads LESSONS.md from ralphDir, returning an array of bullet strings.
+ * Missing file returns []. Blank lines are stripped. Bullets > 120 chars
+ * are truncated and prefixed with 'runner-truncated:'.
+ * @param {string} ralphDir
+ * @returns {string[]}
+ */
+function read(ralphDir) {
+  const filePath = path(ralphDir);
+  let content;
+  try {
+    content = fs.readFileSync(filePath, 'utf8');
+  } catch (e) {
+    if (e.code === 'ENOENT') return [];
+    throw e;
+  }
+
+  const lines = content.split('\n');
+  const bullets = [];
+  for (const line of lines) {
+    const trimmed = line.trim();
+    if (!trimmed) continue;
+    if (trimmed.length > MAX_BULLET_CHARS) {
+      bullets.push('runner-truncated:' + trimmed.slice(0, MAX_BULLET_CHARS));
+    } else {
+      bullets.push(trimmed);
+    }
+  }
+  return bullets;
+}
+
+/**
+ * Returns a markdown section string '## Lessons Learned\n\n<bullets>' using
+ * the last `limit` (default 50) bullets, or '' if there are none.
+ * @param {string} ralphDir
+ * @param {{ limit?: number }} [opts]
+ * @returns {string}
+ */
+function inject(ralphDir, opts) {
+  const limit = (opts && opts.limit != null) ? opts.limit : MAX_INJECT_BULLETS;
+  const bullets = read(ralphDir);
+  if (!bullets.length) return '';
+  const slice = bullets.slice(-limit);
+  return '## Lessons Learned\n\n' + slice.join('\n');
+}
+
+/**
+ * If LESSONS.md has more than `max` non-empty bullets, rewrites it keeping
+ * only the last `max` bullets. Returns the number of bullets dropped (0 if
+ * no write occurred).
+ * @param {string} ralphDir
+ * @param {number} max
+ * @returns {number}
+ */
+function rotate(ralphDir, max) {
+  const filePath = path(ralphDir);
+  let content;
+  try {
+    content = fs.readFileSync(filePath, 'utf8');
+  } catch (e) {
+    if (e.code === 'ENOENT') return 0;
+    throw e;
+  }
+
+  const lines = content.split('\n');
+  const bullets = lines.filter(l => l.trim() !== '');
+  if (bullets.length <= max) return 0;
+
+  const dropped = bullets.length - max;
+  const kept = bullets.slice(-max);
+  fs.writeFileSync(filePath, kept.join('\n') + '\n', 'utf8');
+  return dropped;
+}
+
+module.exports = { path, read, inject, rotate };