Skip to content

Commit af9fdc0

Browse files
nikivdevnikivdev
committed
feat: add router RL signals and CI/CD runbooks
1 parent a10d405 commit af9fdc0

10 files changed

Lines changed: 836 additions & 26 deletions

‎docs/ci-cd-runbook.md‎

Lines changed: 171 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,171 @@
1+
# Flow CI/CD Runbook
2+
3+
This runbook documents how Flow CI/CD is wired today and how to debug it quickly when jobs fail.
4+
5+
## Architecture
6+
7+
- Workflows:
8+
- `.github/workflows/canary.yml`: runs on every push to `main`, publishes/updates the `canary` release/tag.
9+
- `.github/workflows/release.yml`: runs on tag pushes matching `v*`, publishes stable releases.
10+
- Build jobs in both workflows:
11+
- Matrix build: macOS + Linux targets.
12+
- SIMD build: `build-linux-host-simd` (Linux x64 with `--features linux-host-simd-json`).
13+
- Release jobs:
14+
- Gather all build artifacts.
15+
- Publish release assets (and in Canary, force-move `canary` tag to current `main` commit).
16+
17+
## Runner Modes
18+
19+
Flow uses task-driven mode switching (not manual workflow edits):
20+
21+
- `github` mode:
22+
- Standard Linux lanes on `ubuntu-latest`.
23+
- SIMD lane disabled.
24+
- `blacksmith` mode:
25+
- Linux lanes on Blacksmith runners.
26+
- SIMD lane enabled on Blacksmith.
27+
- `host` mode:
28+
- Standard Linux lanes stay on GitHub-hosted runners.
29+
- SIMD lane runs on self-hosted label: `[self-hosted, linux, x64, ci-1focus]`.
30+
31+
Check/switch mode:
32+
33+
```bash
34+
f ci-blacksmith-status
35+
f ci-blacksmith-enable
36+
f ci-blacksmith-enable-apply
37+
f ci-host-enable
38+
f ci-host-enable-apply
39+
f ci-blacksmith-disable
40+
f ci-blacksmith-disable-apply
41+
```
42+
43+
## One-Command Host Setup
44+
45+
Preferred path (painless, idempotent):
46+
47+
```bash
48+
f ci-host-setup
49+
```
50+
51+
If infra host is not configured yet:
52+
53+
```bash
54+
f ci-host-setup <user@ip>
55+
```
56+
57+
What `f ci-host-setup` does:
58+
59+
1. Validates `gh` auth and `infra` host config.
60+
2. Installs/registers the `ci-1focus` self-hosted runner on the Linux host.
61+
3. Waits for runner to report online.
62+
4. Switches workflows to `host` mode with commit + push.
63+
5. Prints final runner health/status.
64+
65+
## Daily Operations
66+
67+
- Check current mode: `f ci-blacksmith-status`
68+
- Check runner service + GitHub registration: `f ci-host-runner-status`
69+
- Reinstall runner if needed: `f ci-host-runner-install`
70+
- Remove runner: `f ci-host-runner-remove`
71+
72+
Stable release flow:
73+
74+
1. Merge version bump to `main`.
75+
2. Push tag `vX.Y.Z`.
76+
3. Watch `Release` workflow.
77+
78+
Canary flow:
79+
80+
1. Push to `main`.
81+
2. Watch `Canary` workflow.
82+
3. Confirm `canary` tag moved and release assets updated.
83+
84+
## Debug Playbook
85+
86+
### 1) Workflow failed or stuck
87+
88+
```bash
89+
gh run list --workflow Canary --limit 10
90+
gh run list --workflow Release --limit 10
91+
gh run view <run-id> --log-failed
92+
gh run watch <run-id>
93+
```
94+
95+
### 2) SIMD lane queued forever
96+
97+
Usually means self-hosted runner routing issue.
98+
99+
```bash
100+
f ci-blacksmith-status
101+
f ci-host-runner-status
102+
python3 ./scripts/ci_host_runner.py health --repo nikivdev/flow
103+
```
104+
105+
Expected healthy state is:
106+
107+
- Host service: `active`
108+
- GitHub runner status: `online`
109+
- Runner has label `ci-1focus`
110+
111+
If not healthy, run:
112+
113+
```bash
114+
f ci-host-runner-install
115+
python3 ./scripts/ci_host_runner.py wait-online --repo nikivdev/flow --timeout-secs 120 --interval-secs 5
116+
```
117+
118+
### 3) Workflows not using expected runner profile
119+
120+
```bash
121+
f ci-blacksmith-status
122+
```
123+
124+
If wrong:
125+
126+
```bash
127+
f ci-host-enable-apply
128+
# or:
129+
f ci-blacksmith-enable-apply
130+
# or:
131+
f ci-blacksmith-disable-apply
132+
```
133+
134+
### 4) `curl ... install.sh` does not fetch expected fresh build
135+
136+
Flow installer defaults to `canary` unless `FLOW_VERSION` is set differently. Check if `canary` moved:
137+
138+
```bash
139+
git ls-remote --tags origin canary
140+
```
141+
142+
Then verify in sandbox (recommended) using:
143+
144+
- `docs/rise-sandbox-feature-test-runbook.md`
145+
146+
That runbook gives an isolated `rise sandbox` smoke test for:
147+
148+
```bash
149+
curl -fsSL https://myflow.sh/install.sh | sh
150+
~/.flow/bin/f --version
151+
```
152+
153+
### 5) Setup task fails mid-install
154+
155+
Re-run:
156+
157+
```bash
158+
f ci-host-setup
159+
```
160+
161+
The installer path is idempotent (it removes old service/config before re-registering). If failure persists, inspect:
162+
163+
```bash
164+
f ci-host-runner-status
165+
gh api repos/nikivdev/flow/actions/runners
166+
```
167+
168+
## Notes
169+
170+
- CI/CD execution is defined in repo workflows; GitHub UI is control plane/visibility (runs, logs, runner state), not the source of truth for pipeline logic.
171+
- Current performance balance: keep general Linux matrix jobs on GitHub-hosted runners, offload expensive SIMD build to the Linux host.

‎docs/index.mdx‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ High-signal docs for the Codex/Claude workflow:
1212
- [`commands/invariants.md`](commands/invariants.md): invariant policy checks (`f invariants`) and commit-time enforcement behavior.
1313
- [`commands/fast.md`](commands/fast.md): low-latency AI task dispatch (`f fast`) via fast daemon client path.
1414
- [`commands/seq-rpc.md`](commands/seq-rpc.md): native `seqd` RPC bridge (`f seq-rpc`) for OS-level agent actions.
15+
- [`ci-cd-runbook.md`](ci-cd-runbook.md): CI/CD architecture, runner-mode operations, and failure debug checklist for canary/release pipelines.
1516
- [`rise.md`](rise.md): full integration guide for installing and operating Rise via Flow (`f install rise`) across adopt/sync/dev/mobile/schema/sandbox workflows.
1617
- [`commands/domains.md`](commands/domains.md): shared local `*.localhost` proxy ownership via `f domains` (prevents per-repo port-80 collisions).
1718
- [`commands/up.md`](commands/up.md): one-command project startup (`f up`) with optional lifecycle domain setup.

‎docs/rise-sandbox-feature-test-runbook.md‎

Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
# Rise Sandbox Feature Test Runbook (Flow)
2+
3+
Use this when you want deterministic, isolated feature checks in a VM and fast feedback for infra tuning.
4+
5+
## Goal
6+
7+
- Verify a feature works in a clean sandbox.
8+
- Avoid host-machine state leaks.
9+
- Capture timings/logs so CI/CD and install paths can be optimized.
10+
11+
## Prereqs
12+
13+
- macOS host.
14+
- `rise` available.
15+
- `vibe` VM binary from `~/repos/lynaghk/vibe`.
16+
17+
`rise sandbox` expects the VM-oriented `vibe`, not the unrelated CLI binary some PATHs contain.
18+
19+
Preflight:
20+
21+
```bash
22+
cd ~/repos/lynaghk/vibe
23+
cargo build --release
24+
```
25+
26+
## Canonical Sandbox Command
27+
28+
From `~/code/rise`:
29+
30+
```bash
31+
rise sandbox "set -euo pipefail; <your commands>; echo SANDBOX_OK" \
32+
--root /Users/nikiv/code/flow \
33+
--expect SANDBOX_OK
34+
```
35+
36+
Why this shape:
37+
38+
- `set -euo pipefail` fails hard on the first real issue.
39+
- `--expect` gives a strict pass/fail marker.
40+
- `--root /Users/nikiv/code/flow` mounts the Flow repo into `/root/project`.
41+
42+
## Feature Test Template
43+
44+
Replace with your feature command:
45+
46+
```bash
47+
rise sandbox "set -euo pipefail; cd /root/project; <feature command>; echo FEATURE_OK" \
48+
--root /Users/nikiv/code/flow \
49+
--expect FEATURE_OK
50+
```
51+
52+
## Installer/Release Verification (Flow)
53+
54+
Use this to verify `curl -fsSL https://myflow.sh/install.sh | sh` pulls the newest canary:
55+
56+
```bash
57+
rise sandbox "set -euo pipefail; curl -fsSL https://myflow.sh/install.sh | sh; ~/.flow/bin/f --version; echo INSTALL_OK" \
58+
--root /Users/nikiv/code/flow \
59+
--expect INSTALL_OK
60+
```
61+
62+
Then verify canary tag points to the commit you expect:
63+
64+
```bash
65+
git ls-remote --tags https://github.com/nikivdev/flow.git canary
66+
```
67+
68+
## Infra Optimization Loop
69+
70+
1. Run the same sandbox test 3-5 times.
71+
2. Record:
72+
- VM boot + script duration from `rise sandbox` output.
73+
- Feature command duration inside script (`time <cmd>` if needed).
74+
- Artifact install/build timing (`f --version`, compile/install steps).
75+
3. Compare before/after infra changes:
76+
- CI runner mode (`github` vs `host` vs `blacksmith`).
77+
- Caching changes.
78+
- Installer path changes.
79+
80+
Sandbox logs are emitted under:
81+
82+
```bash
83+
~/code/flow/out/logs/sandbox-<timestamp>.log
84+
```
85+
86+
## Common Failures
87+
88+
### `vibe: error: unrecognized arguments: --cpus --ram ...`
89+
90+
Cause: wrong `vibe` binary in PATH.
91+
92+
Fix: build/use `~/repos/lynaghk/vibe/target/release/vibe` (rise resolves this first when present).
93+
94+
### Sandbox passes but installed version seems old
95+
96+
Check:
97+
98+
1. Canary tag head:
99+
```bash
100+
git ls-remote --tags https://github.com/nikivdev/flow.git canary
101+
```
102+
2. Latest canary workflow status:
103+
```bash
104+
gh run list -R nikivdev/flow --limit 5
105+
```
106+
107+
If canary tag already points to your target commit, installer should fetch that version even if other optional jobs are still running.
108+

‎docs/rl-signal-capture-runbook.md‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,11 +26,28 @@ From `~/code/flow`:
2626
```bash
2727
export FLOW_RL_SIGNALS=true
2828
export FLOW_RL_SIGNALS_PATH=out/logs/flow_rl_signals.jsonl
29+
export FLOW_RL_SIGNALS_SEQ_MIRROR=true
30+
export FLOW_RL_SIGNALS_SEQ_PATH=~/repos/ClickHouse/ClickHouse/user_files/seq_mem.jsonl
2931
export FLOW_RL_SIGNAL_TEXT=snippet
3032
export FLOW_RL_SIGNAL_MAX_CHARS=4000
3133
```
3234

3335
`f ai everruns ...` now emits structured runtime/tool events into the JSONL file.
36+
`f ai:*` task execution via `ai-taskd` now also emits linked router events:
37+
38+
- `flow.router.decision.v1`
39+
- `flow.router.override.v1` (when a suggested task differs from chosen task)
40+
- `flow.router.outcome.v1`
41+
42+
These are mirrored directly into `seq_mem.jsonl` when `FLOW_RL_SIGNALS_SEQ_MIRROR=true`.
43+
44+
To capture override events, set suggestion context on the command that triggers `f ai:*`:
45+
46+
```bash
47+
export FLOW_ROUTER_SUGGESTED_TASK=ai:flow/noop
48+
export FLOW_ROUTER_OVERRIDE_REASON=manual_user_choice
49+
f ai:flow/dev-check
50+
```
3451

3552
## 3) Inspect quality in real time
3653

@@ -56,6 +73,9 @@ f rl-signal-summary
5673
- `everruns.qa_pair` (prompt/response supervision pair)
5774
- `everruns.run_completed` or `everruns.run_failed`
5875
- `agent.qa.pair` in `seq_mem.jsonl` (Claude/Codex Q/A pairs from background ingest)
76+
- `flow.router.decision.v1` in `seq_mem.jsonl`
77+
- `flow.router.override.v1` in `seq_mem.jsonl` (when suggestion context is provided)
78+
- `flow.router.outcome.v1` in `seq_mem.jsonl`
5979

6080
## 5) Build Harbor snapshot from runtime traces
6181

‎flow.toml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -306,6 +306,10 @@ variables = [
306306
{ key = "ANTHROPIC_API_KEY", default = "" },
307307
{ key = "FLOW_RL_SIGNALS", default = "true" },
308308
{ key = "FLOW_RL_SIGNALS_PATH", default = "out/logs/flow_rl_signals.jsonl" },
309+
{ key = "FLOW_RL_SIGNALS_SEQ_MIRROR", default = "true" },
310+
{ key = "FLOW_RL_SIGNALS_SEQ_PATH", default = "~/repos/ClickHouse/ClickHouse/user_files/seq_mem.jsonl" },
311+
{ key = "FLOW_ROUTER_SUGGESTED_TASK", default = "" },
312+
{ key = "FLOW_ROUTER_OVERRIDE_REASON", default = "" },
309313
{ key = "FLOW_RL_SIGNAL_TEXT", default = "snippet" },
310314
{ key = "FLOW_RL_SIGNAL_MAX_CHARS", default = "4000" },
311315
{ key = "SEQ_CH_MEM_PATH", default = "~/repos/ClickHouse/ClickHouse/user_files/seq_mem.jsonl" },

‎src/ai_everruns.rs‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -246,7 +246,10 @@ fn wait_for_completion_sse(
246246
event_start_ns,
247247
end_unix_nanos(event_start_ns, event_started),
248248
vec![
249-
("tool_calls.requested".to_string(), requested_count.to_string()),
249+
(
250+
"tool_calls.requested".to_string(),
251+
requested_count.to_string(),
252+
),
250253
("tool_calls.unique".to_string(), unique_count.to_string()),
251254
(
252255
"tool_calls.duplicates_filtered".to_string(),
@@ -395,7 +398,10 @@ fn wait_for_completion_poll(
395398
event_start_ns,
396399
end_unix_nanos(event_start_ns, event_started),
397400
vec![
398-
("tool_calls.requested".to_string(), requested_count.to_string()),
401+
(
402+
"tool_calls.requested".to_string(),
403+
requested_count.to_string(),
404+
),
399405
("tool_calls.unique".to_string(), unique_count.to_string()),
400406
(
401407
"tool_calls.duplicates_filtered".to_string(),

0 commit comments

Comments
 (0)