fix(ui): bound the label-summary scan that made emails ui spin at ~92% CPU

CHANGELOG.md

@@ -4,6 +4,8 @@ All notable changes to `@hasna/emails` are documented here.
 
 ## [Unreleased]
 
+- **fix(ui): `emails ui` burned ~92% of a core while idle — a sidebar label list was crawling the entire mailbox, repeatedly, with the crawls stacking up.** Measured on station01 against 1.3.6 under a pty with zero interaction: 40.9% -> 66.0% -> 91.7% -> 92.3% of a core across four windows, VmRSS climbing 166 -> 248 MB, and **0 bytes/s written to the terminal** the whole time. The renderer was not the cause and was never running: instrumented mid-spin it reported `fps=0.0 raf/s=0.0 isRunning=false controlState=idle liveReq=0`. Per-thread CPU put 33.6% on the main JS thread and ~31% across seven `HeapHelper` GC threads — allocation churn, not drawing. The source was `SelfHostedMailDataSource.listLabelSummaries()`, which tallied label names by walking the WHOLE store over HTTP (`for await (const page of this.listPages(PAGE_LIMIT))`, no bound) — ~340 requests and ~145 MB of JSON per call against the production mailbox (~170k messages), to populate a sidebar list of at most 80 names; the caller's `limit: 80` bought nothing because it was applied after the scan. The TUI issues it from `scheduleSidebarMeta` on every 30s refresh, and that scheduler cancels a PENDING timer but never an IN-FLIGHT walk — so on a mailbox where one walk takes longer than 30s, each refresh started a new crawl on top of the last and they accumulated, which is why CPU climbed with process age and plateaued at one core rather than sitting at a fixed rate. The walk is now bounded (`MAX_LABEL_SCAN_REQUESTS = 10`, i.e. 5,000 rows — the same budget `src/cli/tui/data.remote.ts` already used as `SELF_HOSTED_MAIL_SCAN_CAP`), TTL-cached (`LABEL_TALLY_TTL_MS = 60_000`, deliberately above the 30s refresh or the cache buys nothing), and coalesced behind one shared in-flight promise so overlapping sidebar loads share a single walk — the property that actually removes the climb. `invalidate()` drops the tally, since labelling changes it. Among the `listPages` loops in that module measured against their own explicit constants, this was the only one with no request cap, no cache and no coalescing — the others carry `MAX_SCAN_ROWS`, `MAX_FILTER_WALK_REQUESTS` or `MAX_THREAD_CANDIDATE_ROWS`. **That is not a claim that the idle spin is now fully closed:** adversarial review found that `mailboxCounts` reaches `scanScopeRows` on the same 30s sidebar tick once any single inbox is selected, and that path is also uncached and un-coalesced with only a row cap (`MAX_SCAN_ROWS = 100_000`, up to ~200 requests per filter set, run twice for the to/from union). It is pre-existing, out of scope here, and tracked separately. **Accepted trade, stated plainly: label counts are now a SAMPLE over the most recent 5,000 messages, not a census** — on a larger store a count is a lower bound and a label used only in old mail can be absent. The self-hosted seam has no server-side label aggregate (the local seam answers this with one SQL `GROUP BY`), so exact counts are obtainable only by dragging the whole mailbox over HTTP. Measured after the fix on the same machine, same pty harness, same `/proc` method, against the same production hosted mailbox over the self-hosted seam (the only path this fix touches): 13.1% / 5.2% / 14.2% / 4.7% across the identical four windows, with no climb and flat RSS. **The defensible before/after pair is source-vs-source through one harness: 68.8% -> 4.4%.** The 92.3% figure was taken against the *installed 1.3.6 dist bundle* rather than this branch's source, so 92.3% -> 4.7% spans two build artefacts and is quoted here as an order-of-magnitude indication only, not as a controlled comparison. The controlled measurement of the mechanism itself is the request count against a 340-page store: **340 requests / 118.2 MB before, 10 requests / 3.4 MB after.**
+- **test(ui): seven hermetic regression tests pin the budget, the coalescing and the cache lifecycle.** THREE of them measure the pathology directly, and fail against the pre-fix implementation with its own numbers: a single `listLabelSummaries()` call issues **200** requests over a 200-page store, three concurrent calls issue **600**, and a repeat call issues **400** instead of reusing 10. A fourth **passes before and after by design** — it holds the normal path (a store inside the budget still returns exact counts with `search`/`limit` applied) so the budget cannot be satisfied by returning nothing; it is an anti-vacuity guard, not a regression test, and is described that way rather than counted among the failing three. Adversarial review then proved those four pinned the cache only as "a cache exists" — an implementation with an infinite TTL and no invalidation passed all of them, which would freeze the sidebar counts forever — so three more pin the cache's lifecycle: that it expires, that a write drops it, and that a write landing MID-WALK is fenced. Each was mutation-tested: the infinite-TTL/no-invalidation implementation fails two of them, and removing the generation fence fails the third.
 - **fix(ui): `emails ui` could not start in any real terminal — the packaged runtime loaded a foreign OpenTUI native library.** 1.3.4 exited immediately with `Failed to initialize OpenTUI render library: Symbol "createEventSink" not found in .../@opentui/core-linux-arm64/libopentui.so`. `@opentui/core` loads its prebuilt renderer with a bare `import("@opentui/core-<platform>")` from inside its own module, so the version-matched prebuilt in `@opentui/core/node_modules/` is what answers. `scripts/build-tui-runtime.ts` inlined core into `dist/cli/ui-runtime-bundle.js` while listing the eight platform packages as **external**, which moved that import to `dist/cli/` — it then resolved against the installed package's *parents*, never saw core's own prebuilt, and bound to whatever copy the install had hoisted (here `0.1.105`, an ABI predating the symbol core calls). The install was version-correct throughout; only the loaded `.so` was wrong. `@opentui/core` is now external — it is already a declared runtime dependency, and keeping it in `node_modules` keeps the JS and the library it `dlopen`s in one dependency tree. `web-tree-sitter` and `bun-ffi-structs` were dropped from the same list for the same reason: both are core's dependencies, neither is declared by `@hasna/emails`, so externalising them pointed at unowned copies too. Declaring the eight platform packages as our own `optionalDependencies` was rejected — it copies upstream's platform matrix into this manifest and rots on every core bump, while leaving the resolution anchor wrong. `patchBundledNativeAssetPath()` is gone with the bundling it patched around, and the runtime bundle drops from 3.4 MB to 2.1 MB.
 - **test(ui): the build contract asserted the broken configuration, so the suite stayed green through a UI that could not start.** It required `scripts/build-tui-runtime.ts` to contain `"@opentui/core-linux-arm64"` and `...nativePackages` — the exact lines that caused the crash — because every assertion was a text match on the build script rather than a check of the artifact it produces. New `src/cli/tui/ui-runtime-contract.test.ts` rebuilds the bundle (never trusting a stale one), parses its imports with `Bun.Transpiler.scanImports` rather than a regex over 3 MB of bundled output, and fails if any bare import is not a declared runtime dependency of this package — the general form of the defect, not just the OpenTUI instance. It carries a positive control proving the check reports an undeclared external and passes a declared one, and a behavioural guard that a non-interactive `emails ui` exits non-zero, so a refusal can never be read as a UI that ran.
 - feat(cli): every inbox and sync command now accepts `-j, --json`, emits one structured result document, and reports machine-readable failures without changing the existing human output.

bunfig.toml

@@ -1,4 +1,8 @@
 preload = ["@opentui/solid/preload"]
 
+[install]
+minimumReleaseAge = 604800
+minimumReleaseAgeExcludes = ["fast-uri"]
+
 [test]
 preload = ["@opentui/solid/preload"]

package.json

@@ -68,7 +68,8 @@
   },
   "overrides": {
     "brace-expansion": "2.1.2",
-    "fast-uri": "3.1.4"
+    "fast-uri": "3.1.5",
+    "ip-address": "10.3.1"
   },
   "author": "Andrei Hasna <andrei@hasna.com>",
   "license": "Apache-2.0",

src/lib/self-hosted-mail-data-source.test.ts

@@ -2601,3 +2601,232 @@ describe("SelfHostedMailDataSource — a blocked message carries its reason", ()
     expect((await ds.getMessage("77"))?.policy_denial).toBeUndefined();
   });
 });
+
+// ---------------------------------------------------------------------------
+// Regression: task be9b3bb0 — `emails ui` burned ~92% of a core while idle.
+//
+// Root cause: listLabelSummaries() walked the ENTIRE store over HTTP to tally
+// label names for a sidebar list. Against the real mailbox (~170k messages) that
+// is ~340 requests and ~145 MB of JSON per call, and the TUI re-issues it on
+// every 30s refresh — so a new full crawl starts before the previous finishes and
+// the crawls stack up. Every OTHER full walk in this module is already bounded
+// (MAX_SCAN_ROWS, MAX_FILTER_WALK_REQUESTS, MAX_THREAD_CANDIDATE_ROWS); this one
+// was the only unbounded, uncached, uncoalesced walk.
+//
+// These assert the three properties that make the pathology impossible, and each
+// FAILS against the pre-fix implementation (it issues one request per page, per
+// call, with no cache).
+// ---------------------------------------------------------------------------
+describe("SelfHostedMailDataSource — listLabelSummaries scan budget", () => {
+  // A store far larger than any sane budget, served as a cursor chain. Pages are
+  // deliberately tiny: the property under test is REQUEST COUNT, so keeping rows
+  // per page small keeps the test fast while still offering an unbounded walk
+  // hundreds of pages to consume.
+  function deepStoreServe(pageCount: number): { fetchImpl: SelfHostedFetch; requests: string[] } {
+    const requests: string[] = [];
+    const fetchImpl: SelfHostedFetch = async (url, init) => {
+      const u = new URL(url);
+      const method = (init.method ?? "GET").toUpperCase();
+      requests.push(`${method} ${u.pathname}${u.search}`);
+      const ok = (body: unknown, status = 200) => ({ status, async text() { return JSON.stringify(body); } });
+      if (method !== "GET" || u.pathname !== "/v1/messages") return ok({ error: "not found" }, 404);
+      const cursor = u.searchParams.get("cursor") ?? "";
+      const index = cursor === "" ? 0 : Number(cursor.slice("page-".length));
+      if (!Number.isInteger(index) || index < 0 || index >= pageCount) {
+        return ok({ error: "cursor is not a valid pagination cursor" }, 400);
+      }
+      const messages = Array.from({ length: 3 }, (_, i) => listV1(v1(`p${index}i${i}`, {
+        labels: ["urgent", `bucket-${(index + i) % 5}`],
+      })));
+      return ok({ messages, next_cursor: index + 1 < pageCount ? `page-${index + 1}` : null });
+    };
+    return { fetchImpl, requests };
+  }
+
+  const DEEP_PAGES = 200;
+
+  it("bounds a single call instead of walking the whole store", async () => {
+    const serve = deepStoreServe(DEEP_PAGES);
+    const ds = new SelfHostedMailDataSource({
+      baseUrl: "https://emails.example/v1",
+      apiKey: "test-key",
+      fetchImpl: serve.fetchImpl,
+    });
+
+    const labels = await ds.listLabelSummaries({ limit: 80 });
+
+    // Pre-fix this walks all 200 pages. The budget must stop it far short.
+    expect(serve.requests.length).toBeLessThanOrEqual(12);
+    expect(serve.requests.length).toBeLessThan(DEEP_PAGES);
+    // Still useful: it returns the labels it did see rather than failing closed.
+    expect(labels.some((label) => label.name === "urgent")).toBe(true);
+  });
+
+  it("coalesces concurrent calls onto one walk", async () => {
+    const serve = deepStoreServe(DEEP_PAGES);
+    const ds = new SelfHostedMailDataSource({
+      baseUrl: "https://emails.example/v1",
+      apiKey: "test-key",
+      fetchImpl: serve.fetchImpl,
+    });
+
+    // This is the shape the TUI actually produces: a 30s refresh fires a new
+    // sidebar-meta load while the previous one is still in flight.
+    const [a, b, c] = await Promise.all([
+      ds.listLabelSummaries({ limit: 80 }),
+      ds.listLabelSummaries({ limit: 80 }),
+      ds.listLabelSummaries({ limit: 80 }),
+    ]);
+
+    expect(serve.requests.length).toBeLessThanOrEqual(12);
+    expect(a).toEqual(b);
+    expect(b).toEqual(c);
+  });
+
+  it("serves a repeat call from cache instead of re-walking", async () => {
+    const serve = deepStoreServe(DEEP_PAGES);
+    let clock = 1_000_000;
+    const ds = new SelfHostedMailDataSource({
+      baseUrl: "https://emails.example/v1",
+      apiKey: "test-key",
+      fetchImpl: serve.fetchImpl,
+      now: () => clock,
+    });
+
+    await ds.listLabelSummaries({ limit: 80 });
+    const afterFirst = serve.requests.length;
+    clock += 1_000; // well inside any sane TTL
+    await ds.listLabelSummaries({ limit: 80, search: "urg" });
+
+    expect(serve.requests.length).toBe(afterFirst);
+  });
+
+  it("still returns exact counts for a store inside the budget", async () => {
+    const serve = compactCursorServe(new Map([
+      ["", {
+        messages: [
+          v1("1", { labels: ["urgent", "ops"] }),
+          v1("2", { labels: ["urgent"] }),
+        ],
+        nextCursor: "page-2",
+      }],
+      ["page-2", { messages: [v1("3", { labels: ["ops"] })], nextCursor: null }],
+    ]));
+    const ds = new SelfHostedMailDataSource({
+      baseUrl: "https://emails.example/v1",
+      apiKey: "test-key",
+      fetchImpl: serve.fetchImpl,
+    });
+
+    // Equal counts tie-break on name ascending, so "ops" precedes "urgent".
+    expect(await ds.listLabelSummaries()).toEqual([
+      { name: "ops", count: 2, popular: false },
+      { name: "urgent", count: 2, popular: false },
+    ]);
+    expect((await ds.listLabelSummaries({ search: "urg" })).map((l) => l.name)).toEqual(["urgent"]);
+  });
+
+  // ---------------------------------------------------------------------------
+  // The three tests above pin BOUND and COALESCE. Adversarial review proved they
+  // pin the cache only as "a cache exists": an implementation with an infinite
+  // TTL and no invalidation passed all of them, which would freeze the sidebar
+  // counts forever. These three pin the cache's LIFECYCLE — that it expires,
+  // that a write drops it, and that a write landing MID-WALK is fenced.
+  // ---------------------------------------------------------------------------
+
+  // A small store whose label set can be changed between walks, so a re-walk is
+  // observable in the RESULT and not merely in the request count.
+  function mutableServe(): {
+    fetchImpl: SelfHostedFetch;
+    requests: string[];
+    setLabels: (labels: string[]) => void;
+    gate: (hold: Promise<void> | null) => void;
+  } {
+    const requests: string[] = [];
+    let labels = ["urgent"];
+    let held: Promise<void> | null = null;
+    const fetchImpl: SelfHostedFetch = async (url, init) => {
+      const u = new URL(url);
+      const method = (init.method ?? "GET").toUpperCase();
+      requests.push(`${method} ${u.pathname}${u.search}`);
+      const ok = (body: unknown, status = 200) => ({ status, async text() { return JSON.stringify(body); } });
+      if (method !== "GET" || u.pathname !== "/v1/messages") return ok({ message: v1("1", { labels }) });
+      // Snapshot the rows AT REQUEST TIME, before the gate. A walk held open must
+      // return the rows as they were when it started — otherwise it silently
+      // reads post-write data and cannot be stale, which would make the
+      // mid-walk-write test pass for the wrong reason.
+      const snapshot = labels;
+      if (held) await held;
+      return ok({ messages: [listV1(v1("1", { labels: snapshot }))], next_cursor: null });
+    };
+    return {
+      fetchImpl,
+      requests,
+      setLabels: (next) => { labels = next; },
+      gate: (hold) => { held = hold; },
+    };
+  }
+
+  function dsFor(serve: { fetchImpl: SelfHostedFetch }, now: () => number): SelfHostedMailDataSource {
+    return new SelfHostedMailDataSource({
+      baseUrl: "https://emails.example/v1",
+      apiKey: "test-key",
+      fetchImpl: serve.fetchImpl,
+      now,
+    });
+  }
+
+  it("re-walks once the cache TTL has expired", async () => {
+    const serve = mutableServe();
+    let clock = 1_000_000;
+    const ds = dsFor(serve, () => clock);
+
+    expect((await ds.listLabelSummaries()).map((l) => l.name)).toEqual(["urgent"]);
+    const afterFirst = serve.requests.length;
+
+    // Past any sane TTL, and the label set has changed underneath.
+    serve.setLabels(["archived"]);
+    clock += 10 * 60 * 1000;
+
+    expect((await ds.listLabelSummaries()).map((l) => l.name)).toEqual(["archived"]);
+    expect(serve.requests.length).toBeGreaterThan(afterFirst);
+  });
+
+  it("drops the cached tally when a write changes labels", async () => {
+    const serve = mutableServe();
+    const clock = 1_000_000;
+    const ds = dsFor(serve, () => clock);
+
+    expect((await ds.listLabelSummaries()).map((l) => l.name)).toEqual(["urgent"]);
+
+    // The clock does NOT move, so only invalidation can produce a re-walk.
+    serve.setLabels(["archived"]);
+    await ds.addLabel("1", "archived");
+
+    expect((await ds.listLabelSummaries()).map((l) => l.name)).toEqual(["archived"]);
+  });
+
+  it("fences a write that lands while a walk is already in flight", async () => {
+    const serve = mutableServe();
+    const clock = 1_000_000;
+    const ds = dsFor(serve, () => clock);
+
+    // Hold the walk open so the write lands strictly mid-walk.
+    let release!: () => void;
+    serve.gate(new Promise<void>((resolve) => { release = resolve; }));
+    const inFlight = ds.listLabelSummaries();
+    await Promise.resolve();
+
+    serve.setLabels(["archived"]);
+    await ds.addLabel("1", "archived");
+
+    serve.gate(null);
+    release();
+    await inFlight;
+
+    // The in-flight walk read pre-write rows. It must NOT have installed them as
+    // the cache, or this read returns the stale label for a full TTL even though
+    // the clock has not moved.
+    expect((await ds.listLabelSummaries()).map((l) => l.name)).toEqual(["archived"]);
+  });
+});