diff --git a/docs/eval/2026-07-07-model-harness-landscape-survey.md b/docs/eval/2026-07-07-model-harness-landscape-survey.md index 1f1c756..3212373 100644 --- a/docs/eval/2026-07-07-model-harness-landscape-survey.md +++ b/docs/eval/2026-07-07-model-harness-landscape-survey.md @@ -16,6 +16,14 @@ Research project now reported as the fastest-growing agent framework in open source (60k+ stars in two months) on the strength of persistent multi-tier memory + a skill-writing learning loop — a pin-audit is Top-5 item 3. +> **CORRECTION 2026-07-09 (OPS-480, source-verified):** the "persistent +> multi-tier memory" claim above does not hold up against +> `hermes_cli/config.py` on `NousResearch/hermes-agent@main` — the built-in +> memory provider is two flat Markdown files (`MEMORY.md`, `USER.md`) with no +> tier granularity, not three tiers. The skill-writing loop is confirmed +> present and on by default at 0.18.0. See +> `packages/harness-hermes/README.md` Verdict section for the full audit. + ## 1. Models for the cluster ### Pragmatic — fits gateway upstreams or 32GB local tier now @@ -88,6 +96,15 @@ memory + a skill-writing learning loop — a pin-audit is Top-5 item 3. 60k+ stars in two months. **We pin 0.18.x — audit whether the memory/skill loop postdates the pin.** [site](https://hermes-agent.org/) · [writeup](https://www.tencentcloud.com/techpedia/143930) + > **CORRECTION 2026-07-09 (OPS-480, source-verified):** memory is two flat + > Markdown files (`MEMORY.md` agent notes, `USER.md` user profile) injected + > into the system prompt, plus an empty-by-default external-provider slot + > (7 options, none of them Engram) — NOT three-tier semantic/working/ + > episodic; that framing doesn't correspond to any config surface at + > 0.18.0. Skill-writing loop confirmed present and on by default, with + > unrestricted writes (`skills.write_approval: false` upstream default). + > Full audit + Engram-boundary lane decision in + > `packages/harness-hermes/README.md` Verdict section. - **OpenClaw** (ex-Moltbot/Clawdbot) — 280k+ stars (reportedly passed React; 9k→60k in 72h at peak). Local-first gateway, 20+ messaging channels, cron/nodes/canvas first-class, 13.7k-skill marketplace, Markdown memory. diff --git a/packages/harness-hermes/README.md b/packages/harness-hermes/README.md index ba0a45c..c369310 100644 --- a/packages/harness-hermes/README.md +++ b/packages/harness-hermes/README.md @@ -41,3 +41,96 @@ unsigned-paas runs this same upstream (pinned 0.16.0) as a kagent BYO agent with an A2A shim (`unsigned-paas/applications/hermes-agent*`). This package is the *local/workstation* harness; bumping the cluster pin to 0.18.0 is tracked in Linear, not done here. + +## Verdict — 0.18.x memory/skill-loop review (OPS-480, 2026-07-08) + +Review-only: **no version bump.** The 2026-07-07 landscape survey +(`docs/eval/2026-07-07-model-harness-landscape-survey.md`) flagged hermes-agent +as fastest-growing on the strength of "three-tier persistent memory +(semantic/working/episodic) + a closed skill-writing learning loop" and asked +whether that postdates our 0.18.0 pin. Findings below come from reading +`hermes_cli/config.py` on `NousResearch/hermes-agent@main` directly (config +defaults, the v11→v12 provider-schema normalizer, and the write-approval +migration), not from docs or blog summaries, several of which turned out to +be wrong or imprecise (noted below). Upstream has since shipped 0.18.1 and +0.18.2 patch releases (2026-07-07); this review is against the 0.18.0 +"Judgment Release" behavior since that's what's pinned. + +### What's actually there at 0.18.0 + +- **Memory is not three-tiered.** The built-in provider is two flat + Markdown files — `MEMORY.md` (agent notes, 2200-char cap) and `USER.md` + (user profile, 1375-char cap) — stored at `~/.hermes/memories/` and + injected into the system prompt. Defaults: `memory.memory_enabled: true`, + `memory.write_approval: false` (writes land with no review gate). A + `memory.provider` slot exists for 7 external providers (Honcho, Mem0, + Hindsight, Supermemory, OpenViking, holographic, retaindb) but is empty + by default — none of them is Engram, and there is no adapter hook for a + custom provider. The "semantic/working/episodic" framing in the survey is + secondary-source terminology describing the general memory-architecture + pattern, not a config surface this version exposes — there is no + independent on/off switch per tier. +- **The skill-writing loop is present and on by default, unrestricted.** + `/learn` and the `skill_manage` tool distill reusable `SKILL.md` files + into `~/.hermes/skills/` after complex tasks, error-recovery, or user + correction. Governance is `skills.write_approval` (default `false` — free + writes) and `skills.guard_agent_created` (default `false` — no + content-pattern scan). One secondary source claimed a `skills.auto_create` + gate defaults it off; that key does not exist in source — treat that + claim as wrong. +- **Provider-config schema is fine as pinned.** A closed 2026-04 upstream + issue (#8776) reported the `providers:` dict format being write-only + (migration wrote it, nothing read it back) — that's fixed on `main`: + `get_compatible_custom_providers()` now merges both `custom_providers:` + (legacy list) and `providers:` (dict) into one runtime view, and the + normalizer accepts `api`/`url`/`base_url` as URL-key aliases and both + list- and dict-shaped `models:`. Our `config/cli-config.local-endpoint.yaml` + preset (`providers..api`, `models:` as a string list) matches this + and needed no schema fix. A separate, still-**open** bug (#43713) affects + `hermes profile` sub-configs replacing rather than merging the parent's + `providers:` — irrelevant to us since this preset doesn't use profiles. + +### Engram boundary — the real question + +If hermes memory stays on with its default unrestricted writes, hermes and +Engram independently accumulate the same class of fact (operator +preferences, environment conventions) with no reconciliation between the +two stores — not a corruption risk (they're separate files/DBs), but a +drift risk: two silently-diverging "memories of the operator," one of which +(hermes's) is invisible to session-bootstrap, Obsidian, and every other +harness. Engram is this repo's designated cross-harness, cross-session +memory system per the operator's global config. Lane assignment adopted +here: **hermes's built-in memory tier goes off entirely** +(`memory.memory_enabled: false` in the local-endpoint preset); Engram +remains the sole long-term memory of record across all four harnesses. +Skills are a different domain (hermes-local executable procedures, no +Engram equivalent) — the self-improvement loop stays on, but +`skills.write_approval: true` now gates agent-authored skill writes behind +a review step (`/skills pending` / `/skills diff` / `/skills approve`), +since they currently land with zero review and can carry inline shell +(`skills.inline_shell`, off by default, unaffected by this change). + +Related: OPS-482 (in progress, separate branch) is defining +checkpoint-vs-memory boundaries for shepherd's durable-patterns work — same +category of problem, different subsystem. Not a blocker for this verdict; +cross-check the two once OPS-482 lands. + +### Preset diff applied + +```diff ++ memory: ++ memory_enabled: false ++ skills: ++ write_approval: true +``` + +in `config/cli-config.local-endpoint.yaml`. `tests/test_presets.py` (YAML +validity, localhost-only URLs, secretlessness) and +`tests/test_pin_drift.py` (pin/README agreement) both still pass — this +change doesn't touch the version pin. + +### Cluster-pin skew note + +unsigned-paas's kagent BYO agent is still on 0.16.0 against this package's +0.18.0 — a two-minor-version gap. Tracked as a separate bump, not addressed +here. diff --git a/packages/harness-hermes/config/cli-config.local-endpoint.yaml b/packages/harness-hermes/config/cli-config.local-endpoint.yaml index 3eb7851..0a0981d 100644 --- a/packages/harness-hermes/config/cli-config.local-endpoint.yaml +++ b/packages/harness-hermes/config/cli-config.local-endpoint.yaml @@ -49,3 +49,22 @@ providers: # Alternate local endpoints (swap `api`): # llama.cpp: http://localhost:8080/v1 (serve/serve-llamacpp.sh) # vLLM: http://localhost:8000/v1 (serve/serve-vllm.sh) + +# Memory/skill-loop boundary vs Engram (OPS-480, 2026-07-08; see README Verdict +# section for the full reasoning). hermes's built-in memory store (two flat +# files, ~/.hermes/memories/MEMORY.md + USER.md) would otherwise accumulate +# the same class of cross-session facts about the operator as Engram, with no +# reconciliation between the two. Engram is this repo's designated +# cross-harness memory of record (session-bootstrap wiring, all four +# harnesses) -- hermes's own tier stays off so there is exactly one canonical +# long-term store. +memory: + memory_enabled: false + +# The skill-writing self-improvement loop (/learn, skill_manage) lands +# agent-authored SKILL.md files with no review gate by default upstream. +# Skills are hermes-local procedural knowledge -- not Engram's domain -- so +# the loop itself stays on, but agent-authored writes require a look before +# they land. +skills: + write_approval: true