Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions packages/harness-omp/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,14 @@ configuration for gateway/frontier models (delete the file or flip
ignored, as are unknown keys (`generate_image` has no toggle and ships ~505
tokens regardless).

**`config/config.gateway-slim.yml`** — the measured middle ground (OPS-481,
docs/eval/2026-07-09-code-mode-vs-omp-discovery.md): full default tool harness
for gateway/frontier models, skills pinned to the shared allowlist. ~28.9k est
tokens vs ~47k ambient (-38%); the skills block is the profile's only
organically-growing cost (+1.7k/day observed from plugin sprawl). Regression
gate: `tools/measure-context.sh --budget 31000 --settings
config/config.gateway-slim.yml`.

**`tools/measure-context.sh`** measures the real footprint: it runs `omp -p`
against a localhost capture sink under a throwaway profile and prints the
system/tools/skills breakdown. Run it on every pin bump
Expand Down
32 changes: 32 additions & 0 deletions packages/harness-omp/config/config.gateway-slim.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# omp gateway-slim profile — full default TOOL harness, TRIMMED skill discovery.
#
# The measured middle ground (docs/eval/2026-07-09-code-mode-vs-omp-discovery.md,
# OPS-481, row D): the gateway/frontier profile's skill auto-discovery block is
# its largest and only ORGANICALLY-GROWING context cost (~16.5k est tokens and
# +1.7k in a single day from ambient plugin growth). Pinning skills to the
# shared allowlist — and nothing else — cut a live gateway profile from
# ~47.0k to ~28.9k est tokens (-38%) with zero tool-availability change.
#
# vs the other presets:
# config.local.yml — 32k-local-model profile: slims tools AND skills (~13k)
# this file — gateway models (200k ctx): full tools, slim skills (~29k)
# no settings file — omp defaults: full tools, ambient skills (~45k and growing)
#
# Install: copy to ~/.omp/agent/config.yml (never clobber live config).
# Keys are NESTED maps — dotted keys are silently ignored (verified live).
# The allowlist is CI-synced against packages/skills/ (tests/preset.test.ts).

# --- role -> model mapping ---------------------------------------------------
modelRoles:
default: llm/zai/GLM-5.2

# --- skill discovery scoping (the ~16.5k lever; tools deliberately untouched) -
skills:
includeSkills: [local-model-triage]
enablePiUser: true # ~/.omp/agent/skills (omp-native user dir)
enablePiProject: true # .omp/skills in-repo
enableClaudeUser: false
enableClaudeProject: false
enableCodexUser: false
enableAgentsUser: false
enableAgentsProject: false
31 changes: 31 additions & 0 deletions packages/harness-omp/tests/preset.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -85,10 +85,41 @@ describe("config.local.yml", () => {
});
});

describe("config.gateway-slim.yml", () => {
const GATEWAY_SLIM = readFileSync(join(PKG, "config/config.gateway-slim.yml"), "utf8");
const doc = Bun.YAML.parse(GATEWAY_SLIM) as Record<string, any>;

test("tools deliberately untouched (full gateway harness)", () => {
expect(doc.tools).toBeUndefined();
});

test("includeSkills allowlist stays in sync with packages/skills", () => {
const skillsDir = join(PKG, "../skills");
const shared = readdirSync(skillsDir, { withFileTypes: true })
.filter((e) => e.isDirectory() && existsSync(join(skillsDir, e.name, "SKILL.md")))
.map((e) => e.name)
.sort();
const allowlist = ([...(doc.skills?.includeSkills ?? [])] as string[]).sort();
expect(allowlist).toEqual(shared);
});

test("modelRoles point at models declared in models.yml", () => {
const models = Bun.YAML.parse(MODELS) as Record<string, any>;
const declared = new Set<string>();
for (const [prov, p] of Object.entries<any>(models.providers)) {
for (const m of p.models ?? []) declared.add(`${prov}/${m.id}`);
}
for (const [role, id] of Object.entries<string>(doc.modelRoles ?? {})) {
expect(declared.has(id), `${role}: ${id}`).toBeTrue();
}
});
});

describe("secretlessness", () => {
test.each([
["models.yml", MODELS],
["config.local.yml", SETTINGS],
["config.gateway-slim.yml", readFileSync(join(PKG, "config/config.gateway-slim.yml"), "utf8")],
])("%s carries no secret-shaped strings", (_name, text) => {
for (const needle of SECRETY) expect(text).not.toInclude(needle);
});
Expand Down
Loading