fix(web): warn when a configured search/extract backend is discarded

[juniperbevensee]

The failure

A fleet had web.search_backend: brave set. The registered provider name is brave-free, so the value was discarded and a different provider (firecrawl) served every search on all seven agents, indefinitely, with nothing logged at any level. The config file and the runtime disagreed and nothing said so — the only symptom was "web search returns empty", indistinguishable from a provider outage. Diagnosing it required reading the dispatcher source.

This PR was rewritten twice after audits

Both audits found the change emitting false diagnostics — disqualifying for a PR arguing that a wrong message turns a typo into an unfalsifiable bug report. Recording what was wrong, since the failures are more instructive than the fix:

Round 1 — the classifier lied.

• _KNOWN_BACKENDS was a verbatim duplicate of _LEGACY_WEB_BACKENDS 132 lines above it in the same file, used with the opposite meaning to that constant's own docstring. A registered third-party provider missing its key was therefore reported as a typo, with a "valid values" list excluding the user's own provider.
• The dispatch guard if backend: was dead code — _get_backend() never returns empty — so a fresh install with no config announced "Configured search backend 'brave-free' is not a registered provider", contradicting the tool's own error 15 lines later.
• web.backend — the only key hermes tools writes — wasn't covered at all.

Round 2 — the replacement dispatch warner lied differently. It hardcoded web.%s_backend while its value could come from web.backend, naming a key absent from the operator's config. It asserted "the credential is set", inferred from configured == resolved — a fact _resolve_backend never establishes, since it returns any legacy-or-registered name without probing availability.

What it does now

One report site: the config layer, in _warn_discarded_backend, which knows which key it read. Classification uses _LEGACY_WEB_BACKENDS or _registered_web_provider(...) — the same test _resolve_backend uses — so a registered provider is never called a typo.

WARNING tools.web_tools: web.search_backend='brave' is not a known backend — falling
back to 'firecrawl', so your configured provider is NOT being used. Valid values:
brave-free, ddgs, exa, firecrawl, parallel, searxng, tavily, xai.

versus a real backend missing its credential:

WARNING tools.web_tools: web.search_backend='tavily' is a recognised backend but is
not available (its API key, URL or plugin is missing) — falling back to 'firecrawl'.

The dispatch-layer warner is removed, not repaired. Beyond being wrong, it was near-dead: every bundled provider supports search, and the extract dispatcher already returns a precise typed error for a genuine capability mismatch a few lines below. A warning that misstates the key, invents a credential fact, and duplicates a better error is worse than silence. A test asserts it stays gone.

_valid_backend_names filters by capability. An extract_backend typo was previously answered with a list including brave-free/ddgs/searxng/xai — none of which can extract — swapping one broken config for another and contradicting the extract dispatcher's own "firecrawl, tavily, exa, or parallel".

Deduped per (key, value): these paths run on every dispatch and every hermes tools repaint, so an un-deduped warning becomes the spam it exists to surface. Silence is preserved wherever nothing was lost — no configured value, an available backend, or a fallback resolving to the same provider.

Behavior

Logging only. Confirmed by AST comparison that _resolve_backend's body is identical to the previous _get_backend; the new wrapper returns resolved unconditionally with the warn in try/except.

Testing

12 tests, mutation-verified rather than trusted green — reverting each of six behaviours individually fails a test (silence the warner → 5 failures; misclassify registered providers, warn-when-unconfigured, drop web.backend coverage, capability-blind list, break dedup → 1 each). 265 web tests pass, ruff clean.

The dedup reset lives in tests/tools/conftest.py so it guards the whole directory: the set is process-global, and pytest tests/tools/ shares one interpreter (cross-file isolation only holds under scripts/run_tests_parallel.py, which CI uses).

Upstream

This code is upstream (NousResearch/hermes-agent) — the fallback block is kshitijk4poor's "refactor(web): dispatch all three tools through web_search_registry", present verbatim on upstream/main at tools/web_tools.py:685-691. Upstream issue #71929 already documents this bug, with the identical brave vs brave-free example and the words "silently fails"; its proposed fix is a config-form dropdown, which does nothing for agents whose config.yaml is machine-written. So this is a different layer of fix to an already-reported problem, not a novel finding. No upstream PR is opened here.

🤖 Generated with Claude Code