Skip to content

Cloud deployment add agent health monitoring and auto restart on crash#417

Open
clintjeff2 wants to merge 2 commits into
Bitcoindefi:mainfrom
clintjeff2:Cloud-deployment-add-agent-health-monitoring-and-auto-restart-on-crash-#198
Open

Cloud deployment add agent health monitoring and auto restart on crash#417
clintjeff2 wants to merge 2 commits into
Bitcoindefi:mainfrom
clintjeff2:Cloud-deployment-add-agent-health-monitoring-and-auto-restart-on-crash-#198

Conversation

@clintjeff2

Copy link
Copy Markdown
Contributor

Motivation

  • Cloud-deployed agents can fail silently; we need health visibility, error tracking, and a simple degraded-state trigger to surface problematic agents without blocking calls.
  • Persisting errors atomically allows operators to inspect recent failures and count errors in a 24h window.

Description

  • Add an atomic JSON-backed error store at lib/agents/agent-error-store.ts that records invocation errors to ./.data/agent-errors.json, exposes recordAgentExecutionError, recordAgentExecutionSuccess, recordAgentInvocation, getAgentErrorCount24h, and getAgentHealthSummary, and marks an agent degraded after 3 consecutive failures.
  • Wrap cloud function and runtime agent executions to record invocations and errors (app/agent-functions/[id]/route.ts, lib/agent-runtime/agent.ts, lib/agent-runtime/cloud-agents.ts) so degraded agents remain callable, log a warning, and surface a degraded boolean.
  • Add an authenticated admin health endpoint GET /api/admin/agents/[id]/health that returns { agentId, status, lastSeen, errorCount24h, degraded } (app/api/admin/agents/[id]/health/route.ts).
  • Surface health info in the registry API/UI by returning errorCount24h and degraded from app/api/registry/route.ts and showing a degraded badge + Errors 24h in app/registry/page.tsx.
  • Add tests for the error store behavior covering last-seen, 24h error counts, degraded transition after 3 consecutive errors, and that a success breaks the error streak (__tests__/agent-error-store.test.ts).

Testing

  • Ran unit tests: npx vitest run __tests__/agent-error-store.test.ts __tests__/api/registry/registry.test.ts __tests__/api/agents/health.test.ts and all tests passed (3 files, 16 tests passed).
  • Typecheck: npx tsc --noEmit reported failures unrelated to these changes due to missing optional dependencies (lru-cache, @wagmi/connectors) in the workspace.
  • Full Next build: npm run build was attempted but blocked by external Google Fonts fetch failures during the Next.js build (network/font fetching issue) and not by the new code.

Closes #198

…d-auto-restart

Add agent health monitoring and degraded-state reporting
@sonarqubecloud

Copy link
Copy Markdown

@clintjeff2

Copy link
Copy Markdown
Contributor Author

@leocagli, review and merge.

1 similar comment
@clintjeff2

Copy link
Copy Markdown
Contributor Author

@leocagli, review and merge.

@clintjeff2

Copy link
Copy Markdown
Contributor Author

@leocagli, please review and merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cloud deployment: add agent health monitoring and auto-restart on crash

1 participant