Skip to content

Agent M3: query-optimization and database-assessment templates plus eval gates #330

Description

@cevheri

Goal

Complete the remaining two Phase 1 task templates on the M2 runtime and build the eval/safety gates that make all three production-ready.

Scope

Templates

  • query-optimization: EXPLAIN of the original statement, index/rewrite recommendations as R3 recommendation cards (never executed; "Apply to editor" hands ownership to the user), estimated before/after plan comparison artifact, explicit "estimates only - EXPLAIN ANALYZE is policy-denied" honesty in the report
  • database-assessment: bounded per-table profiling (progressive deepening: basic, distribution, pattern; sample thresholds + timeout fallback to catalog stats), finding vocabulary (high_null, constant, low_cardinality, fk_unindexed, suspected_pii), monitor snapshot, graded report following the profiling lexicon (completeness/uniqueness/consistency/validity)
  • Goal verifiers per template (e.g. optimize requires a before/after plan artifact; empty-result answers must deepen or state uncertainty, never invent "0 rows = answer")
  • Text ER (Mermaid erDiagram / compact relation list) as agent-facing context artifact with minimal | medium | full detail levels; phantom notation for out-of-scope FK targets
  • Profiling methodology written as a short skill/lexicon doc wired into system prompts and report.compose (same vocabulary explains findings to the user)

Eval and safety gates

  • Deterministic CI evals on AI SDK mock providers: canonical PostgreSQL/SQLite scenarios for all three templates, crash/retry/cancel scenarios
  • Scheduled/manual real-model eval job (not in PR CI)
  • Prompt-injection fixtures: hostile table/column names, comments and sample values attempting tool abuse or report manipulation
  • Policy unit gates: Planning DB executions = 0, R2+ executions = 0, duplicate executions = 0, citation present for every final finding

Acceptance criteria

  • All three templates pass their scripted eval scenarios on both reference providers.
  • Injection fixtures produce denials/neutralized text, never tool escalation.
  • Repair budget exhaustion lands in needs_input/failed with partial evidence preserved in the timeline.

Part of #325

Metadata

Metadata

Assignees

No one assigned

    Labels

    aiArtificial intelligenceenhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions