feat: Assistant test scenarios, test runner, and evaluation framework

[granludo]

Summary

Add structured test scenarios per assistant — educators can define expected prompts/conversations, run them through the real completion pipeline, and evaluate the results. This is the foundation for the evaluation framework described in #172.

Motivation

Educators need to test their assistants with realistic prompts before and after making changes. Currently there's no structured way to:

• Save a set of test prompts for an assistant
• Run them through the real pipeline (RAG, prompt processing, LLM)
• Track results over time
• Record evaluations (good/bad/mixed)

This feature is also a prerequisite for the AAC test-and-evaluate skill (#326), where the agent can generate, run, and analyze tests conversationally.

Implementation

Database (Migration 15)

Three new tables:

• assistant_test_scenarios — what to test (title, messages, expected behavior, type)
• assistant_test_runs — what happened (input, output, token usage, model, assistant snapshot)
• assistant_test_evaluations — was it good (verdict, notes, evaluator)

Service layer

backend/lamb/services/test_service.py:

• Scenarios CRUD (create, list, get, update, delete)
• Test runner: calls completion pipeline with stream=False, captures response + token usage
• Evaluations CRUD

API endpoints

Under /creator/assistant/{id}/tests/:

• Scenarios: POST/GET/PUT/DELETE
• Run: POST (execute scenarios through real pipeline)
• Results: GET (list runs with responses)
• Evaluate: POST (submit verdict)

CLI (lamb test)

lamb test scenarios <assistant-id>       List test scenarios
lamb test add <assistant-id> <title>     Add a scenario
lamb test run <assistant-id>             Run all/specific scenarios
lamb test runs <assistant-id>            List runs with results
lamb test run-detail <run-id>            Show full input + output
lamb test evaluate <run-id>              Submit evaluation
lamb test evaluations <assistant-id>     List evaluations

Test runner

Calls the same completion endpoint as production (/creator/assistant/{id}/chat/completions) with stream=False. This ensures tests exercise the real RAG, prompt processors, and connector — no shortcuts.

Related issues

• EVALS BABY... We need to think carefully howto create, execute and report evals for assistants #172 — EVALS BABY (parent issue, this is the first concrete implementation)
• [In corresponding Repository] (Evaluation Framework) Calibrate Reliability Profiles (85,90,95) #315 — Calibrate Reliability Profiles (will use this framework)
• [In corresponding Repository] (Evaluation Framework) Create&Research Gold-Standard Datasets (questions&answers) #316 — Gold-Standard Datasets (will use this framework to store/run them)
• AAC Agent-Assisted Creator (Pastor) — backend + CLI prototype #326 — AAC Pastor (will get a test-and-evaluate skill leveraging this)
• Assistant Versioning System #321 — Assistant Versioning (test runs track assistant snapshots, versioning replaces with version IDs)

[DanPenso]

I plan a small PR adding
Documentation/evaluation_design_notes.md (scenario intent, teacher-facing grounding
guidance) and a link from DOCUMENTATION_INDEX.md. Generic examples only.
Happy to adjust scope per feedback.

[DanPenso]

Opened PR #429 — docs-only evaluation design notes (purpose + scenario types + index link). Feedback welcome on scope and placement.