Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Documentation/DOCUMENTATION_INDEX.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ This index helps you find exactly what you need in the LAMB documentation. Start
| Add Knowledge Base support | [lamb_architecture_v2.md](./lamb_architecture_v2.md) | §8.1 Knowledge Base |
| Implement LTI integration | [lamb_architecture_v2.md](./lamb_architecture_v2.md) | §8.2 LTI Integration |
| Add assistant sharing | [lamb_architecture_v2.md](./lamb_architecture_v2.md) | §8.3 Assistant Sharing |

| Design assistant test scenarios | [evaluation_design_notes.md](./evaluation_design_notes.md) | Full doc |
---

## 📁 Documentation Map
Expand Down
35 changes: 35 additions & 0 deletions Documentation/evaluation_design_notes.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# Evaluation Design Notes

Notes for anyone building or reviewing assistant test scenarios in LAMB. Related to [#327](https://github.com/Lamb-Project/lamb/issues/327) and [#172](https://github.com/Lamb-Project/lamb/issues/172).


---

## Purpose

Educators change system prompts, knowledge bases, and models often. Without saved test scenarios and recorded runs, it is hard to see whether a change helped or made things worse.

LAMB runs tests through the same completion pipeline as production (RAG, prompt processors, LLM). Each run stores the input, response, and an assistant configuration snapshot for later comparison.

---

## Scenario types (evaluation intent)

When writing scenarios in the Assistant Control Centre, match the scenario to what you want to learn.

| Intent | What you are testing | Example student question (fictional) |
|--------|----------------------|--------------------------------------|
| **Grounded answer** | Reply stays faithful to course materials | "What is a food chain?" |
| **KB miss** | Safe behaviour when retrieval is weak or empty | "Does quantum physics affect photosynthesis in this course?" |
| **Partial context** | No over-claiming when only fragments are retrieved | "Summarise the whole ecology unit in one paragraph." |
| **Tone / register** | Voice matches the pedagogical role in the system prompt | "lol idk what mitosis is" |

The platform also labels scenario **type**: Normal (`single_turn`), Multi-turn (`multi_turn`), and Adversarial (`adversarial`). Use adversarial scenarios for off-topic or misleading prompts where refusal or redirection is the expected behaviour.

---

## Next steps (planned)

- Verdict labels for teachers (good / bad / mixed and grounding vocabulary)
- Symmetric evaluation when comparing two assistant configurations
- Fictional worked examples