Preview

.kiro/specs/batch-test-runner/.config.kiro

@@ -0,0 +1 @@
+{"specId": "2ae19710-3758-46c5-b3a1-9bade9b7f332", "workflowType": "requirements-first", "specType": "feature"}

.kiro/specs/batch-test-runner/design.md

@@ -0,0 +1,253 @@
+# Design Document: Batch Test Runner
+
+## Overview
+
+The Batch Test Runner adds automated grading for code questions. It has two parts:
+
+1. **Test Case Editor** — a UI component embedded in `QuestionManager` that lets instructors define `input`/`expected_output` pairs on any `code`-type question. These are stored in the existing `test_cases` JSONB column on the `questions` table.
+
+2. **Batch Run API** — a new `POST /api/instructor/exercises/[id]/batch-test-run` endpoint that fetches all final submissions for an exercise, sends each one to the Runner Service against its question's test cases, writes `tests_passed` back to the `submissions` table, then calls the existing `recalculate-scores` endpoint to update session scores.
+
+The feature is synchronous: the instructor clicks "Run Tests", waits for the response, and sees a summary. No SSE or polling is needed.
+
+---
+
+## Architecture
+
+```mermaid
+sequenceDiagram
+    participant I as Instructor browser
+    participant B as BatchTestRun API
+    participant DB as Database
+    participant R as Runner Service
+    participant S as RecalculateScores API
+
+    I->>B: POST /api/instructor/exercises/[id]/batch-test-run
+    B->>DB: Fetch final submissions with question test_cases
+    loop For each submission
+        B->>R: POST RUNNER_URL/run (code, language, stdin=test_case.input)
+        R-->>B: {stdout, exit_code, ...}
+        B->>B: Compare stdout.trim() == expected_output.trim()
+    end
+    B->>DB: UPDATE submissions SET tests_passed = ...
+    B->>S: POST /api/instructor/exercises/[id]/recalculate-scores
+    S-->>B: {recalculated: N}
+    B-->>I: {processed, passed, failed, recalculated}
+```
+
+The Test Case Editor is purely a client-side React component layered on top of the existing `PUT /api/instructor/exercises/[id]/questions/[qid]` endpoint — no new API surface is needed for it.
+
+---
+
+## Components and Interfaces
+
+### 1. `TestCaseEditor` (new React component)
+
+**Location:** `app/instructor/exercises/[id]/TestCaseEditor.tsx`
+
+**Props:**
+```ts
+interface Props {
+  exerciseId: string;
+  questionId: string;
+  initialTestCases: TestCase[];
+  onSaved?: (testCases: TestCase[]) => void;
+}
+
+interface TestCase {
+  input: string;
+  expected_output: string;
+}
+```
+
+**Behaviour:**
+- Only rendered when the parent passes a `code`-type question
+- Displays a list of existing test cases (input + expected_output)
+- Add row: two text inputs + "Add" button; validates that `input` is non-empty before saving
+- Edit in-place: click a row to make it editable
+- Delete: remove a row and save immediately
+- All mutations send the full updated `test_cases` array via `PUT /api/instructor/exercises/[id]/questions/[qid]`
+- Shows a toast on success/failure
+
+**Integration:** `QuestionManager` renders `<TestCaseEditor>` in the expanded question panel, below the starter-code section, only when `q.type === 'code'`.
+
+### 2. `BatchRunButton` (new React component)
+
+**Location:** `app/instructor/exercises/[id]/submissions/BatchRunButton.tsx`
+
+**Props:**
+```ts
+interface Props {
+  exerciseId: string;
+  onComplete?: () => void;  // called after successful run so parent can refresh
+}
+```
+
+**States:** idle → loading → success summary | error
+
+**Behaviour:**
+- Renders a "Run Tests" button
+- On click: disables button, shows spinner, calls `POST /api/instructor/exercises/[id]/batch-test-run`
+- On success: shows `"Processed N submissions — P passed, F failed"` and calls `onComplete`
+- On error: shows error message, re-enables button
+
+**Integration:** Added to `app/instructor/exercises/[id]/submissions/page.tsx` alongside the Export CSV button. `onComplete` triggers a router refresh (`router.refresh()`) to reload updated scores.
+
+### 3. `POST /api/instructor/exercises/[id]/batch-test-run` (new API route)
+
+**Location:** `app/api/instructor/exercises/[id]/batch-test-run/route.ts`
+
+**Request:** No body required.
+
+**Response (200):**
+```ts
+{
+  processed: number;   // total submissions evaluated
+  passed: number;      // submissions where tests_passed = true
+  failed: number;      // submissions where tests_passed = false
+  recalculated: number; // sessions rescored (from recalculate-scores)
+}
+```
+
+**Error responses:**
+- `401` — no session
+- `403` — session role is not `instructor`
+- `404` — exercise not found
+- `503` — `RUNNER_URL` not configured
+
+---
+
+## Data Models
+
+No schema changes are required. The feature uses three existing columns:
+
+| Table | Column | Type | Purpose |
+|-------|--------|------|---------|
+| `questions` | `test_cases` | `JSONB` | Array of `{input, expected_output}` objects |
+| `submissions` | `tests_passed` | `boolean` | Set by batch runner; used by scoring |
+| `submissions` | `is_final` | `boolean` | Gate: only final submissions are processed |
+
+The `test_cases` column already exists and is already consumed by `lib/scoring.ts` when computing whether a code submission counts as a passing answer.
+
+### TestCase shape (stored in `questions.test_cases`):
+```json
+[
+  { "input": "Hello\nWorld", "expected_output": "dlroW\nolleH" },
+  { "input": "abc", "expected_output": "cba" }
+]
+```
+
+---
+
+## Correctness Properties
+
+*A property is a characteristic or behavior that should hold true across all valid executions of a system — essentially, a formal statement about what the system should do. Properties serve as the bridge between human-readable specifications and machine-verifiable correctness guarantees.*
+
+### Property 1: Test case validation rejects empty input
+
+*For any* string composed entirely of whitespace characters (including the empty string), attempting to save it as a test case `input` field SHALL be rejected without issuing a PUT request, and the test case list SHALL remain unchanged.
+
+**Validates: Requirements 1.5**
+
+---
+
+### Property 2: Test case list mutation preserves unmodified entries
+
+*For any* existing list of test cases, adding, editing, or deleting a single entry SHALL result in a saved list where all entries not targeted by the mutation are present and unchanged.
+
+**Validates: Requirements 1.2, 1.3, 1.4**
+
+---
+
+### Property 3: Submission selection filter
+
+*For any* exercise, the batch runner SHALL process exactly the set of submissions where `is_final = true` AND the corresponding question has at least one test case — no more, no less.
+
+**Validates: Requirements 2.1, 4.3, 4.4**
+
+---
+
+### Property 4: tests_passed reflects unanimous pass
+
+*For any* submission run against N test cases, `tests_passed` SHALL equal `true` if and only if every individual test case result has `actual.trim() === expected.trim()`; otherwise it SHALL be `false`.
+
+**Validates: Requirements 2.3, 2.4, 2.6**
+
+---
+
+### Property 5: Batch run idempotence
+
+*For any* set of submissions, running the batch runner twice in sequence SHALL produce the same `tests_passed` values on all submissions as running it once — the second run overwrites without accumulation.
+
+**Validates: Requirements 2.8, 4.1**
+
+---
+
+### Property 6: Response counts are consistent
+
+*For any* batch run, the returned JSON SHALL satisfy: `passed + failed === processed`, all four fields (`processed`, `passed`, `failed`, `recalculated`) are present and non-negative integers.
+
+**Validates: Requirements 2.9**
+
+---
+
+### Property 7: Exercise scope isolation
+
+*For any* batch run targeting exercise A, the `tests_passed` column on submissions belonging to any other exercise SHALL remain unmodified.
+
+**Validates: Requirements 5.3**
+
+---
+
+## Error Handling
+
+| Condition | Behaviour |
+|-----------|-----------|
+| `RUNNER_URL` not set | Return `503` immediately; no submissions processed |
+| Runner returns non-2xx | Mark submission `tests_passed = false`; continue to next |
+| Runner network error / timeout | Mark submission `tests_passed = false`; continue |
+| `response_text` is null or blank | Mark `tests_passed = false`; skip runner call |
+| Question has no test cases | Skip all submissions for that question; do not touch `tests_passed` |
+| Exercise not found | Return `404` |
+| No valid instructor session | Return `403` |
+| `recalculate-scores` fails | Log error, return partial result with a warning field; do not fail the whole request |
+
+Runner errors are per-submission failures, not fatal. The batch run always completes and always calls recalculate-scores (unless the exercise is not found or auth fails).
+
+---
+
+## Testing Strategy
+
+### Unit tests (example-based)
+
+- `TestCaseEditor` renders existing test cases from props
+- `TestCaseEditor` is hidden when `question.type === 'written'`
+- `TestCaseEditor` shows success toast on successful save
+- `BatchRunButton` renders in idle state
+- `BatchRunButton` disables and shows spinner while loading
+- `BatchRunButton` displays summary on success
+- `BatchRunButton` shows error and re-enables on failure
+- Batch run API returns `403` when called without instructor session
+- Batch run API returns `404` for unknown exercise ID
+- Batch run API returns `503` when `RUNNER_URL` is not configured
+
+### Property-based tests
+
+Use [fast-check](https://github.com/dubzzz/fast-check) (already compatible with the project's TypeScript/Node stack). Each property test runs a minimum of **100 iterations**.
+
+Tag format: `// Feature: batch-test-runner, Property N: <property text>`
+
+| Property | What to generate | What to assert |
+|----------|-----------------|----------------|
+| P1: Empty input rejected | Arbitrary whitespace-only strings (including `""`) | No PUT fired; list unchanged |
+| P2: Mutation preserves unmodified entries | Random test case arrays; random mutation target | All non-mutated entries survive intact |
+| P3: Submission selection filter | Random mix of final/non-final submissions and questions with/without test cases | Processed set equals expected filtered set |
+| P4: tests_passed unanimous pass | Random arrays of `{actual, expected}` pairs | `tests_passed === arr.every(r => r.actual.trim() === r.expected.trim())` |
+| P5: Idempotence | Random submissions + deterministic mock runner | Two runs yield identical `tests_passed` for all submissions |
+| P6: Response counts consistent | Random submission arrays | `passed + failed === processed`; all four fields present |
+| P7: Exercise scope isolation | Two exercises A and B with overlapping users | After running on A, B submissions unchanged |
+
+### Integration test
+
+One integration test (requires DB + `RUNNER_URL`):
+- Create an exercise with a code question with two test cases, create a session with a final submission, run batch test, verify `tests_passed` is set correctly and `score` is updated on the session.

.kiro/specs/batch-test-runner/requirements.md

@@ -0,0 +1,98 @@
+# Requirements Document
+
+## Introduction
+
+The Batch Test Runner feature allows instructors to define automated test cases on code questions and then batch-execute all student submissions for an exercise against those test cases. After running, it saves each submission's pass/fail result and triggers a score recalculation for all participants. This replaces the current manual-override-only workflow for code exercises with an objective, automated grading path.
+
+The feature integrates with the existing `/api/run-code` infrastructure (external RUNNER_URL service), the `test_cases` JSONB column on questions, the `tests_passed` boolean column on submissions, and the existing `recalculate-scores` endpoint.
+
+## Glossary
+
+- **Batch_Runner**: The backend endpoint (`POST /api/instructor/exercises/[id]/batch-test-run`) that orchestrates running all submissions for an exercise against their question's test cases and saving results.
+- **Test_Case_Editor**: The UI component embedded in QuestionManager that allows instructors to add, edit, and delete input/expected_output pairs on a code question.
+- **Test_Case**: A JSON object with `input` (string) and `expected_output` (string) representing one automated test for a code question.
+- **Exercise**: A collection of questions with associated sessions and submissions on the exam platform.
+- **Question**: A single problem within an exercise; may be of type `code` or `written`. Only code questions support test cases.
+- **Submission**: A student's response to one question within a session, stored in the `submissions` table.
+- **Session**: A student's participation record for an exercise, stored in the `sessions` table.
+- **Runner_Service**: The external HTTP service at `RUNNER_URL` that compiles and executes code.
+- **Recalculate_Scores**: The existing `POST /api/instructor/exercises/[id]/recalculate-scores` endpoint that recomputes scores for all sessions of an exercise.
+- **tests_passed**: The boolean column on the `submissions` table that records whether a submission passed all test cases.
+- **Batch_Progress_UI**: The React component on the submissions page that displays real-time progress of a batch run to the instructor.
+
+---
+
+## Requirements
+
+### Requirement 1: Define Test Cases on Code Questions
+
+**User Story:** As an instructor, I want to add input/expected_output test cases to a code question, so that submissions can be graded automatically rather than relying solely on manual review.
+
+#### Acceptance Criteria
+
+1. WHEN an instructor views a code question in the QuestionManager, THE Test_Case_Editor SHALL display the current list of test cases for that question.
+2. WHEN an instructor adds a test case with a non-empty `input` and `expected_output`, THE Test_Case_Editor SHALL save the updated test cases to the question's `test_cases` column via a PUT request to `/api/instructor/exercises/[id]/questions/[qid]`.
+3. WHEN an instructor deletes a test case, THE Test_Case_Editor SHALL remove that entry and save the updated list.
+4. WHEN an instructor edits an existing test case, THE Test_Case_Editor SHALL update that entry and save the updated list.
+5. IF the `input` field is empty when saving a test case, THEN THE Test_Case_Editor SHALL reject the entry and display a validation error message.
+6. THE Test_Case_Editor SHALL only be visible for questions where `type === 'code'`.
+7. WHEN test cases are saved successfully, THE Test_Case_Editor SHALL display a success notification.
+
+---
+
+### Requirement 2: Batch Test Run API Endpoint
+
+**User Story:** As an instructor, I want a single endpoint that runs all final submissions for an exercise against their question's test cases, so that I can grade everyone at once without manual intervention.
+
+#### Acceptance Criteria
+
+1. WHEN a POST request is made to `/api/instructor/exercises/[id]/batch-test-run` by an instructor, THE Batch_Runner SHALL fetch all final submissions for the exercise that belong to questions with at least one test case.
+2. FOR EACH submission fetched, THE Batch_Runner SHALL retrieve the corresponding question's test cases and submit the submission's `response_text` and question's `language` to the Runner_Service.
+3. WHEN the Runner_Service returns results for all test cases of a submission, THE Batch_Runner SHALL set `tests_passed = true` on that submission if and only if every test case output matches the expected output.
+4. WHEN the Runner_Service returns results where at least one test case output does not match, THE Batch_Runner SHALL set `tests_passed = false` on that submission.
+5. WHEN all submissions have been processed, THE Batch_Runner SHALL call the Recalculate_Scores endpoint for the exercise.
+6. IF a submission's `response_text` is empty or null, THEN THE Batch_Runner SHALL set `tests_passed = false` without calling the Runner_Service.
+7. IF the Runner_Service is unavailable (RUNNER_URL not configured), THEN THE Batch_Runner SHALL return a 503 response with an explanatory error message.
+8. THE Batch_Runner SHALL be idempotent — re-running it SHALL update `tests_passed` on all matching submissions, overwriting previous results.
+9. WHEN the batch run completes, THE Batch_Runner SHALL return a JSON response containing `processed` (count of submissions evaluated), `passed` (count that passed all tests), `failed` (count that failed), and `recalculated` (count of sessions rescored).
+
+---
+
+### Requirement 3: Real-Time Progress Feedback in the UI
+
+**User Story:** As an instructor, I want to see progress while the batch run executes, so that I know the operation is working and can estimate when it will finish.
+
+#### Acceptance Criteria
+
+1. THE Batch_Progress_UI SHALL display a "Run Tests" button on the exercise submissions page.
+2. WHEN the instructor clicks "Run Tests", THE Batch_Progress_UI SHALL disable the button and display a loading indicator.
+3. WHILE the batch run is in progress, THE Batch_Progress_UI SHALL show a status message indicating the operation is running.
+4. WHEN the Batch_Runner responds with a success result, THE Batch_Progress_UI SHALL display the summary: number of submissions processed, number passed, and number failed.
+5. IF the Batch_Runner returns an error, THEN THE Batch_Progress_UI SHALL display the error message and re-enable the "Run Tests" button.
+6. WHEN the batch run completes successfully, THE Batch_Progress_UI SHALL reload the submissions data to reflect updated scores.
+7. THE Batch_Progress_UI SHALL only be visible to instructors.
+
+---
+
+### Requirement 4: Idempotent Results and Score Integration
+
+**User Story:** As an instructor, I want to re-run the batch test at any time and always get up-to-date results, so that I can re-grade after editing test cases or after students resubmit.
+
+#### Acceptance Criteria
+
+1. WHEN the Batch_Runner processes a submission that already has a `tests_passed` value, THE Batch_Runner SHALL overwrite the existing value with the newly computed result.
+2. WHEN the Batch_Runner completes, THE Recalculate_Scores endpoint SHALL use the updated `tests_passed` values to compute each session's score, consistent with the existing scoring logic for code questions with test cases.
+3. WHEN a question has no test cases defined, THE Batch_Runner SHALL skip all submissions for that question without setting `tests_passed` on them.
+4. WHEN a question has test cases defined but a submission is not final, THE Batch_Runner SHALL skip that submission without setting `tests_passed` on it.
+
+---
+
+### Requirement 5: API Authorization and Safety
+
+**User Story:** As a platform administrator, I want the batch test run endpoint to be protected, so that only instructors can trigger batch operations.
+
+#### Acceptance Criteria
+
+1. WHEN a request to `/api/instructor/exercises/[id]/batch-test-run` is made without a valid instructor session, THE Batch_Runner SHALL return a 403 response.
+2. WHEN the exercise ID in the request does not correspond to an existing exercise, THE Batch_Runner SHALL return a 404 response.
+3. THE Batch_Runner SHALL not execute any submissions for exercises it does not own in the current request context — it SHALL only process submissions belonging to the exercise specified in the URL.