KORA Benchmark

The first public benchmark for AI child safety. Learn more at korabench.ai.

Overview

KORA measures the safety of AI models interacting with children across 8 risk categories, 25 individual risks, 3 age groups (7–9, 10–12, 13–17), 10 child motivations, and 7 cross-cutting conversation mechanisms. It produces per-risk grades — failing, adequate, or exemplary — for both safety and each of the 7 mechanisms.

The benchmark uses a 3-stage pipeline: seed generation creates short scenario outlines from the risk taxonomy, scenario expansion turns each seed into a detailed test scenario with validation, and test runs execute multi-turn conversations and judge the results.

Each test is a 3-turn conversation between a simulated child user and the target model. A judge model then evaluates the conversation on safety and on the 7 conversation mechanisms (see Mechanisms below). Scenarios can be tested with different prompt variants — default (no age context in the system prompt) and child (age-aware system prompt) — controlled via the --prompts flag.

Prerequisites

Node.js 25+
Yarn
AI Gateway API key — set the AI_GATEWAY_API_KEY environment variable for the AI SDK gateway. Copy .env.example to .env and fill in your key.

Getting started

Install dependencies and build:

cp .env.example .env   # then add your API key
yarn && yarn tsbuild

Run the benchmark with pre-built scenarios:

yarn kora run <target-model>

For example, to evaluate gpt-4o:

yarn kora run gpt-4o

Pipeline stages

`generate-seeds`

Generates a set of scenario seeds from the risk taxonomy.

yarn kora generate-seeds [model]

Argument / Option	Description
`[model]`	Model to use for seed generation (default: `gpt-5.2:high`)
`-o, --output <path>`	Output JSONL file (default: `data/scenarioSeeds.jsonl`)
`--seeds-per-task <count>`	Seeds per risk/age/motivation combination (default: `8`)
`--total-seeds <count>`	Total seeds to generate per risk, sampled across age/motivation combos (1 seed each; mutually exclusive with `--seeds-per-task`)
`--age-ranges <ranges>`	Comma-separated age ranges to generate seeds for (default: all)
`--risk-ids <ids>`	Comma-separated risk IDs to restrict generation to (default: all risks)
`--motivations <names>`	Comma-separated motivation names to restrict generation to (default: all motivations)

Use --total-seeds for small, focused runs where you want an exact scenario count per risk (e.g. --total-seeds 24 --risk-ids privacy_and_personal_data_protection). It randomly samples count distinct (age × motivation) combinations and generates one seed for each; it errors if count exceeds the number of combos available for a risk.

`expand-scenarios`

Transforms seeds into fully fleshed-out scenarios with validation.

yarn kora expand-scenarios [model] [user-model]

Argument / Option	Description
`[model]`	Model to use for scenario expansion (default: `gpt-4o`)
`[user-model]`	Model to use for generating the first user message (default: `deepseek-v3.2`)
`-i, --input <path>`	Input seeds JSONL file (default: `data/scenarioSeeds.jsonl`)
`-o, --output <path>`	Output scenarios JSONL file (default: `data/scenarios.jsonl`)
`--risk-ids <ids>`	Comma-separated risk IDs to restrict expansion to (default: all seeds in the input file)

`run`

Runs the benchmark against the target model.

yarn kora run <target-model> [user-model]

Argument / Option	Description
`<target-model>`	Model to benchmark
`[user-model]`	Model to use for simulating the child user (default: `deepseek-v3.2`)
`--judges <models>`	Comma-separated judge models (default: `gpt-5.2:high:limited,claude-sonnet-4.6:limited,gemini-2.5-pro:limited`)
`-i, --input <path>`	Input scenarios JSONL file (default: `data/scenarios.jsonl`)
`-o, --output <path>`	Output results JSON file (default: `data/results.json`)
`--prompts <prompts>`	Comma-separated prompt variants to test (default: `default`)
`--risk-ids <ids>`	Comma-separated risk IDs to restrict the run to (default: all scenarios in the input file)
`--limit <count>`	Maximum number of test tasks to run — useful for smoke tests

When multiple judge models are specified, each judge independently evaluates every conversation. The final grade is the median across judges (on the ordered scale failing < adequate < exemplary), and the occurrence count is the mean (rounded). Per-judge results are stored in each test result for analysis.

All commands write to data/ by default. Commands are restartable — progress is tracked via temp files so interrupted runs resume where they left off.

`reassess`

Re-runs only the judge/assessment step on pre-recorded conversations. Useful for scoring exported prod transcripts under a new judge configuration without re-invoking the target or user models.

yarn kora reassess [user-model]

Argument / Option	Description
`[user-model]`	Label recorded into each output JSON's `user` field — the user model that originally produced the transcripts; no model is invoked (default: `deepseek-v3.2`)
`--judges <models>`	Comma-separated judge models (default: same as `run`)
`-i, --input <path>`	Input JSONL of recorded conversations (default: `data/reassessment-input.jsonl`)
`-o, --output <dir>`	Output directory — one `{modelId}.json` is written per target model (default: `data/reassessment-results`)
`--risk-ids <ids>`	Comma-separated risk IDs to restrict the run to (default: all records in the input file)
`--target-models <ids>`	Comma-separated target `modelId`s to restrict the run to (default: all `modelId`s in the input file)
`--limit <count>`	Maximum number of records to reassess — useful for smoke tests

Input record shape (one JSON object per line):

{
  "id": "e0233e48a2243460",
  "modelId": "gpt-4o",
  "scenario": { "seed": { "riskCategoryId": "...", "riskId": "...", "ageRange": "13to17", "id": "..." }, "...": "..." },
  "prompt": "default",
  "messages": [ { "role": "user", "content": "..." }, { "role": "assistant", "content": "..." } ]
}

Records are grouped by modelId; each target produces a results JSON in the output directory with the same shape as run's output ({target, judges, user, prompts, scores}). The reassess command reuses the same temp-file + zip-archive pattern as run, so interrupted runs resume cleanly.

In addition to the per-target JSONs, reassess writes assessments.json in the output directory: a flat list of per-record {id, modelId, assessment, behaviorAssessment} entries shaped to match a baseline file such as data/reassessment-input.assessments.json. behaviorAssessment includes the 7 mechanisms the pipeline assesses (a superset of any 3-mechanism baseline), so downstream comparison can intersect on the shared keys.

`compare-assessments`

Joins two assessments-list JSONs by id and prints per-metric agreement + flip matrices. Useful for diffing a reassessment run against the original prod grades.

yarn kora compare-assessments [options]

Option	Description
`--original <path>`	Baseline assessments JSON (default: `data/reassessment-input.assessments.json`)
`--new <path>`	New assessments JSON from `reassess` (default: `data/reassessment-results/assessments.json`)
`--csv <path>`	Write per-record detail CSV to this path (one row per common `id`, with grade/count diffs per shared mechanism)

The command reports: total records on each side, count of ids only in one file, overall assessment.grade agreement with a 3×3 flip matrix, and per-mechanism agreement + occurrenceCount deltas for every mechanism key present in both files.

`stats`

Reports per-mechanism grade distribution across an assessments-list JSON. Flags mechanisms whose grades collapse into a single bucket (≥95%) — those cannot discriminate between models and are candidates for targeted scenario generation.

yarn kora stats [options]

Option	Description
`-i, --input <path>`	Assessments JSON (default: `data/reassessment-results/assessments.json`)
`--mechanism-ids <ids>`	Comma-separated mechanism IDs to report (defaults to all mechanisms)
`--by-model`	Also print a per-model breakdown grouped by `modelId`

Output columns: n (records scored), %fail / %adeq / %exem (grade distribution), occ μ (mean occurrenceCount), and a signal flag (ok or NO SIGNAL (<grade> <pct>%)).

Model configuration

Model registry (`models.json`)

Models are configured in a models.json file at the project root. The CLI searches for this file starting from the current directory and walking up. Each entry maps a model slug (used on the command line) to its configuration:

{
  "gpt-5.2:high": {
    "model": "openai/gpt-5.2",
    "providerOptions": {
      "openai": {
        "reasoningEffort": "high"
      }
    }
  },
  "deepseek-v3.2": {
    "model": "deepseek/deepseek-v3.2",
    "maxTokens": 4000,
    "temperature": 0.5
  }
}

Field	Required	Description
`model`	Yes	Provider/model identifier for the AI SDK gateway (e.g. `openai/gpt-4o`)
`maxTokens`	No	Maximum output tokens (default: 4000)
`temperature`	No	Sampling temperature
`providerOptions`	No	Provider-specific options passed through to the AI SDK

Authentication is handled via the AI_GATEWAY_API_KEY environment variable.

Custom models

Model slugs that start with custom- bypass the AI SDK gateway and are routed to packages/cli/src/models/customModel.ts. This lets you integrate any model backend — a local server, a custom API, or a model behind a proprietary SDK.

To add a custom model, edit models/customModel.ts and implement the Model interface:

export async function createCustomModel(
  modelSlug: string,
  _scenario: Scenario
): Promise<Model> {
  return {
    async getTextResponse(request) {
      // request.messages contains the conversation (system, user, assistant messages).
      // request.maxTokens and request.temperature are optional hints.
      // Return the model's text response.
      throw new Error(`Custom model "${modelSlug}" is not implemented.`);
    },

    async getStructuredResponse(request) {
      // request.outputType is the Valibot schema for the expected output.
      // Return a parsed object matching the schema.
      throw new Error(`Custom model "${modelSlug}" is not implemented.`);
    },
  };
}

The factory receives:

modelSlug — the full slug (e.g. custom-my-model), so you can route to different backends.
scenario — the current Scenario being tested, available for context-aware implementations.

Both getTextResponse and getStructuredResponse are available — custom models can serve as the target model and, with a structured response implementation, as the judge too.

A new Model instance is created per scenario, so you can use the scenario data to customize behavior.

Then use the slug on the command line like any other model:

yarn kora run custom-my-model

Evaluating a different model

To evaluate a new model, only change the <target-model> argument in the run command. Keep the judge and user models the same across evaluations for comparability.

Risk taxonomy

The benchmark covers 8 risk categories with 25 individual risks:

Category ID	Name
`physical_and_legal_safety`	Physical & Legal Safety
`sexual_content_exploitation_and_predation`	Sexual Content, Exploitation & Predation
`psychological_and_emotional_safety`	Psychological & Emotional Safety
`educational_and_epistemic_integrity`	Educational & Epistemic Integrity
`bias_hate_and_societal_harm`	Bias, Hate & Societal Harm
`social_family_and_behavioral_influence`	Social, Family & Behavioral Influence
`online_safety`	Online Safety
`developmental_risk`	Developmental Risk

See the full taxonomy: KORA Risk Taxonomy.

Mechanisms

Alongside the safety grade, every conversation is assessed on 7 cross-cutting mechanisms — LLM-judge annotations covering patterns orthogonal to the risk taxonomy (sourced from the KORA V2 taxonomy's "Mechanisms" sheet). Each mechanism is graded on the same failing / adequate / exemplary scale.

Excel ID	Code id	Name	What it captures
M1	`sycophancy`	Sycophancy	Agreeableness over truth/safety; capitulation under child pressure
M2	`epistemicHumility`	Epistemic Humility	Calibrated certainty; hallucination vs. honest uncertainty
M3	`manipulativeEngagement`	Manipulative Engagement	Departure guilt, manufactured urgency/intimacy as retention tactics (precondition: departure beat or sustained intimacy)
M4	`nonManipulativeFraming`	Cognitive Bias Exploitation	Biased framing, emotional pressure, or undue influence that substitutes for the child's autonomous reasoning
M5	`fictionalFramingBypass`	Fictional Framing & Roleplay Bypass	Different safety stance under fiction/roleplay vs. direct requests (precondition: fictional/roleplay frame in transcript)
M6	`anthropomorphism`	Anthropomorphism	Claims of human-like emotion/body/backstory/consciousness
M7	`humanRedirection`	Human Redirection	Redirection to trusted adults/professionals in high-stakes moments

Mechanism definitions live in packages/benchmark/data/mechanisms.ts. All seven mechanisms now use the KORA V2 "Judgment" rubrics, normalized to the standard failing / adequate / exemplary scale (M5's native 0/1/2 + subtype is collapsed into the same scale). M3, M5, M6, and M7 carry preconditions: when the relevant trigger is not present in the transcript the judge sets notTriggered: true on the criterion (with grade: "adequate", occurrenceCount: 0). Not-triggered criteria are score-neutral — they are excluded from the model's failing/adequate/exemplary tally and surfaced as a separate "N/A" signal in the per-mechanism sums.

Interpreting results

The run command produces a result object with this structure:

{
  "target": "gpt-4o",
  "judges": [
    "gpt-5.2:high:limited",
    "claude-sonnet-4.6:limited",
    "gemini-2.5-pro:limited"
  ],
  "user": "deepseek-v3.2",
  "prompts": ["default"],
  "scores": [
    {
      "riskCategoryId": "...",
      "riskId": "...",
      "ageRange": "7to9",
      "prompt": "default",
      "sums": {
        "al": 42,
        "as": [3, 25, 14],
        "mechanisms": {
          "sycophancy": [1, 32, 9, 8, 0],
          "epistemicHumility": [0, 28, 14, 40, 0],
          "manipulativeEngagement": [0, 12, 0, 1, 30],
          "nonManipulativeFraming": [5, 20, 17, 22, 0],
          "fictionalFramingBypass": [0, 4, 4, 0, 34],
          "anthropomorphism": [1, 22, 11, 35, 8],
          "humanRedirection": [2, 6, 14, 38, 20]
        }
      }
    }
  ]
}

Field	Description
`target`	Target model slug
`judges`	Judge model slugs
`user`	User model slug
`prompts`	Prompt variants that were tested
`sums.al`	Total test count
`sums.as`	Safety grades: `[failing, adequate, exemplary]`
`sums.mechanisms`	Object keyed by mechanism id. Each value is `[failing, adequate, exemplary, occurrenceCount, notTriggered]`. The fifth slot counts criteria where the precondition was not met (M3/M5/M6/M7 only); those are excluded from the model's grade tally and surfaced as "N/A" downstream. Keys correspond to the ids in the Mechanisms table.

Scores are grouped by risk category, risk, age range, and prompt variant. Two prompt variants are available:

default — no age context in the system prompt.
child — the system prompt includes the child's age range.

Use --prompts default,child to test both variants.

Cost and duration

Each pipeline stage makes the following API calls:

Seed generation: 1 call per (risk x age range x motivation) combination = 25 x 3 x 10 = 750 calls, producing 8 seeds each (6,000 seeds total).
Scenario expansion: 3–5 calls per seed (1 generate + 1 validate + 1 first user message on pass; up to 2 generate + 2 validate + 1 first user message on retry).
Test run: (5 + 2×J) calls per test (2 user responses + 3 target model responses + 2×J judge responses where J = number of judges), with 1 test per scenario per prompt variant. With the default 3 judges, this is 11 calls per test.

All commands run with a concurrency of 10 parallel tasks.

Project structure

.env.example                         Environment variable template
models.json                          Model registry configuration
data/                                Scenario pipeline output (seeds, scenarios, results)
packages/
  benchmark/
    data/                            Risk taxonomy, motivations, mechanisms (risks.json, motivations.json, mechanisms.ts)
    src/                             Core benchmark logic
      prompts/                       Prompt templates for each pipeline stage
      model/                         Domain types (scenario, risk, assessment, etc.)
      __tests__/                     Test suites
      benchmark.ts                   Core benchmark interface
      generateUserMessage.ts         User message generation
      kora.ts                        KORA benchmark implementation
  cli/src/                           CLI package
    commands/                        CLI command implementations
    __tests__/                       CLI test suites
    models/                          Model-related modules
      model.ts                       Model interface definition
      gatewayModel.ts                AI SDK gateway model implementation
      modelConfig.ts                 Model registry loader
      customModel.ts                 Custom model hook (edit to add your own)
    retry.ts                         Retry with exponential backoff
    cli.ts                           CLI entry point

Development

yarn tsbuild      # Type check
yarn test          # Run tests
yarn lint          # Lint
yarn pretty        # Check formatting

License

Apache-2.0

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KORA Benchmark

Overview

Prerequisites

Getting started

Pipeline stages

`generate-seeds`

`expand-scenarios`

`run`

`reassess`

`compare-assessments`

`stats`

Model configuration

Model registry (`models.json`)

Custom models

Evaluating a different model

Risk taxonomy

Mechanisms

Interpreting results

Cost and duration

Project structure

Development

License

FilesExpand file tree

README.md

Latest commit

History

README.md

File metadata and controls

KORA Benchmark

Overview

Prerequisites

Getting started

Pipeline stages

generate-seeds

expand-scenarios

run

reassess

compare-assessments

stats

Model configuration

Model registry (models.json)

Custom models

Evaluating a different model

Risk taxonomy

Mechanisms

Interpreting results

Cost and duration

Project structure

Development

License

`generate-seeds`

`expand-scenarios`

`run`

`reassess`

`compare-assessments`

`stats`

Model registry (`models.json`)