pipeline.md

X+SlidesBench Pipeline Internals

This page explains how the code implements each benchmark stage. The README is the user-facing command guide; this file is the implementation map for extending or debugging the pipeline.

Repo-local agent skills in skills/ mirror the same workflow at a higher level: xslidesbench-setup, xslidesbench-probes, xslidesbench-deck-generation, xslidesbench-evaluation, and xslidesbench-ablations. Use those skills for agent handoffs; use this document when changing or debugging implementation details.

Stage 1: Source Preparation

Main code:

• src/stages/s01_sources.py
• src/shared/source_text.py

The pipeline expects each case to have one source file and optional source_metadata.json. Supported text extraction paths are:

• PDF: MinerU markdown extraction by default.
• HTML: script/style/navigation removal, tag stripping, and whitespace cleanup.
• Markdown/plain text: direct UTF-8 read.

For PDFs, extract_pdf_text() chooses XSLIDESBENCH_PDF_PARSER or mineru. MINERU_API_KEY uses the hosted API, while MINERU_API_URL uses an offline or compatible endpoint. MINERU_BASE_URL defaults to the hosted base URL when left empty. Parsed markdown is cached by source-file hash under XSLIDESBENCH_MINERU_CACHE_DIR. PyMuPDF remains available only through XSLIDESBENCH_PDF_PARSER=pymupdf for explicit local debugging.

extract_source_text() applies max_chars after extraction. This limit controls cost and latency; it is not a model context limit. Metadata supplies the scene, display name, category, topic, and other source-side descriptors used by source selection and result summaries.

Stage 2: Probe Construction

Main code:

• src/stages/s02_probes.py

Probes are generated before audience conditioning. A probe is a source-grounded question-answer unit with evidence, level, modality, and theme. The generator is called k times per source or source chunk, so the final bank is a union of independent samples.

Long-source handling is in chunk_source_text(). The preferred setting for large-context models is source_chars=1000000, chunk_chars=0, and chunk_overlap=0, so it sends the whole retained source excerpt in one prompt:

• if chunk_chars <= 0 or the source is shorter than chunk_chars, the source is sent as one chunk;
• otherwise, character windows of length chunk_chars are created;
• the step size is chunk_chars - max(0, chunk_overlap), with a minimum step of 1 character;
• each chunk is prefixed as [Source chunk chars start-end];
• generated probes receive source_chunk_index and source_chunk_count in metadata.

source_chars is the maximum number of extracted source characters retained after parsing. It controls runtime and API cost; it is not the model context window. Chunking reduces request size but can remove cross-section context. For models with large context windows, keep --chunk-chars 0 for high-quality final probe generation. Use positive chunk sizes mainly when latency, budget, or endpoint limits make full-source prompting unreliable.

After all chunks and all k generations are complete, deduplicate_probes() runs hard deduplication over the full union. A pair is treated as duplicate when either evidence token overlap is at least 0.80 and answer similarity is at least 0.70, or question similarity is at least 0.90 and answer similarity is at least 0.70. Citation-list and reference-only questions are filtered before deduplication.

Per case, the stage writes:

• source_context.txt: extracted source text used by later stages;
• raw/chunk_XX/*.json: raw generation requests and responses;
• probes_deduped.jsonl: final probe bank;
• probe_generation_status.json: counts, chunk settings, and model id.

Stage 3: Audience Utility Weighting

Main code:

• src/stages/s03_weights.py

The same probe bank is weighted separately for each audience profile. The utility judge receives batches of probes, with batch_size=12 by default. For every probe-audience pair, the utility judge runs k independent judgments and snaps each raw value to {0.0, 0.3, 0.6, 1.0}. The stored utility weight is the mean of these snapped values.

The metadata keeps all sampled values, rationales, judge model id, and standard deviation. If the standard deviation is greater than 0.25, the weight row is marked as high variance so it can be inspected or re-run.

This stage writes one JSONL file per audience:

probes/<case_label>/weights/<audience>.jsonl

Stage 4: Deck Generation Or Import

Main code:

• src/generators/xslidesbench.py
• src/generators/slidetailor.py
• src/generators/notebooklm.py

Deck rows are driven by run sheets. Each row describes the system, condition, audience, case label, source document, prompt path, and preferred output deck path. The normalized run layout stores generated decks under:

data/runs/<run_name>/decks/<system>/<condition>/<audience>/<case_label>/

The X+SlidesBench local adapter runner reads the run sheet, loads environment variables, skips already completed outputs unless forced, and writes per-row status plus batch summaries. The reusable batch entrypoints live under src/tools and are intentionally thin wrappers around the public xslidesbench stage commands.

The SlideTailor workflow has two steps. First, the preparation logic builds the local generator dataset, configuration files, preference files, and slidetailor_run_sheet.csv. It currently prepares PDF sources only. Both audience-agnostic and audience-conditioned rows use template_mode=source_template_generation: each deck is generated directly from the source document and the reference template, with the row-specific preference guidelines applied. Second, the runner executes rows. It supports row filters by condition, audience, and case label. It also supports GPU parallelism through stable case sharding:

md5(case_label) % num_shards == shard_index

NotebookLM is imported manually: exported PDF decks are copied into the normalized deck tree, and evaluation then treats them like any other PDF deck.

Stage 5: Deck Evaluation

Main code:

• src/stages/s05_evaluation.py
• src/shared/decks.py
• src/shared/rendering.py

The evaluation logic expands deck rows into evaluation jobs. A conditioned deck is evaluated for its target audience. An audience-agnostic deck is evaluated for every active audience, because the same generic deck can be scored under different utility weight vectors.

Before judging, the script ensures the required audience weights exist. If a weight file is missing, it calls xslidesbench weights for that case and audience.

Deck text is extracted from PPTX or PDF. Probe answerability uses the concatenated deck text when visible text is available. judge_probes() batches multiple probes into one prompt with batch_size=12 by default. Each batch contains:

• the full extracted deck text;
• the source context;
• a JSON list of probe ids, questions, expected answers, evidence, levels, modalities, and themes.

The judge must return one judgment per probe. Missing judgments are filled as uncovered with zero confidence, preserving deterministic probe order. Existing score files are reused only when their judge_input_mode matches the current text-or-image mode.

Correctness is scored separately from answerability. The default scorer first extracts atomic factual claims from the deck, retrieves local source snippets for each claim, verifies the claims in batches, and aggregates the verification labels into the public correctness scalar. This keeps the metric name stable while avoiding the brittleness of a single all-or-nothing deck-level judgment. Speaker notes are not used by the main scoring path.

Correctness judgments are cacheable because they depend on the deck, source text, model, and claim-verification settings, but not on the audience weight vector. The evaluate command therefore writes reusable entries under a run-level correctness_cache/ directory unless --correctness-cache-dir overrides the location.

Each evaluation job writes:

• deck_stats.json and extracted visible text;
• rendered assets when needed;
• probe_scores.jsonl;
• correctness.json, including extracted claims and claim verification labels;
• metrics.json;
• run_manifest.json and status/log files.

Stage 6: Metric Scoring

Main code:

• src/stages/s06_metrics.py

Metrics join three files by probe_id: source probes, audience weights, and probe scores. Probes are included only when their audience weight is at least tau_a. This implements the audience-relevant denominator:

included probes = {probe | utility_weight(audience, probe) >= tau_a}

Audience Coverage is recovered utility divided by available utility over the included probes. Recovered utility is the weighted sum of covered probes.

Efficiency is reported with three cost normalizations:

• time: 0.25 * slide_count + visible_word_count / 130;
• slide: max(slide_count, 1);
• token/word proxy: max(visible_word_count, 1).

If correctness is available, safe_efficiency_time multiplies time-normalized efficiency by correctness. With the default claim-level scorer, contradictions and unsupported claims reduce this multiplier, but one weak claim does not automatically collapse the whole deck to zero.

Stage 7: Result Aggregation

Main code:

• src/stages/s07_aggregation.py

The generic aggregator scans metric files, infers system/audience/case identity, and groups rows by (system_id, audience). For each metric key it reports the mean and bootstrap confidence intervals over row-level results.

Benchmark-specific summaries read probes, weights, scores, and deck statistics for each metric row, then recompute domain-wise utility recovery. Domain labels are inferred from probe theme, question, answer, modality, and level using a keyword-based classifier with the domains:

context, method, evidence, limitations, implementation, implications

This keeps Domain-wise Coverage derived from the same probe-score files as Audience Coverage while exposing which kinds of information each deck covers.

Artifact Data Formats

The pipeline uses JSON for per-case metadata and aggregate outputs, and JSONL for row-oriented probe, weight, and score files. The examples below show the stable fields that downstream tools should expect. Additional metadata fields may appear as the pipeline records model ids, chunk settings, prompts, or debugging information.

Source Manifest

Source metadata identifies the input topic and its intended benchmark context. It is usually stored as source_metadata.json beside the source file.

{
  "case_id": "case_000",
  "title": "Document title",
  "source_path": "data/sources/case_000/source.pdf",
  "source_type": "pdf",
  "scene": "academic_research_talk",
  "valid_audiences": ["specialists", "learners", "decision_makers"],
  "metadata": {}
}

Probe JSONL

Each line in probes_deduped.jsonl is one source-grounded probe. Probes are audience-agnostic at generation time; audience conditioning is introduced by the weighting stage.

{
  "probe_id": "pb_0001",
  "source_case_id": "case_000",
  "scene": "academic_research_talk",
  "question": "What is the main claim?",
  "answer": "The source-supported answer.",
  "evidence": [{"locator": "Section 1", "text": "Evidence text"}],
  "level": 2,
  "modality": "text",
  "theme": "method",
  "metadata": {}
}

Audience Weight JSONL

Each weight row assigns one probe a utility value for one audience profile.

{
  "probe_id": "pb_0001",
  "audience": "specialists",
  "utility_weight": 1.0,
  "reason": "Why this probe matters for the audience.",
  "metadata": {}
}

Deck Artifacts

The deck stage stores generator outputs under system, condition, audience, and case directories. A prepared deck can be PPTX or PDF. Deck statistics are stored as JSON and point to the extracted visible text used by evaluation.

{
  "deck_path": "data/runs/run/decks/case_000.pptx",
  "slide_count": 8,
  "visible_word_count": 700,
  "extracted_text_path": "data/runs/run/decks/case_000.txt",
  "parser": "pptx",
  "warnings": []
}

Probe Score JSONL

Each score row records whether the deck contains enough visible evidence to answer one probe.

{
  "probe_id": "pb_0001",
  "covered": true,
  "answer": "Deck-supported answer.",
  "cited_evidence": "Slide 3",
  "metadata": {}
}

Metrics JSON

Compact aggregate input uses metrics/<system>/<audience>/<case>.json. The aggregator also accepts full xslidesbench evaluate output folders that contain a metrics.json file next to run_manifest.json.

{
  "audience": "specialists",
  "tau_a": 0.7,
  "included_probe_count": 42,
  "available_utility": 39.5,
  "recovered_utility": 24.0,
  "coverage": 0.6076,
  "efficiency_time": 3.2,
  "efficiency_slide": 2.0,
  "correctness": 0.9,
  "safe_efficiency_time": 2.88
}