Expand wvs-swe from 3 to 6 documents - #2
Draft
irony wants to merge 5 commits into
Draft
Conversation
Snapshot of the validated length-matched 3-document dataset ahead of adding three new genres (news article, policy PM, email thread). Kept so the v2 analysis remains reproducible while the corpus grows. The 3-doc set stays canonical until the expanded run is validated.
Adds three new genres alongside the existing minutes, interview transcript and meeting notes: internal news article (nyhetsbrev_01), HR policy memo (policy_pm_01), and planning email thread (mejltrad_01). Each document keeps the same structure — 8 WVS value statements + 6 length-matched neutral controls, every span verbatim-verified against the text. Motivation: the per-document gap varied sharply (−4 / −23 / −32 pp), so three documents was the real generalisability weakness. Six documents across more genres gives a firmer base for the survival-gap claim and fills the thin upper length band. The 3-document set is preserved at documents.3docs-backup.json. The expanded run (2026-08-12...-wvs-swe-v3-6docs, 162 configs) is in progress and will be validated separately before it becomes canonical.
Expanded run across six documents (9 models x 3 personas x 6 docs = 162 configs) plus a policy-memo re-run with strengthened control spans. Findings: - Five workplace documents (minutes, interview, notes, news article, email) show a consistent short-version gap of -19 to -25 pp (values trimmed more than neutral facts), all separable from zero. - The policy memo is a genre boundary: the model trimmed administrative detail harder than the values (+37 to +53 pp), because a policy summary is expected to keep only substance. Strengthening the control spans did not change this — it is a property of the genre, not the spans. Reported separately, not averaged in. - Two independent judges (gpt-oss-120b, Mistral Small) re-scored every item: agreement 85-89%, Cohen's kappa 0.65-0.73 with the primary judge. documents.json holds the six-document corpus (8 value + 6 neutral per doc, all spans verbatim-verified). The 3-document set is preserved at documents.3docs-backup.json.
…r run Adds a multi-provider layer to chat_completion: models prefixed "claude-" are routed to the Anthropic Messages API (system-hoisting, finish_reason normalisation, retry/backoff), everything else keeps the OpenAI-compatible endpoint. This lets the same harness drive frontier models alongside the open ones, judged by the same Kimi K3 panel. Claude Opus 5 result (6 docs x 3 personas, five workplace documents for the headline): short-version gap -26.3 pp [-32.1, -21.5] — squarely in the same range as the open models, NOT an exemption. Same theme hierarchy (LGBTQ+ 30.8% and secular-rational 26.7% cut hardest; trust 92.3% and autonomy 85.7% survive). Same genre boundary on the policy memo (+63.9 pp, even stronger than the open models). Confirms the effect is not specific to open models. Two configs dropped short-version items to a judge reasoning leak (not a model failure); 16/18 configs have complete data.
Adds scripts/extract_surviving_spans.py: for each model x document in the example set, asks the judge (Kimi K3) to quote the verbatim fragment of the model's short version that carries each value statement ruled present or toned_down. Output: data/wvs-swe/surviving_spans.json [doc_id][model][item_id] = quoted substring (null when absent). 183/218 present-or-toned items yielded a verbatim quote, each sanity-checked to appear in the short text. Drives the blog example cards, which now highlight exactly which part of each short version survived — replacing an earlier stem-matching heuristic that was too noisy.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Expand wvs-swe from 3 to 6 documents
Status: draft — work in progress, do not merge.
The headline analysis showed the survival gap varies sharply by document
(−4 pp interview, −23 pp minutes, −32 pp notes), so three documents is the
real generalisability weakness. This branch expands the corpus to six
documents across more genres before the blog piece ships.
Plan
nyhetsbrev_01(internal news article)policy_pm_01(HR policy memo)mejltrad_01(planning email thread)per-document breakdown.
Done so far
documents.3docs-backup.json).before the dataset is written.
The 3-document set and its run (
2026-08-11T14-09-11-wvs-swe-v2-lengthmatched)remain canonical until the expanded run passes validation.