Reproducibility fixes, metric layer, and a data-quality audit

.env.example

@@ -0,0 +1,2 @@
+OPENAI_API_KEY=
+DEEPSEEK_API_KEY=

.gitignore

@@ -0,0 +1,7 @@
+.env
+__pycache__/
+*.pyc
+results/
+batch_results/
+*.csv
+_sample_rows.json

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 MAKIEval contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -16,11 +16,13 @@ We have released the MAKIEval dataset on Hugging Face:
 
 The released dataset currently contains:
 
-* 🤖 13 LLMs
+* 🤖 **7 LLMs** (see paper Table 1)
 * 🌐 13 Languages
-* 🗺️ Multiple Countries and Regions
+* 🗺️ 19 Countries and Regions
 * 🎭 6 Cultural Domains
 
+> **Note on scale:** The Hugging Face release contains approximately **5.25M rows**. The paper reports 85.8 million generated texts; at the response level the expected order of magnitude is ≈6M (1,716 prompts × 500 responses × 7 models). The difference may reflect aggregation, filtering, or release packaging — we report HF counts here rather than restating the paper figure.
+
 ### Cultural Domains
 
 * 🍽️ Food
@@ -55,17 +57,134 @@ entities
 
 ```text
 code/
-    analysis_*.py
+    analysis_*.py            # topic-specific Wikidata linking
+    data_loading.py          # Hugging Face dataset loader
     entity_extraction.py
+    lm_utils.py
+    metrics.py               # granularity, diversity, specificity, consensus
     prompt_construct.py
     run_experiment.py
+    run_metrics.py           # CLI for metric computation
+    validate_fidelity.py      # paper-facing fidelity checks
 
 meta_info/
     country.json
     name.json
     prompt.json
+
+tests/
+    test_data_loading.py
+    test_metrics.py
+
+docs/
+    DEVIATIONS.md
+    FIDELITY_REPORT.md
+```
+
+---
+
+## 🔁 Reproduce (metrics)
+
+> ⚠️ **Use `--full` for paper-comparable numbers. The default 5,000-row limit is only a quick smoke test; Diversity and Consensus in this mode are not comparable to the paper values.**
+
+### 1. Install
+
+```bash
+python -m venv .venv
+source .venv/bin/activate   # Windows: .venv\Scripts\activate
+pip install -r requirements.txt
+```
+
+### 2. Compute metrics from the public dataset
+
+No API key is required for `run_metrics.py`; API-backed generation and entity
+extraction still require `OPENAI_API_KEY` / `DEEPSEEK_API_KEY` in the
+environment.
+
+```bash
+python code/run_metrics.py --model Qwen2.5-7B-Instruct --topic beverage --language ar --seed 42 --limit 5000
+```
+
+Outputs are written to `results/metrics/`:
+
+* `group_metrics.csv` — granularity, diversity, culture specificity per (model, topic, language, country)
+* `culture_consensus.csv` — pairwise Jaccard scores across languages
+* `parse_report.json` — entity JSON parse statistics
+
+`run_metrics.py` defaults to a 5,000-row limit for quick checks. Add `--full` to process all rows matching the selected filters.
+
+### 3. Run tests
+
+```bash
+pytest -q tests/
+```
+
+---
+
+## Published-data Quality Audit
+
+`quality_report.py` audits the published Hugging Face rows without generation,
+GPU, or API keys. The default command reservoir-samples 200 rows per
+`model x language x topic` slice and writes `docs/DATA_QUALITY_REPORT.md`.
+See docs/RESULTS.md for an audit summary with figures.
+
+```bash
+python code/quality_report.py --sample 200 --seed 42
+```
+
+Use `--full` to scan every row for the quality checks. The report includes
+degenerate text checks, language leakage estimates, missing-QID rates,
+surface-form to QID inconsistency, suspicious extraction review candidates,
+and observed group coverage.
+
+## Smoke Reproduction With Together
+
+Small-scale generation can be run through Together AI to avoid local GPU/model
+downloads. Set `TOGETHER_API_KEY` in the environment; do not commit keys.
+
+```bash
+python code/smoke_generation.py \
+  --model Qwen2.5-7B-Instruct \
+  --language en \
+  --topic books \
+  --country "United States" \
+  --num-responses 3 \
+  --faithful \
+  --seed 42
 ```
 
+Outputs are written to `results/smoke/`. Compare a smoke run with the matching
+published Hugging Face slice:
+
+```bash
+python code/compare_to_published.py \
+  --model Qwen2.5-7B-Instruct \
+  --language en \
+  --topic books \
+  --country "United States" \
+  --published-limit 3
+```
+
+`--faithful` locks local-model decoding values to the paper setup:
+`temperature=0.7`, `top_p=0.9`, `top_k=10`, and `max_tokens=100`. By default,
+the CLI reuses the exact prompt string from the selected Hugging Face slice.
+
+### Model Access Matrix
+
+| Paper model | Default Together route | Task B feasibility |
+|---|---|---|
+| Qwen2.5-7B-Instruct | `Qwen/Qwen2.5-7B-Instruct-Turbo` | Recommended smoke default; provider Turbo endpoint. |
+| Llama-3.1-8B-Instruct | `meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo` | Depends on Together availability. |
+| Mistral-7B-Instruct-v0.1 | `mistralai/Mistral-7B-Instruct-v0.1` | Depends on Together availability. |
+| aya-expanse-8b | `CohereLabs/aya-expanse-8b` | Depends on Together availability. |
+| Llama-3.3-70B-Instruct | `meta-llama/Llama-3.3-70B-Instruct-Turbo` | API feasible; local full-scale requires large GPUs. |
+| ChatGPT-4o-mini | `openai/gpt-oss-120b` | **PROXY MODEL - NOT IN PAPER SET** for generation/extraction when OpenAI API is unavailable. |
+| DeepSeek-V3 | `deepseek-ai/DeepSeek-V3` | Depends on Together availability; native DeepSeek API remains separate. |
+
+Entity extraction in the paper uses GPT-4o-mini. This branch can substitute a
+Together model when `OPENAI_API_KEY` is intentionally unavailable; such runs are
+reported as proxy extraction and are not exact paper-faithful extraction.
+
 ---
 
 ## ⚙️ Pipeline

code/analysis.py

@@ -0,0 +1,96 @@
+# RECONSTRUCTED from paper Section 3.3 & 4.3; original analysis.py was not released in upstream repo.
+
+import re
+import time
+from functools import lru_cache
+
+import pandas as pd
+from SPARQLWrapper import SPARQLWrapper, JSON
+
+sparql = SPARQLWrapper("https://query.wikidata.org/sparql")
+sparql.addCustomHttpHeader("User-Agent", "MAKIEval/1.0 (cultural-awareness-evaluation)")
+
+COUNTRY_SOURCE_COLUMNS = (
+    "Author Countries",
+    "Performer Countries",
+    "Origin Countries",
+)
+
+
+@lru_cache(maxsize=4096)
+def _query_wikidata_origin_country(qid: str) -> str:
+    """Resolve origin country for a QID (P495 preferred, then P17)."""
+    if not qid or qid == "NA":
+        return "NA"
+
+    query = f"""
+    SELECT ?countryLabel WHERE {{
+      OPTIONAL {{ wd:{qid} wdt:P495 ?country. }}
+      OPTIONAL {{ wd:{qid} wdt:P17 ?country. }}
+      ?country rdfs:label ?countryLabel.
+      FILTER(LANG(?countryLabel) = "en")
+    }}
+    LIMIT 1
+    """
+    for attempt in range(3):
+        try:
+            sparql.setQuery(query)
+            sparql.setReturnFormat(JSON)
+            results = sparql.query().convert()
+            bindings = results["results"]["bindings"]
+            if bindings and "countryLabel" in bindings[0]:
+                return bindings[0]["countryLabel"]["value"]
+            return "NA"
+        except Exception:
+            time.sleep(1 + attempt)
+    return "NA"
+
+
+def _parse_countries_from_information(information: str) -> str:
+    """Best-effort parse of country names embedded in the Information field."""
+    if not information or information == "NA":
+        return "NA"
+
+    # Wikidata info strings often end with comma-separated English country labels.
+    parts = [part.strip() for part in information.split(",") if part.strip()]
+    if not parts:
+        return "NA"
+
+    # Heuristic: trailing tokens that look like country names (Title Case words).
+    country_like = []
+    for part in reversed(parts):
+        if re.match(r"^[A-Z][a-zA-Z\s\-']+$", part):
+            country_like.insert(0, part)
+        else:
+            break
+
+    return ", ".join(country_like) if country_like else "NA"
+
+
+def _row_origin_country(row: pd.Series) -> str:
+    for column in COUNTRY_SOURCE_COLUMNS:
+        if column in row.index and row[column] not in (None, "", "NA"):
+            return str(row[column])
+
+    information = row.get("Information", "NA")
+    parsed = _parse_countries_from_information(str(information))
+    if parsed != "NA":
+        return parsed
+
+    qid = row.get("Q_ID", "NA")
+    return _query_wikidata_origin_country(str(qid))
+
+
+def process_wikidata_country_info(df: pd.DataFrame) -> pd.DataFrame:
+    """
+    Enrich Wikidata entity rows with origin-country metadata for culture specificity.
+
+    Expected input columns include:
+    Q_ID, Original Label, Label, Information, is_category, Alternative QIDs
+    """
+    if df.empty:
+        return df
+
+    result = df.copy()
+    result["Origin Country"] = result.apply(_row_origin_country, axis=1)
+    return result