diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index b649d1e..bd8ccf3 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -11,6 +11,63 @@ benchmark integrity. --- +## New here? Start in 60 seconds + +| Question | Answer | +|----------|--------| +| **What is Dali?** | Open evidence infrastructure for legal AI — reproducible benchmarks, verification taxonomies, and evidence artifacts. See [README.md](README.md) and [docs/METHODOLOGY.md](docs/METHODOLOGY.md). | +| **Good first issues** | Filter GitHub for labels `good first issue`, `benchmark`, or `corpus`. Corpus records, Tier 2 prompts, and docs are often the fastest path. | +| **Run locally** | `git clone https://github.com/yenklabs/Dali && cd Dali && python -m venv .venv && source .venv/bin/activate && pip install -r requirements.txt && pytest tests/` | +| **Coding standards** | Match existing style; keep diffs focused; schema and methodology changes need a spec or issue first. See [docs/for-engineers.md](docs/for-engineers.md). | +| **Where to ask** | Open a [GitHub Discussion](https://github.com/yenklabs/Dali/discussions) or issue with context and what you tried. | +| **Submit a PR** | Fork → branch → `pytest tests/` green → open PR with what changed and why. Use the PR checklist below. | + +Prefer no terminal? Use the MCP contributor tools (`lint`, `score`, `replay`, `probe`, `draft`, `pack`) — see [tools/mcp/README.md](tools/mcp/README.md). + +--- + +## Ways to contribute + +### Evidence + +- Evidence corpus submissions +- Jurisdiction packs +- Source validation + +### Datasets + +- Dataset improvements +- Metadata quality +- Documentation + +### Benchmarks + +- Prompt packs +- Evaluation workflows +- Reproducibility + +### Models + +- Baseline implementations +- Training pipelines +- Evaluation metrics + +### Tooling + +- MCP tools +- Parsers +- Evidence processing + +### Documentation + +- Tutorials +- Methodology +- Examples + +See also [contribution tracks](#contribution-tracks) below for file-level starting points and [docs/RESEARCH-ROADMAP.md](docs/RESEARCH-ROADMAP.md) for ecosystem direction. + +--- + ## Philosophy Dali is built on a simple assumption: diff --git a/README.md b/README.md index 33ad8d5..a0cc81c 100644 --- a/README.md +++ b/README.md @@ -31,7 +31,7 @@ Evidence Packages · Replay Engine · APIs | Public asset | Location | | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| Seed evidence corpus | [legal-ai-failure-database](https://huggingface.co/datasets/yenklabs/legal-ai-failure-database) | +| Seed evidence corpus | [open-evidence-corpus](https://huggingface.co/datasets/yenklabs/open-evidence-corpus) | | Seed benchmark sample (public) | [dali-citation-benchmark](https://huggingface.co/datasets/yenklabs/dali-citation-benchmark) — 5 hand-curated cases, 14 authorities, for methodology review and contribution | | Full evaluation run | [data/results/](https://github.com/yenklabs/Dali/blob/main/data/results) and [LEADERBOARD.md](https://github.com/yenklabs/Dali/blob/main/docs/LEADERBOARD.md) — 524 citations, 3 models, 5 jurisdiction tracks | | Verification taxonomy | [dali-verification-taxonomy](https://huggingface.co/datasets/yenklabs/dali-verification-taxonomy) | @@ -40,6 +40,39 @@ Evidence Packages · Replay Engine · APIs Full index: [huggingface.co/yenklabs](https://huggingface.co/yenklabs) +## Research artifacts + +Dali publishes reusable research assets that support reproducible evaluation. Seed samples and the full evaluation run are named separately. + +### Datasets + +- [Dali Open Evidence Corpus](datasets/open-evidence-corpus/) — [Hugging Face](https://huggingface.co/datasets/yenklabs/open-evidence-corpus) · *available* +- [Dali Citation Benchmark (Seed Sample)](datasets/citation-benchmark/) — [Hugging Face](https://huggingface.co/datasets/yenklabs/dali-citation-benchmark) · *available* (5 cases / 14 authorities) +- [Dali Verification Taxonomy](datasets/verification-taxonomy/) — [Hugging Face](https://huggingface.co/datasets/yenklabs/dali-verification-taxonomy) · *available* +- [Dali Evaluation Prompts](datasets/evaluation-prompts/) — *planned* + +### Benchmarks + +- [Reproducible evaluation workflows](benchmarks/) +- Cross-jurisdiction benchmark suite — [`data/benchmark/`](data/benchmark/) +- [Full evaluation run / releases](data/results/) — 524 citations · 3 models · 5 jurisdiction tracks · *available* + +### Models + +Planned baseline research models built from open evidence artifacts. [Models roadmap](models/README.md) — all *planned*: + +- Verification Taxonomy Classifier +- Citation Risk Classifier +- Authority Matching Baseline +- Proposition Support Classifier + +Models support the evidence ecosystem. They do not replace it. + +### Evidence + +- [Reusable evidence artifacts](evidence/) supporting reproducibility and independent verification +- [Methodology](methodology/) and [research roadmap](docs/RESEARCH-ROADMAP.md) + ## Why does it matter? AI systems lack a standard way to create, exchange, verify, and preserve evidence. The legal industry has been an early proving ground — court-documented incidents since [*Mata v. Avianca*](docs/CASE-STUDIES.md#1-mata-v-avianca-inc-sdny-2023) (2023), including [*United States v. Cohen*](docs/CASE-STUDIES.md#2-united-states-v-cohen-sdny-2023) and [*Park v. Kim*](docs/CASE-STUDIES.md#3-park-v-kim-2d-cir-2024), which anchor the Tier 1 canonical corpus in [data/benchmark/tier1/corpus/citation_failure_cases.json](data/benchmark/tier1/corpus/citation_failure_cases.json). Dali consolidates missing public infrastructure into one MIT-licensed, deterministically replayable verification layer, with reproducibility defined through [cryptographic lineage](docs/cryptographic-lineage.md) and the public [methodology](docs/METHODOLOGY.md). diff --git a/benchmarks/README.md b/benchmarks/README.md new file mode 100644 index 0000000..6c9592c --- /dev/null +++ b/benchmarks/README.md @@ -0,0 +1,28 @@ +# Dali Benchmarks + +Reproducible evaluation workflows, cross-jurisdiction benchmark suites, and versioned benchmark releases. + +## In this repository + +| Asset | Location | +|-------|----------| +| Tier 1 canonical corpus | [`data/benchmark/tier1/`](../data/benchmark/tier1/) | +| Tier 2 evaluation prompts | [`data/benchmark/tier2/`](../data/benchmark/tier2/) | +| Published run artifacts | [`data/results/`](../data/results/) | +| Deterministic Tier 1 runner | [`dali/runners/run_integrity.py`](../dali/runners/run_integrity.py) | +| Tier 2 synthetic runner | [`dali/runners/run_synthetic.py`](../dali/runners/run_synthetic.py) | +| Leaderboard | [`docs/LEADERBOARD.md`](../docs/LEADERBOARD.md) | + +## Hugging Face + +- [Dali Citation Benchmark](https://huggingface.co/datasets/yenklabs/dali-citation-benchmark) +- [Organization profile](https://huggingface.co/yenklabs) + +## Reproduce a run + +```bash +pip install -r requirements.txt +python -m tools.cli replay +``` + +See [docs/quickstart.md](../docs/quickstart.md). diff --git a/datasets/README.md b/datasets/README.md new file mode 100644 index 0000000..3a661df --- /dev/null +++ b/datasets/README.md @@ -0,0 +1,12 @@ +# Dali Datasets + +Published dataset artifacts for the Dali open evidence ecosystem. Canonical evaluation data lives under [`data/benchmark/`](../data/benchmark/); Hugging Face mirrors ship from [yenklabs on Hugging Face](https://huggingface.co/yenklabs/datasets). + +| Dataset | Directory | Hugging Face | +|---------|-----------|--------------| +| Dali Open Evidence Corpus | [open-evidence-corpus/](open-evidence-corpus/) | [open-evidence-corpus](https://huggingface.co/datasets/yenklabs/open-evidence-corpus) | +| Dali Citation Benchmark | [citation-benchmark/](citation-benchmark/) | [dali-citation-benchmark](https://huggingface.co/datasets/yenklabs/dali-citation-benchmark) | +| Dali Verification Taxonomy | [verification-taxonomy/](verification-taxonomy/) | [dali-verification-taxonomy](https://huggingface.co/datasets/yenklabs/dali-verification-taxonomy) | +| Dali Evaluation Prompts | [evaluation-prompts/](evaluation-prompts/) | forthcoming | + +Models support the evidence ecosystem. They do not replace it. diff --git a/datasets/citation-benchmark/README.md b/datasets/citation-benchmark/README.md new file mode 100644 index 0000000..da19b78 --- /dev/null +++ b/datasets/citation-benchmark/README.md @@ -0,0 +1,18 @@ +# Dali Citation Benchmark + +Open benchmark for evaluating legal AI citation verification. + +## In this repository + +- Tier 1 corpus: [`data/benchmark/tier1/`](../data/benchmark/tier1/) +- Tier 2 prompts: [`data/benchmark/tier2/`](../data/benchmark/tier2/) +- Published results: [`data/results/`](../data/results/) +- Runners: [`dali/runners/`](../dali/runners/) + +## Hugging Face + +[yenklabs/dali-citation-benchmark](https://huggingface.co/datasets/yenklabs/dali-citation-benchmark) + +## Contribute + +Prompt packs, evaluation workflows, and reproducibility — see [CONTRIBUTING.md](../CONTRIBUTING.md#ways-to-contribute). diff --git a/datasets/evaluation-prompts/README.md b/datasets/evaluation-prompts/README.md new file mode 100644 index 0000000..bf848c6 --- /dev/null +++ b/datasets/evaluation-prompts/README.md @@ -0,0 +1,16 @@ +# Dali Evaluation Prompts + +Cross-jurisdiction evaluation prompts for testing legal AI systems. + +## In this repository + +- Tier 2 JSONL corpora: [`data/benchmark/tier2/`](../data/benchmark/tier2/) +- Contributor tools: [`tools/mcp/`](../tools/mcp/) (`probe`, `draft`, `pack`) + +## Status + +Mirrored on Hugging Face as **Dali Evaluation Prompts** — forthcoming. + +## Contribute + +New prompt packs and evaluation workflows — see [CONTRIBUTING.md](../CONTRIBUTING.md#ways-to-contribute). diff --git a/datasets/open-evidence-corpus/README.md b/datasets/open-evidence-corpus/README.md new file mode 100644 index 0000000..27df55f --- /dev/null +++ b/datasets/open-evidence-corpus/README.md @@ -0,0 +1,17 @@ +# Dali Open Evidence Corpus + +Documented evidence records for reproducible legal AI research. + +## In this repository + +Tier 1 canonical cases and related evidence records: + +- [`data/benchmark/tier1/corpus/citation_failure_cases.json`](../data/benchmark/tier1/corpus/citation_failure_cases.json) + +## Hugging Face + +[yenklabs/open-evidence-corpus](https://huggingface.co/datasets/yenklabs/open-evidence-corpus) + +## Contribute + +Evidence corpus submissions, jurisdiction packs, and source validation — see [CONTRIBUTING.md](../CONTRIBUTING.md#ways-to-contribute). diff --git a/datasets/verification-taxonomy/README.md b/datasets/verification-taxonomy/README.md new file mode 100644 index 0000000..ad52765 --- /dev/null +++ b/datasets/verification-taxonomy/README.md @@ -0,0 +1,16 @@ +# Dali Verification Taxonomy + +Shared vocabulary for classifying legal AI verification outcomes. + +## In this repository + +- Ontology and failure classes: [`dali/schemas/ontology.md`](../dali/schemas/ontology.md) +- Policy and taxonomy modules: [`dali/corpus/`](../dali/corpus/) + +## Hugging Face + +[yenklabs/dali-verification-taxonomy](https://huggingface.co/datasets/yenklabs/dali-verification-taxonomy) + +## Contribute + +Taxonomy improvements and documentation — see [CONTRIBUTING.md](../CONTRIBUTING.md#ways-to-contribute). diff --git a/docs/RESEARCH-ROADMAP.md b/docs/RESEARCH-ROADMAP.md new file mode 100644 index 0000000..e39397a --- /dev/null +++ b/docs/RESEARCH-ROADMAP.md @@ -0,0 +1,66 @@ +# Dali Research Roadmap + +Future public artifacts follow this progression. Each layer builds on the previous one. + +```text +Open Evidence Corpus + ↓ +Verification Taxonomy + ↓ +Citation Benchmark + ↓ +Baseline Research Models + ↓ +Interactive Hugging Face Spaces + ↓ +Community Contributions + ↓ +Annual Evidence Reports +``` + +## Repository map + +```text +Dali/ +├── datasets/ → published dataset artifacts (index + HF mirrors) +├── models/ → lightweight baseline research models (roadmap) +├── benchmarks/ → evaluation workflows and releases +├── methodology/ → public methodology index +├── evidence/ → evidence artifact index +├── docs/ → methodology, specs, guides +├── tools/ → CLI, MCP, scripts +└── CONTRIBUTING.md +``` + +Canonical runtime code remains in `dali/`; canonical benchmark data remains in `data/benchmark/` during the transition. Top-level indexes link to both. + +## Guiding principle + +The benchmark is not the product. The datasets are not the product. The models are not the product. + +Together they form Dali's open evidence infrastructure. + +Every new artifact should be: + +- Open +- Reproducible +- Independently verifiable +- Community extensible +- Useful beyond Dali itself + +## Models roadmap + +| Release | Model | Purpose | +|---------|-------|---------| +| v0.1 | Verification Taxonomy Classifier | Predict standardized verification outcome labels | +| v0.2 | Citation Risk Classifier | Estimate citation verification risk from evidence metadata | +| v0.3 | Authority Matching Baseline | Reproducible authority matching baseline | +| v0.4 | Proposition Support Classifier | Classify proposition support relationships | + +See [`models/README.md`](../models/README.md). + +## Hugging Face ecosystem + +- [Organization](https://huggingface.co/yenklabs) +- [Datasets](https://huggingface.co/yenklabs/datasets) +- Planned Spaces: Evidence Explorer, Benchmark Dashboard, Citation Verification Demo diff --git a/evidence/README.md b/evidence/README.md new file mode 100644 index 0000000..3e948ea --- /dev/null +++ b/evidence/README.md @@ -0,0 +1,21 @@ +# Dali Evidence + +Reusable evidence artifacts supporting reproducibility and independent verification. + +## In this repository + +| Artifact | Location | +|----------|----------| +| Tier 1 evidence records | [`data/benchmark/tier1/corpus/`](../data/benchmark/tier1/corpus/) | +| Integrity result schema | [`dali/schemas/`](../dali/schemas/) | +| Lineage and hashing | [`dali/corpus/lineage.py`](../dali/corpus/lineage.py), [`docs/cryptographic-lineage.md`](../docs/cryptographic-lineage.md) | +| Case studies | [`docs/CASE-STUDIES.md`](../docs/CASE-STUDIES.md) | +| Evidence JSON spec | [`docs/specs/RFC-001-evidence-json-v1.md`](../docs/specs/RFC-001-evidence-json-v1.md) | + +## Hugging Face + +- [Dali Open Evidence Corpus](https://huggingface.co/datasets/yenklabs/open-evidence-corpus) + +## Contribute + +Evidence corpus submissions, jurisdiction packs, and source validation — see [CONTRIBUTING.md](../CONTRIBUTING.md#ways-to-contribute). diff --git a/methodology/README.md b/methodology/README.md new file mode 100644 index 0000000..f0f4a70 --- /dev/null +++ b/methodology/README.md @@ -0,0 +1,22 @@ +# Dali Methodology + +Public, versioned methodology for citation integrity evaluation and evidence reconstructability. + +## Canonical documents + +| Document | Path | +|----------|------| +| Methodology overview | [`docs/METHODOLOGY.md`](../docs/METHODOLOGY.md) | +| Policy versioning | [`docs/policy-versioning.md`](../docs/policy-versioning.md) | +| Cryptographic lineage | [`docs/cryptographic-lineage.md`](../docs/cryptographic-lineage.md) | +| Reviewer guide | [`docs/reviewer-guide.md`](../docs/reviewer-guide.md) | +| Specs (RFCs) | [`docs/specs/`](../docs/specs/) | + +## Datasets and taxonomy + +Methodology outputs are grounded in open artifacts: + +- [`datasets/verification-taxonomy/`](../datasets/verification-taxonomy/) +- [`datasets/citation-benchmark/`](../datasets/citation-benchmark/) + +Every published result should cite a **policy version** and produce replay-verifiable hashes. diff --git a/models/README.md b/models/README.md new file mode 100644 index 0000000..cb5f706 --- /dev/null +++ b/models/README.md @@ -0,0 +1,26 @@ +# Dali Research Models + +Dali publishes lightweight, reproducible baseline models trained from open evidence artifacts. + +These models exist to support research, benchmarking, and evaluation. + +They are intentionally transparent, reproducible, and extensible. + +## Current roadmap + +- [Verification Taxonomy Classifier](verification-taxonomy-classifier/) — predict standardized verification outcome labels +- [Citation Risk Classifier](citation-risk-classifier/) — estimate citation verification risk from evidence metadata +- [Authority Matching Baseline](authority-matching-baseline/) — reproducible baseline for authority matching experiments +- [Proposition Support Classifier](proposition-support-classifier/) — classify proposition support relationships for legal authorities + +Models are released alongside datasets, methodologies, and benchmark updates. + +## Guiding principle + +Models support the evidence ecosystem. They do not replace it. + +The benchmark is not the product. The datasets are not the product. The models are not the product. Together they form Dali's open evidence infrastructure. + +## Hugging Face + +Model releases will publish under [yenklabs on Hugging Face](https://huggingface.co/yenklabs) when available. diff --git a/models/authority-matching-baseline/README.md b/models/authority-matching-baseline/README.md new file mode 100644 index 0000000..ea80348 --- /dev/null +++ b/models/authority-matching-baseline/README.md @@ -0,0 +1,5 @@ +# Dali Authority Matching Baseline + +**Status:** planned (v0.3) + +Reproducible baseline for authority matching experiments, aligned with Dali evaluation methodology. diff --git a/models/citation-risk-classifier/README.md b/models/citation-risk-classifier/README.md new file mode 100644 index 0000000..89dcb2c --- /dev/null +++ b/models/citation-risk-classifier/README.md @@ -0,0 +1,5 @@ +# Dali Citation Risk Classifier + +**Status:** planned (v0.2) + +Estimate citation verification risk using evidence metadata from the [Dali Open Evidence Corpus](../datasets/open-evidence-corpus/) and [Citation Benchmark](../datasets/citation-benchmark/). diff --git a/models/proposition-support-classifier/README.md b/models/proposition-support-classifier/README.md new file mode 100644 index 0000000..c3d36b9 --- /dev/null +++ b/models/proposition-support-classifier/README.md @@ -0,0 +1,5 @@ +# Dali Proposition Support Classifier + +**Status:** planned (v0.4) + +Classify proposition support relationships for legal authorities — existence vs. support-stage verification. diff --git a/models/verification-taxonomy-classifier/README.md b/models/verification-taxonomy-classifier/README.md new file mode 100644 index 0000000..ec480bd --- /dev/null +++ b/models/verification-taxonomy-classifier/README.md @@ -0,0 +1,7 @@ +# Dali Verification Taxonomy Classifier + +**Status:** planned (v0.1) + +Predict standardized verification outcome labels using the [Dali Verification Taxonomy](../datasets/verification-taxonomy/). + +Training data and evaluation protocol will ship with the first public model release.