Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 57 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,63 @@ benchmark integrity.

---

## New here? Start in 60 seconds

| Question | Answer |
|----------|--------|
| **What is Dali?** | Open evidence infrastructure for legal AI — reproducible benchmarks, verification taxonomies, and evidence artifacts. See [README.md](README.md) and [docs/METHODOLOGY.md](docs/METHODOLOGY.md). |
| **Good first issues** | Filter GitHub for labels `good first issue`, `benchmark`, or `corpus`. Corpus records, Tier 2 prompts, and docs are often the fastest path. |
| **Run locally** | `git clone https://github.com/yenklabs/Dali && cd Dali && python -m venv .venv && source .venv/bin/activate && pip install -r requirements.txt && pytest tests/` |
| **Coding standards** | Match existing style; keep diffs focused; schema and methodology changes need a spec or issue first. See [docs/for-engineers.md](docs/for-engineers.md). |
| **Where to ask** | Open a [GitHub Discussion](https://github.com/yenklabs/Dali/discussions) or issue with context and what you tried. |
| **Submit a PR** | Fork → branch → `pytest tests/` green → open PR with what changed and why. Use the PR checklist below. |

Prefer no terminal? Use the MCP contributor tools (`lint`, `score`, `replay`, `probe`, `draft`, `pack`) — see [tools/mcp/README.md](tools/mcp/README.md).

---

## Ways to contribute

### Evidence

- Evidence corpus submissions
- Jurisdiction packs
- Source validation

### Datasets

- Dataset improvements
- Metadata quality
- Documentation

### Benchmarks

- Prompt packs
- Evaluation workflows
- Reproducibility

### Models

- Baseline implementations
- Training pipelines
- Evaluation metrics

### Tooling

- MCP tools
- Parsers
- Evidence processing

### Documentation

- Tutorials
- Methodology
- Examples

See also [contribution tracks](#contribution-tracks) below for file-level starting points and [docs/RESEARCH-ROADMAP.md](docs/RESEARCH-ROADMAP.md) for ecosystem direction.

---

## Philosophy

Dali is built on a simple assumption:
Expand Down
35 changes: 34 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ Evidence Packages · Replay Engine · APIs

| Public asset | Location |
| ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Seed evidence corpus | [legal-ai-failure-database](https://huggingface.co/datasets/yenklabs/legal-ai-failure-database) |
| Seed evidence corpus | [open-evidence-corpus](https://huggingface.co/datasets/yenklabs/open-evidence-corpus) |
| Seed benchmark sample (public) | [dali-citation-benchmark](https://huggingface.co/datasets/yenklabs/dali-citation-benchmark) — 5 hand-curated cases, 14 authorities, for methodology review and contribution |
| Full evaluation run | [data/results/](https://github.com/yenklabs/Dali/blob/main/data/results) and [LEADERBOARD.md](https://github.com/yenklabs/Dali/blob/main/docs/LEADERBOARD.md) — 524 citations, 3 models, 5 jurisdiction tracks |
| Verification taxonomy | [dali-verification-taxonomy](https://huggingface.co/datasets/yenklabs/dali-verification-taxonomy) |
Expand All @@ -40,6 +40,39 @@ Evidence Packages · Replay Engine · APIs

Full index: [huggingface.co/yenklabs](https://huggingface.co/yenklabs)

## Research artifacts

Dali publishes reusable research assets that support reproducible evaluation. Seed samples and the full evaluation run are named separately.

### Datasets

- [Dali Open Evidence Corpus](datasets/open-evidence-corpus/) — [Hugging Face](https://huggingface.co/datasets/yenklabs/open-evidence-corpus) · *available*
- [Dali Citation Benchmark (Seed Sample)](datasets/citation-benchmark/) — [Hugging Face](https://huggingface.co/datasets/yenklabs/dali-citation-benchmark) · *available* (5 cases / 14 authorities)
- [Dali Verification Taxonomy](datasets/verification-taxonomy/) — [Hugging Face](https://huggingface.co/datasets/yenklabs/dali-verification-taxonomy) · *available*
- [Dali Evaluation Prompts](datasets/evaluation-prompts/) — *planned*

### Benchmarks

- [Reproducible evaluation workflows](benchmarks/)
- Cross-jurisdiction benchmark suite — [`data/benchmark/`](data/benchmark/)
- [Full evaluation run / releases](data/results/) — 524 citations · 3 models · 5 jurisdiction tracks · *available*

### Models

Planned baseline research models built from open evidence artifacts. [Models roadmap](models/README.md) — all *planned*:

- Verification Taxonomy Classifier
- Citation Risk Classifier
- Authority Matching Baseline
- Proposition Support Classifier

Models support the evidence ecosystem. They do not replace it.

### Evidence

- [Reusable evidence artifacts](evidence/) supporting reproducibility and independent verification
- [Methodology](methodology/) and [research roadmap](docs/RESEARCH-ROADMAP.md)

## Why does it matter?

AI systems lack a standard way to create, exchange, verify, and preserve evidence. The legal industry has been an early proving ground — court-documented incidents since [*Mata v. Avianca*](docs/CASE-STUDIES.md#1-mata-v-avianca-inc-sdny-2023) (2023), including [*United States v. Cohen*](docs/CASE-STUDIES.md#2-united-states-v-cohen-sdny-2023) and [*Park v. Kim*](docs/CASE-STUDIES.md#3-park-v-kim-2d-cir-2024), which anchor the Tier 1 canonical corpus in [data/benchmark/tier1/corpus/citation_failure_cases.json](data/benchmark/tier1/corpus/citation_failure_cases.json). Dali consolidates missing public infrastructure into one MIT-licensed, deterministically replayable verification layer, with reproducibility defined through [cryptographic lineage](docs/cryptographic-lineage.md) and the public [methodology](docs/METHODOLOGY.md).
Expand Down
28 changes: 28 additions & 0 deletions benchmarks/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# Dali Benchmarks

Reproducible evaluation workflows, cross-jurisdiction benchmark suites, and versioned benchmark releases.

## In this repository

| Asset | Location |
|-------|----------|
| Tier 1 canonical corpus | [`data/benchmark/tier1/`](../data/benchmark/tier1/) |
| Tier 2 evaluation prompts | [`data/benchmark/tier2/`](../data/benchmark/tier2/) |
| Published run artifacts | [`data/results/`](../data/results/) |
| Deterministic Tier 1 runner | [`dali/runners/run_integrity.py`](../dali/runners/run_integrity.py) |
| Tier 2 synthetic runner | [`dali/runners/run_synthetic.py`](../dali/runners/run_synthetic.py) |
| Leaderboard | [`docs/LEADERBOARD.md`](../docs/LEADERBOARD.md) |

## Hugging Face

- [Dali Citation Benchmark](https://huggingface.co/datasets/yenklabs/dali-citation-benchmark)
- [Organization profile](https://huggingface.co/yenklabs)

## Reproduce a run

```bash
pip install -r requirements.txt
python -m tools.cli replay
```

See [docs/quickstart.md](../docs/quickstart.md).
12 changes: 12 additions & 0 deletions datasets/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# Dali Datasets

Published dataset artifacts for the Dali open evidence ecosystem. Canonical evaluation data lives under [`data/benchmark/`](../data/benchmark/); Hugging Face mirrors ship from [yenklabs on Hugging Face](https://huggingface.co/yenklabs/datasets).

| Dataset | Directory | Hugging Face |
|---------|-----------|--------------|
| Dali Open Evidence Corpus | [open-evidence-corpus/](open-evidence-corpus/) | [open-evidence-corpus](https://huggingface.co/datasets/yenklabs/open-evidence-corpus) |
| Dali Citation Benchmark | [citation-benchmark/](citation-benchmark/) | [dali-citation-benchmark](https://huggingface.co/datasets/yenklabs/dali-citation-benchmark) |
| Dali Verification Taxonomy | [verification-taxonomy/](verification-taxonomy/) | [dali-verification-taxonomy](https://huggingface.co/datasets/yenklabs/dali-verification-taxonomy) |
| Dali Evaluation Prompts | [evaluation-prompts/](evaluation-prompts/) | forthcoming |

Models support the evidence ecosystem. They do not replace it.
18 changes: 18 additions & 0 deletions datasets/citation-benchmark/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# Dali Citation Benchmark

Open benchmark for evaluating legal AI citation verification.

## In this repository

- Tier 1 corpus: [`data/benchmark/tier1/`](../data/benchmark/tier1/)
- Tier 2 prompts: [`data/benchmark/tier2/`](../data/benchmark/tier2/)
- Published results: [`data/results/`](../data/results/)
- Runners: [`dali/runners/`](../dali/runners/)

## Hugging Face

[yenklabs/dali-citation-benchmark](https://huggingface.co/datasets/yenklabs/dali-citation-benchmark)

## Contribute

Prompt packs, evaluation workflows, and reproducibility — see [CONTRIBUTING.md](../CONTRIBUTING.md#ways-to-contribute).
16 changes: 16 additions & 0 deletions datasets/evaluation-prompts/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# Dali Evaluation Prompts

Cross-jurisdiction evaluation prompts for testing legal AI systems.

## In this repository

- Tier 2 JSONL corpora: [`data/benchmark/tier2/`](../data/benchmark/tier2/)
- Contributor tools: [`tools/mcp/`](../tools/mcp/) (`probe`, `draft`, `pack`)

## Status

Mirrored on Hugging Face as **Dali Evaluation Prompts** — forthcoming.

## Contribute

New prompt packs and evaluation workflows — see [CONTRIBUTING.md](../CONTRIBUTING.md#ways-to-contribute).
17 changes: 17 additions & 0 deletions datasets/open-evidence-corpus/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Dali Open Evidence Corpus

Documented evidence records for reproducible legal AI research.

## In this repository

Tier 1 canonical cases and related evidence records:

- [`data/benchmark/tier1/corpus/citation_failure_cases.json`](../data/benchmark/tier1/corpus/citation_failure_cases.json)

## Hugging Face

[yenklabs/open-evidence-corpus](https://huggingface.co/datasets/yenklabs/open-evidence-corpus)

## Contribute

Evidence corpus submissions, jurisdiction packs, and source validation — see [CONTRIBUTING.md](../CONTRIBUTING.md#ways-to-contribute).
16 changes: 16 additions & 0 deletions datasets/verification-taxonomy/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# Dali Verification Taxonomy

Shared vocabulary for classifying legal AI verification outcomes.

## In this repository

- Ontology and failure classes: [`dali/schemas/ontology.md`](../dali/schemas/ontology.md)
- Policy and taxonomy modules: [`dali/corpus/`](../dali/corpus/)

## Hugging Face

[yenklabs/dali-verification-taxonomy](https://huggingface.co/datasets/yenklabs/dali-verification-taxonomy)

## Contribute

Taxonomy improvements and documentation — see [CONTRIBUTING.md](../CONTRIBUTING.md#ways-to-contribute).
66 changes: 66 additions & 0 deletions docs/RESEARCH-ROADMAP.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
# Dali Research Roadmap

Future public artifacts follow this progression. Each layer builds on the previous one.

```text
Open Evidence Corpus
Verification Taxonomy
Citation Benchmark
Baseline Research Models
Interactive Hugging Face Spaces
Community Contributions
Annual Evidence Reports
```

## Repository map

```text
Dali/
├── datasets/ → published dataset artifacts (index + HF mirrors)
├── models/ → lightweight baseline research models (roadmap)
├── benchmarks/ → evaluation workflows and releases
├── methodology/ → public methodology index
├── evidence/ → evidence artifact index
├── docs/ → methodology, specs, guides
├── tools/ → CLI, MCP, scripts
└── CONTRIBUTING.md
```

Canonical runtime code remains in `dali/`; canonical benchmark data remains in `data/benchmark/` during the transition. Top-level indexes link to both.

## Guiding principle

The benchmark is not the product. The datasets are not the product. The models are not the product.

Together they form Dali's open evidence infrastructure.

Every new artifact should be:

- Open
- Reproducible
- Independently verifiable
- Community extensible
- Useful beyond Dali itself

## Models roadmap

| Release | Model | Purpose |
|---------|-------|---------|
| v0.1 | Verification Taxonomy Classifier | Predict standardized verification outcome labels |
| v0.2 | Citation Risk Classifier | Estimate citation verification risk from evidence metadata |
| v0.3 | Authority Matching Baseline | Reproducible authority matching baseline |
| v0.4 | Proposition Support Classifier | Classify proposition support relationships |

See [`models/README.md`](../models/README.md).

## Hugging Face ecosystem

- [Organization](https://huggingface.co/yenklabs)
- [Datasets](https://huggingface.co/yenklabs/datasets)
- Planned Spaces: Evidence Explorer, Benchmark Dashboard, Citation Verification Demo
21 changes: 21 additions & 0 deletions evidence/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Dali Evidence

Reusable evidence artifacts supporting reproducibility and independent verification.

## In this repository

| Artifact | Location |
|----------|----------|
| Tier 1 evidence records | [`data/benchmark/tier1/corpus/`](../data/benchmark/tier1/corpus/) |
| Integrity result schema | [`dali/schemas/`](../dali/schemas/) |
| Lineage and hashing | [`dali/corpus/lineage.py`](../dali/corpus/lineage.py), [`docs/cryptographic-lineage.md`](../docs/cryptographic-lineage.md) |
| Case studies | [`docs/CASE-STUDIES.md`](../docs/CASE-STUDIES.md) |
| Evidence JSON spec | [`docs/specs/RFC-001-evidence-json-v1.md`](../docs/specs/RFC-001-evidence-json-v1.md) |

## Hugging Face

- [Dali Open Evidence Corpus](https://huggingface.co/datasets/yenklabs/open-evidence-corpus)

## Contribute

Evidence corpus submissions, jurisdiction packs, and source validation — see [CONTRIBUTING.md](../CONTRIBUTING.md#ways-to-contribute).
22 changes: 22 additions & 0 deletions methodology/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# Dali Methodology

Public, versioned methodology for citation integrity evaluation and evidence reconstructability.

## Canonical documents

| Document | Path |
|----------|------|
| Methodology overview | [`docs/METHODOLOGY.md`](../docs/METHODOLOGY.md) |
| Policy versioning | [`docs/policy-versioning.md`](../docs/policy-versioning.md) |
| Cryptographic lineage | [`docs/cryptographic-lineage.md`](../docs/cryptographic-lineage.md) |
| Reviewer guide | [`docs/reviewer-guide.md`](../docs/reviewer-guide.md) |
| Specs (RFCs) | [`docs/specs/`](../docs/specs/) |

## Datasets and taxonomy

Methodology outputs are grounded in open artifacts:

- [`datasets/verification-taxonomy/`](../datasets/verification-taxonomy/)
- [`datasets/citation-benchmark/`](../datasets/citation-benchmark/)

Every published result should cite a **policy version** and produce replay-verifiable hashes.
26 changes: 26 additions & 0 deletions models/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# Dali Research Models

Dali publishes lightweight, reproducible baseline models trained from open evidence artifacts.

These models exist to support research, benchmarking, and evaluation.

They are intentionally transparent, reproducible, and extensible.

## Current roadmap

- [Verification Taxonomy Classifier](verification-taxonomy-classifier/) — predict standardized verification outcome labels
- [Citation Risk Classifier](citation-risk-classifier/) — estimate citation verification risk from evidence metadata
- [Authority Matching Baseline](authority-matching-baseline/) — reproducible baseline for authority matching experiments
- [Proposition Support Classifier](proposition-support-classifier/) — classify proposition support relationships for legal authorities

Models are released alongside datasets, methodologies, and benchmark updates.

## Guiding principle

Models support the evidence ecosystem. They do not replace it.

The benchmark is not the product. The datasets are not the product. The models are not the product. Together they form Dali's open evidence infrastructure.

## Hugging Face

Model releases will publish under [yenklabs on Hugging Face](https://huggingface.co/yenklabs) when available.
5 changes: 5 additions & 0 deletions models/authority-matching-baseline/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# Dali Authority Matching Baseline

**Status:** planned (v0.3)

Reproducible baseline for authority matching experiments, aligned with Dali evaluation methodology.
5 changes: 5 additions & 0 deletions models/citation-risk-classifier/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# Dali Citation Risk Classifier

**Status:** planned (v0.2)

Estimate citation verification risk using evidence metadata from the [Dali Open Evidence Corpus](../datasets/open-evidence-corpus/) and [Citation Benchmark](../datasets/citation-benchmark/).
5 changes: 5 additions & 0 deletions models/proposition-support-classifier/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# Dali Proposition Support Classifier

**Status:** planned (v0.4)

Classify proposition support relationships for legal authorities — existence vs. support-stage verification.
7 changes: 7 additions & 0 deletions models/verification-taxonomy-classifier/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Dali Verification Taxonomy Classifier

**Status:** planned (v0.1)

Predict standardized verification outcome labels using the [Dali Verification Taxonomy](../datasets/verification-taxonomy/).

Training data and evaluation protocol will ship with the first public model release.
Loading