feat: add DeepEval RAG metrics to benchmark pipeline (#361)#362

Open

kiyotis wants to merge 38 commits into

mainfrom

361-ragas-benchmark-metrics

Commits on May 28, 2026

docs: add tasks.md for issue #361 DeepEval RAG metrics

kiyotis
and
claude
committed
docs: update tasks.md — revise T7-T11 for correct benchmark flow

kiyotis
and
claude
committed
docs: clarify T1 — rename to judge LLM connection method investigation

kiyotis
and
claude
committed
docs: update tasks.md — T1 done, add notes.md with investigation results

kiyotis
and
claude
committed
chore: add benchmark requirements.txt and setup.sh install step

kiyotis
and
claude
committed
test: add DeepEval metric computation tests (RED)

kiyotis
and
claude
committed
feat: add DeepEval metric computation to evaluate.py

kiyotis
and
claude
committed
feat: add DeepEval metric columns to benchmark report

kiyotis
and
claude
committed
docs: add DeepEval metrics design to benchmark-design.md

kiyotis
and
claude
committed
fix: support workflow_details fallback in build_deepeval_test_case and add asyncio.run

kiyotis
and
claude
committed
chore: add .deepeval/ to .gitignore

kiyotis
and
claude
committed
fix: set AWS_CA_BUNDLE from SSL_CERT_FILE for aiobotocore SSL in DeepEval

kiyotis
and
claude
committed
docs: update tasks.md — T7完了

kiyotis
and
claude
committed
docs: add DeepEval validation results (SC2)

kiyotis
and
claude
committed
docs: update HOW-TO-RUN.md for DeepEval integration

kiyotis
and
claude
committed
docs: add diff check and update tasks.md — T9/T10/T11完了

kiyotis
and
claude
committed
docs: update tasks.md — 全タスク完了

kiyotis
and
claude
committed
docs: update tasks.md — add T12-T14 for LLM judge removal

kiyotis
and
claude
committed
docs: update tasks.md — T12-T19 with full impact scope

kiyotis
and
claude
committed
docs: update tasks.md — add T19 QA baseline rerun after DeepEval replacement

kiyotis
and
claude
committed
docs: update tasks.md — add reason to scores, remove metrics/diagnostics duplication

kiyotis
and
claude
committed
docs: rewrite benchmark-design.md for DeepEval replacement

kiyotis
and
claude
committed
docs: rewrite HOW-TO-RUN.md for DeepEval replacement

kiyotis
and
claude
committed
test: update tests for DeepEval-only evaluation

kiyotis
and
claude
committed
feat: remove LLM judges from evaluate.py, use DeepEval only

kiyotis
and
claude
committed
feat: remove LLM judge columns from report.py

kiyotis
and
claude
committed
feat: remove --with-deepeval flag, DeepEval always runs

kiyotis
and
claude
committed
docs: update tasks.md — T12-T17 complete

kiyotis
and
claude
committed
docs: update tasks.md — T12-T18 done, T19-T20 remaining

kiyotis
and
claude
committed
chore: opt out of DeepEval telemetry + update tasks.md (T19 run-1 done)

kiyotis
and
claude
committed

Commits on May 29, 2026