Skip to content

feat(core): add content preprocessors for llm graders#964

Merged
christso merged 5 commits intomainfrom
feat/963-content-preprocessors
Apr 8, 2026
Merged

feat(core): add content preprocessors for llm graders#964
christso merged 5 commits intomainfrom
feat/963-content-preprocessors

Conversation

@christso
Copy link
Copy Markdown
Collaborator

@christso christso commented Apr 7, 2026

Summary

  • add content-file preprocessing support for llm-grader candidate extraction
  • support suite-level and per-evaluator preprocessors config with command resolution
  • add tests for UTF-8 fallback, command preprocessors, YAML parsing, and llm-grader prompt injection

Status

This is a draft handoff-safe checkpoint for issue #963. The pushed branch passes the repository pre-push checks.

Verification

  • bun test packages/core/test/evaluation/content-preprocessor.test.ts packages/core/test/evaluation/preprocessors-yaml.test.ts packages/core/test/evaluation/llm-grader-multimodal.test.ts packages/core/test/evaluation/validation/eval-schema-sync.test.ts
  • bun x tsc -p packages/core/tsconfig.json --noEmit
  • repository pre-push hooks on git push

Closes #963

@cloudflare-workers-and-pages
Copy link
Copy Markdown

cloudflare-workers-and-pages Bot commented Apr 7, 2026

Deploying agentv with  Cloudflare Pages  Cloudflare Pages

Latest commit: d7becdc
Status: ✅  Deploy successful!
Preview URL: https://886185d6.agentv.pages.dev
Branch Preview URL: https://feat-963-content-preprocesso.agentv.pages.dev

View logs

Copy link
Copy Markdown
Collaborator Author

christso commented Apr 8, 2026

Final verification is complete on the latest branch tip 62b4beca.

What changed since the draft checkpoint:

  • fixed implicit default llm-grader to inherit suite-level preprocessors
  • fixed relative ContentFile.path resolution to use workspace/target base paths
  • fixed MIME-vs-alias preprocessor override merging
  • added regression coverage for those cases

Verification completed:

  • manual CLI e2e via bun apps/cli/src/cli.ts eval ... using a custom file-output target that returned a relative ContentFile path and a custom grader target that only passed when the transformed file text appeared in the grader prompt
  • bun test packages/core/test/evaluation/content-preprocessor.test.ts packages/core/test/evaluation/preprocessors-yaml.test.ts packages/core/test/evaluation/orchestrator.test.ts packages/core/test/evaluation/llm-grader-multimodal.test.ts packages/core/test/evaluation/validation/eval-schema-sync.test.ts
  • bun x tsc -p packages/core/tsconfig.json --noEmit
  • repo pre-push hooks on git push

I’d treat the implementation as ready for review now. The PR is still marked draft only because I haven’t changed the GitHub draft flag via tooling here.

@christso christso marked this pull request as ready for review April 8, 2026 03:59
Copy link
Copy Markdown
Collaborator Author

christso commented Apr 8, 2026

Follow-up docs/example update is pushed on top of the review-ready branch.

New commits:

  • 294d87fd docs(examples): add content preprocessor example
  • d7becdc8 chore(examples): remove generated preprocessor artifact

Added:

  • runnable example under examples/features/preprocessors/
  • docs updates in evaluators/llm-graders.mdx, evaluation/eval-cases.mdx, and evaluation/examples.mdx
  • feature index entry in examples/features/README.md

Verified:

  • bun apps/cli/src/cli.ts eval examples/features/preprocessors/evals/dataset.eval.yaml --target file_output
  • repo pre-push hooks on git push

@christso christso merged commit c3bdc37 into main Apr 8, 2026
4 checks passed
@christso christso deleted the feat/963-content-preprocessors branch April 8, 2026 04:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(core): Content preprocessor for LLM graders to handle binary agent outputs

1 participant