feat(eval): classify a2ui_scorer failures and add failure_distribution metric (#2200)

[Varun-S10]

Description

Overview

This PR adds failure categorization and a distribution metric to a2ui_scorer so that evaluation runs provide clearer insights into why samples did not pass.

What is Changing

1. Failure Categorization (classify_exception):

   • Classifies errors into specific categories based on exception type and validation details (e.g. validation_error:missing_field, validation_error:type_mismatch, validation_error:extra_field, integrity_error, parse_error).
   • Categorizes pre-validation cases cleanly: no_model_output, solver_step_compilation_failure, and no_a2ui_payload_found.

2. Score Metadata:

   • Saves the category information in Score.metadata (failure_category, coarse_category, error_type, and error details where available).

3. Additive Metric (failure_distribution):

   • Adds a @metric failure_distribution to a2ui_scorer to show counts and proportions of each failure reason across an evaluation run.

4. Backwards Compatibility:

   • Score.value continues to return 1.0 or 0.0, and the accuracy() metric is unchanged so existing baselines and reporting tools stay fully compatible.

5. Tests:

   • Added unit tests in eval/tests/test_scorers.py covering all failure categories and the metric calculation.

Fixes #2200

Pre-launch Checklist

One time:

• I signed the CLA.
• I read the Contributors Guide.
• I read the Style Guide.
• I have built and run at least one sample.

For this PR:

• I have updated the relevant CHANGELOG.md file.
• I updated/added relevant documentation.
• My code changes (if any) have tests.
• If my branch is on a fork, I have verified that scripts/e2e_test.sh passes.

If you need help, consider asking for advice on the discussion board.

[gemini-code-assist]

Code Review

This pull request introduces failure classification and distribution metrics to the A2UI evaluation framework, allowing for detailed categorization of validation, parse, integrity, and compilation errors during scoring. It also updates the ANTLR parser generation hook to fallback to existing files on failure. A critical issue was identified in eval/a2ui_eval/scorers.py where setting ExpressCompilerError = () as an import fallback causes a runtime TypeError during isinstance checks, as nested tuples are not permitted. It is recommended to define a dummy exception class instead.

[gemini-code-assist]

Setting ExpressCompilerError = () as a fallback when the import fails will cause a runtime TypeError inside classify_exception on line 102. In Python, isinstance(e, (A2uiCompileError, A2uiCompilationError, ExpressCompilerError)) will evaluate to isinstance(e, (A2uiCompileError, A2uiCompilationError, ())). Since () is a tuple and not a type, isinstance raises a TypeError when encountering a nested tuple.

To fix this, define ExpressCompilerError as a dummy exception class in the except ImportError block so that it remains a valid type for isinstance checks.

Suggested change

	try:
	from a2ui.inference_formats.experimental.express.errors import ExpressCompilerError
	except ImportError:
	ExpressCompilerError = () # type: ignore[misc,assignment]
	try:
	from a2ui.inference_formats.experimental.express.errors import ExpressCompilerError
	except ImportError:
	class ExpressCompilerError(Exception): # type: ignore[no-redef]
	pass

[Varun-S10]

Applied the suggested change.

[Varun-S10]

Hi @ditman, Could you please take a look?

[FelippeRoza]

Thanks for picking this up. Keeping Score.value and accuracy() unchanged is the part I was most concerned about, since it means the committed baselines under eval/baselines/ stay comparable across the change.

One question on scope: the pack_specs_hook.py hunk looks unrelated to failure classification. Was it meant to be part of this PR?

I ask because I hit that same ANTLR failure setting eval/ up locally, so I am sympathetic to wanting it fixed, but I am not sure the fallback is safe as written. If generation fails at a point where specification/inference_formats/express/Express.g4 has changed, the build now succeeds against the previously generated parser and the only signal is a print to stdout. That would surface later as confusing Express parsing behaviour rather than as a build failure, which seems worse than the current hard error.

In my case the underlying cause was antlr4-tools failing to fetch antlr4-4.13.2-complete.jar even though the Maven URL was reachable from the same machine. Seeding the jar into ~/.m2/repository/org/antlr/antlr4/4.13.2/ was enough for a clean build. A JDK is also required to build a2ui-agent-sdk from source and I could not find that documented anywhere. Happy to file both separately if that would help.

If you would rather keep the fallback, would it be worth gating it on the generated files being newer than Express.g4, so a stale parser still fails the build?