feat(uipath-agents): discover llm-as-judge models via CLI [AL-515]#2121
feat(uipath-agents): discover llm-as-judge models via CLI [AL-515]#2121apetraru-uipath wants to merge 1 commit into
Conversation
|
Claude finished @apetraru-uipath's task in 1m 5s —— View job PR Review: feat(uipath-agents): discover llm-as-judge models via CLI AL-515
SummaryThis PR replaces the "ask the user for a model ID" guidance in two guardrail reference docs with a CLI-driven discovery step ( Change-by-Change Review1.
|
| Area | Status | Notes |
|---|---|---|
| Frontmatter | OK | Not touched |
| E2E Tests | OK | Existing llm_as_judge smoke task still valid; discovery-path test is a nice-to-have |
| Skill Body | OK | Not touched |
| References & Assets | OK | Both changed files consistent; all relative links resolve |
| Repo Hygiene | OK | No secrets, no cross-skill refs, changes scoped to one skill |
Issues for Manual Review
- CLI command existence: The review assumes
uip agent guardrails llm-as-judge-modelsships in the CLI (PR description references UiPath/cli#3113). Verify the command is available in@uipath/cli@latestbefore this merges, or coordinate release timing.
Conclusion
Clean, well-scoped documentation change. Both files are updated consistently and follow repo conventions (--output json, prescriptive language, graceful fallback). No issues found — approve.
383b5fa to
3ed5571
Compare
| and the message "Response violated the judge rule." The judge rule is: "The agent must | ||
| only provide factual account information and must never offer refunds or price | ||
| negotiation." For the model parameter, discover an available model by running | ||
| `uip agent guardrails llm-as-judge-models --output json` and use one of the returned |
There was a problem hiding this comment.
The test's prompt should not discole HOW the LLM as judge model should be discovered via the CLI. It should be inferred based on the skill's instructions.
…pts [AL-515] The smoke-test prompts spelled out how to find the model (run `uip agent guardrails llm-as-judge-models`). A prompt should read like a normal user request and let the agent infer discovery from the skill's instructions. Remove the "how" from both the low-code and coded prompts; the command_executed check still verifies the skill drove discovery. Addresses andreibalas-uipath's review comment on #2121. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The guardrail skills told the agent to "ask the user which model ID" for an llm_as_judge guardrail. Now that `uip agent guardrails llm-as-judge-models` lists the models available on the tenant, point the skills at it: run the command, pick a ModelId from the result, and fall back to asking only if it returns nothing or fails. - Coded guardrail skills (guardrails.md, guardrails-recommend.md): discover the model via the command instead of asking. - Low-code guardrail skills: add llm_as_judge parity — the enum/text/text-list parameter types, an llm_as_judge row in the validators reference, the model discovery note, and stop treating an empty enum Options list as invalid. - Smoke tests: add a low-code llm_as_judge task (scaffold + assert the agent.json shape) and update the coded task to let the skill drive model discovery. The prompts read like a normal user request — they don't spell out the discovery command; a command_executed check verifies the skill drove discovery. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
c717fe5 to
31255b4
Compare
3 participants
What changed?
The coded guardrail skills (
guardrails.md,guardrails-recommend.md) told the agent to "ask the user which model ID" for anllm_as_judgeguardrail'smodelparameter. Now that the CLI shipsuip agent guardrails llm-as-judge-models(PR UiPath/cli#3113), the skills point at it instead: run the command, pick aModelIdfrom the result, and fall back to asking only if it returns nothing or fails.guardrails.md— the LLM-as-judge tenant-availability note now runsuip agent guardrails llm-as-judge-models --output jsonand recommends a non-preview model (Haiku/mini class as a sound judge default).guardrails-recommend.md— theenum/modelparameter-mapping row now discovers models via the command instead of asking blindly.How has this been tested?
Doc-only change. The referenced command already exists and is tested in the CLI repo. The existing
llm_as_judgetest task (which passes an explicit model) still exercises the valid explicit-model path.Are there any breaking changes?
None — guidance-only. The CLI-side deterministic reviewer rule that validates the chosen model is a separate PR in UiPath/cli.
🤖 Generated with Claude Code