test(maestro-case): cover inline resource creation

[song-zhao-25]

Summary

• add an E2E task for creating a missing low-code agent as an in-solution sibling
• add an E2E task for creating a missing pure-compute API workflow as an in-solution sibling
• pin and verify customerName:string input and greeting:string output wiring across the case/resource boundary
• require Agent-tool delegation, sibling registration, local registry audit, and valid case bindings
• validate both sibling projects locally instead of running case debug, which cannot execute a solution containing sibling projects
• discover the generated API case project from the solution manifest instead of assuming its folder matches the case name

Validation

• coder-eval 0.8.3 plan for both task YAMLs
• scripts/check-cli-verbs.py --json for both task YAMLs
• Python compilation for both deterministic checkers
• YAML parse and git diff --check
• agent coder-eval passed with score 1.000: run 29621632043
• API case-project discovery and full checker passed against the artifact from run 29622427925, whose generated case project was named CreateMissingApiWorkflowCaseProject
• local API coder-eval 0.8.3: case-project discovery/validation passed; full checker correctly failed because the generated output mapping omitted load-bearing originalVar: "greeting"

Live E2E status

The revised agent task passes end to end. The API task now tolerates valid Case Management project folder names by resolving the single CaseManagement project from the solution manifest. Its latest local run remains red on a separate skill-output defect: the generated greeting extraction omitted originalVar, even though case validation passed. The grader intentionally retains this assertion because the skill contract marks originalVar as load-bearing.

[github-actions]

Claude finished @song-zhao-25's task in 2m 24s —— View job

---

PR Review: test(maestro-case): cover inline resource creation

• Read review criteria and project conventions
• Analyze full diff
• Read all changed files in full
• Validate task YAML structure and tags
• Check Python checker scripts
• Check fixtures
• Verify no secrets/paths/cross-skill deps
• Post review findings

---

Summary

Adds two well-structured E2E test tasks for uipath-maestro-case: one for inline low-code agent creation and one for inline API workflow creation. Both tasks exercise the create-on-missing gate, Agent-tool delegation, sibling registration, pinned I/O contracts (customerName:string → greeting:string), case bindings, and local validation of both the case and sibling projects. The PR is test-only (no skill changes) and scoped entirely to tests/tasks/uipath-maestro-case/.

Change-by-Change Review

1. tests/tasks/uipath-maestro-case/create_missing_agent/create_missing_agent.yaml

Severity: OK

Well-formed task YAML. Tags include all required dimensions: uipath-maestro-case (skill, first), e2e (tier), lifecycle:generate, mode:build, plus shape:single-node, resource, and feature:registry — all from the closed vocabulary. task_id follows the skill-case-* convention. The initial_prompt is goal-oriented and defers to the skill for workflow. Simulation constraints are thorough and cover all interactive gates. The agent block overrides only what differs from the experiment (adds Agent to allowed_tools, sdk_options, and generous run_limits for an e2e test). Success criteria are appropriately weighted (1.0 → 2.0 → 3.0 → 6.0) and grade via side effects (skill_triggered, command_executed on Agent tool, run_command for case validate, run_command for the full deterministic checker). post_run wires cleanup via the shared script.

2. tests/tasks/uipath-maestro-case/create_missing_agent/check_create_missing_agent.py

Severity: OK

Thorough deterministic checker. Reuses the shared _shared/case_check module (same skill, not cross-skill). Validates: caseplan tasks are properly nested (2D lanes), case validates via CLI, agent task is not a skeleton, both name and folderPath bindings have correct resource/resourceSubType/resourceKey, customerName input wiring via =vars.<id>, greeting output wiring including source, var, and originalVar. Sibling validation checks ProjectType, agent.json exists, entry-point I/O schema, solution registration, and registry audit. The uip agent validate CLI verb is confirmed valid in the catalog snapshot.

Minor observation: assert_workflow_valid in the API checker lacks check=False on subprocess.run (line 199–204), which means a non-zero exit will raise CalledProcessError before the fail() path is reached. Since both paths result in a failure exit, this is functionally correct but inconsistent with the agent checker (line 184) which explicitly passes check=False. Not blocking.

3. tests/tasks/uipath-maestro-case/create_missing_agent/fixtures/sdd.md

Severity: OK

Minimal, well-structured SDD fixture. One stage, one agent task, two case variables (customerName, greeting), <UNRESOLVED> placeholders for folder/identity (exercises the create-on-missing path). No secrets or personal paths.

4. tests/tasks/uipath-maestro-case/create_missing_api_workflow/create_missing_api_workflow.yaml

Severity: OK

Mirrors the agent task structure with appropriate differences: feature:api-workflow added to tags, task_id reflects the API workflow variant, simulation constraints name InlineGreetingApiQ74 and omit the "choose Low-code" step (not applicable for API workflows). The --case-only flag on the weight-3.0 criterion is a smart split: it validates just the case-project discovery path (via solution manifest) before the full checker runs at weight 6.0.

5. tests/tasks/uipath-maestro-case/create_missing_api_workflow/check_create_missing_api_workflow.py

Severity: Low

The API checker is more sophisticated than the agent checker: it discovers the case project from the solution manifest via resolve_caseplan() rather than hardcoding the path. This is the right approach (as noted in the PR description, the generated API case project folder name may differ from the case name). The --case-only mode cleanly separates case-only validation from the full check.

One minor finding:

• subprocess.run without check=False (check_create_missing_api_workflow.py:199): The assert_workflow_valid function calls subprocess.run without check=False. On non-zero exit, Python raises subprocess.CalledProcessError instead of reaching the fail() call on line 205. The error message from CalledProcessError is less informative (no stdout/stderr in the message by default). The agent checker at line 184 passes check=False explicitly. This is a minor inconsistency — functionally both fail, but the error reporting differs.

6. tests/tasks/uipath-maestro-case/create_missing_api_workflow/fixtures/sdd.md

Severity: OK

Properly adapted from the agent SDD: api-workflow task type, InlineGreetingApiQ74 resource, greetingIn output variable (distinct from the agent's greeting), Api binding sub-type, CMW case identifier prefix. All differences are intentional and consistent with the checker expectations.

What's Missing

Nothing identified. CODEOWNERS already covers both /skills/uipath-maestro-case/ and /tests/tasks/uipath-maestro-case/. Both test tasks have appropriate tags, valid CLI verbs, and thorough success criteria. The PR description confirms coder-eval runs, CLI verb checks, Python compilation, and YAML parse — all passing.

Area Ratings

Area	Status	Notes
Frontmatter	N/A	No skill changes — test-only PR
E2E Tests	OK	Two well-structured e2e tasks with deterministic checkers
Skill Body	N/A	No skill changes
References & Assets	OK	SDD fixtures are minimal and correct
Repo Hygiene	OK	No secrets, no cross-skill deps, CODEOWNERS covered

Issues for Manual Review

• The PR body notes the API task's latest local run is "red on a separate skill-output defect" (the generated greeting extraction omits originalVar). This is a known skill-side issue, not a test defect — the grader intentionally retains the originalVar assertion because the skill contract marks it as load-bearing. Worth tracking whether the skill fix lands before this test is expected to pass in CI.

Conclusion

Clean PR. The two E2E tasks are well-designed, properly tagged, and thoroughly validated by deterministic Python checkers that reuse the shared case_check module. The only minor inconsistency is the missing check=False in the API checker's subprocess.run call — a polish item that doesn't affect correctness. Approve.

---

[github-actions]

Claude finished @song-zhao-25's task in 3m 12s —— View job

---

Coder-eval task lint (advisory)

2 task YAMLs changed; verdict counts: 0 Critical, 0 High, 0 Medium, 0 Low, 2 OK.

Rubric: .claude/commands/lint-task.md. This check is advisory and never blocks merge.

Evidence of passing run

• ✅ Author confirms passing run: "agent coder-eval passed with score 1.000: run 29621632043" and "API case-project discovery and full checker passed against the artifact from run 29622427925". API task's full checker intentionally fails on a separate skill-output defect (originalVar omission); author documents this as a grader-correct finding, not a task bug.

Per-task lint

tests/tasks/uipath-maestro-case/create_missing_agent/create_missing_agent.yaml — verdict: OK

No issues. Required tags present (skill, tier:e2e, mode:build). Prompt is goal-oriented without procedure leaks. Python checker (check_create_missing_agent.py) validates caseplan bindings, sibling project.uiproj type, entry-points.json I/O schema, solution manifest registration, local registry audit, and runs uip agent validate — strong coverage that can't be gamed. No env_packages, run-limit fields correctly in run_limits:.

tests/tasks/uipath-maestro-case/create_missing_api_workflow/create_missing_api_workflow.yaml — verdict: OK

No issues. Required tags present (adds feature:api-workflow). Same clean prompt pattern. Python checker (check_create_missing_api_workflow.py) discovers the CaseManagement project from the solution manifest (rather than assuming folder name), validates caseplan bindings including the greetingIn variable indirection, verifies sibling project.uiproj type, entry-points.json schema, solution registration, local registry audit, and runs uip api-workflow validate. The --case-only mode (used in the w=3.0 criterion) provides a lighter validation that verifies solution-manifest-based discovery independently. Strong, non-gameable coverage.

Within-PR duplicates

No duplicate clusters detected. Both tasks share a YAML scaffold (same sandbox, criteria shape, simulation structure) but exercise materially distinct resource types — agent vs API workflow — with different skill delegation targets, different project structures (ProjectType: Agent vs Api), and independent checkers validating type-specific artifacts. Per the rubric's scaffold-reuse carve-out, this is good template reuse.

Conclusion

✅ All changed tasks pass the rubric. Evidence of passing run confirmed.

---