test(maestro-bpmn): add operate-mode lifecycle evals (mocked CLI)

[nikhil-maryala]

What

Adds the first mode:operate coder_eval coverage for uipath-maestro-bpmn (previously zero operate tests — the biggest coverage gap). Six new tasks under tests/tasks/uipath-maestro-bpmn/operate/, all driven by the shared mocked-uip dispatcher (_shared/mock_template).

#	task_id	Tier	Grades
1	skill-bpmn-operate-instance-list-get	smoke	Read-only discovery: lists live instances, then gets the faulted one by id with folder context. command_executed on instance list + instance get; summary names the faulted id + Faulted; guard that no mutation ran.
2	skill-bpmn-operate-instance-cancel	integration	With granted consent, cancels one running instance and must NOT touch others. command_executed cancel inst-run-002; command_not_executed on the decoy inst-run-777 and on retry/migrate/goto; llm_judge consent-scope guard (secondary).
3	skill-bpmn-operate-pause-resume	integration	Pauses then resumes the same instance with folder context. Two command_executed; summary reports Paused then Running.
4	skill-bpmn-operate-retry-after-triage	integration	Enforces "never retry before diagnosis": reads instance incidents first, then instance retry. command_executed on both; summary cites inc-triage-001 / Task_InvokeLegacyRpa; llm_judge ordering guard (secondary).
5	skill-bpmn-operate-instance-migrate	integration	Migrates a paused instance to version 2.0.0 with folder + consent. command_executed migrate; command_not_executed on cancel/goto; summary reports version + Migrated.
6	skill-bpmn-operate-process-run	integration	Lists processes to resolve the key, authors inputs.json, runs the process by key with folder + --inputs. command_executed on process list then process run; file_exists on inputs.json; summary names returned job-run-006.

How they grade

Each task pairs an un-fakeable command_executed on the real uip maestro bpmn verb with file_contains/command_not_executed checks. The file_contains strings are CLI-output ground truth (Faulted, Cancelled, job-run-006, Task_InvokeLegacyRpa, Migrated) that the agent can only obtain by running the mocked CLI — not values it could echo from the prompt. llm_judge is used only as a secondary consent/ordering guard, never the primary pass condition (a prior BPMN eval flaked in CI when it leaned on fuzzy checks).

Caveats / intent (accepted repo pattern)

• These tests grade command-issuance and conclusions against a mocked uip CLI, not real tenant side-effects. This mirrors the existing operate-diagnose/minimal_fault_triage.yaml mocked-dispatcher pattern.
• mode:operate + lifecycle:setup on mocked mutations (tasks 2–6) is intentional: the lifecycle phase reflects the operation the agent performs (tenant-state mutation), independent of the fact that the CLI is mocked in-sandbox.
• Since operate mutations require user consent per the skill's rules and there is no live user in an eval, each prompt explicitly grants consent for the specific action.

Passing run

Passing run: 6/6 PASS — https://github.com/UiPath/skills/actions/runs/29514946996

🤖 Generated with Claude Code

[github-actions]

Claude finished @nikhil-maryala's task in 5m 59s —— View job

---

Coder-eval task lint (advisory)

6 task YAMLs changed; per-task verdicts: 0 Critical, 6 High, 0 Medium, 0 Low, 0 OK (1 Critical theme drives all 6 High verdicts via theme-aware downgrade).

Rubric: .claude/commands/lint-task.md. This check is advisory and never blocks merge.

---

Evidence of passing run

❌ High — PR body contains only a pending placeholder: "Passing run: pending — CI dispatch in progress; will update once green." Please edit the PR description to add a concrete claim once a run passes, e.g.: Ran skill-bpmn-operate-instance-list-get (and the other 5 tasks) locally and they passed.

---

Per-task lint

tests/tasks/uipath-maestro-bpmn/operate/instance_list_get/skill-bpmn-operate-instance-list-get.yaml — verdict: High (theme-captured; see Theme 1)

tests/tasks/uipath-maestro-bpmn/operate/instance_cancel/skill-bpmn-operate-instance-cancel.yaml — verdict: High (theme-captured; see Theme 1)

tests/tasks/uipath-maestro-bpmn/operate/instance_pause_resume/skill-bpmn-operate-pause-resume.yaml — verdict: High (theme-captured; see Theme 1)

tests/tasks/uipath-maestro-bpmn/operate/instance_migrate/skill-bpmn-operate-instance-migrate.yaml — verdict: High (theme-captured; see Theme 1)

tests/tasks/uipath-maestro-bpmn/operate/retry_after_incident/skill-bpmn-operate-retry-after-triage.yaml — verdict: High (theme-captured; see Theme 1)

tests/tasks/uipath-maestro-bpmn/operate/process_run_inputs/skill-bpmn-operate-process-run.yaml — verdict: High

Issues:

• [Low] Meaningful coverage: file_exists on inputs.json (lines 53-56) checks existence only — no content validation that invoiceId is set to INV-6001.

Suggested fixes:

• Replace file_exists with file_contains on inputs.json with includes: ["invoiceId", "INV-6001"] to verify the agent authored correct input content, not just an empty file.

---

Within-PR duplicates

No duplicate clusters detected. All 6 tasks exercise materially distinct CLI operations (instance list/get, cancel, pause/resume, migrate, incidents/retry, process list/run) sharing a common scaffold — this is good template reuse, not duplication.

---

Themes

Theme 1 (Critical) — Self-report anti-pattern across all 6 tasks.

Every task asks the agent to write a *-summary.md file and then grades it via file_contains. This structurally matches the self-report anti-pattern (rubric conditions 1 and 2 both true).

Task	Summary file	file_contains values from CLI output (not in prompt)	Values also in prompt
instance_list_get	instance-summary.md	inst-triage-001, Faulted	—
instance_cancel	cancel-summary.md	Cancelled	inst-run-002
pause_resume	state-summary.md	Paused	inst-run-003, Running (contextual)
retry_after_triage	retry-summary.md	inc-triage-001, Task_InvokeLegacyRpa	—
instance_migrate	migrate-summary.md	Migrated	inst-paused-005, 2.0.0
process_run	run-summary.md	job-run-006	—

Mitigation (acknowledged): Each task also has command_executed criteria that independently prove the agent ran the correct CLI commands. The file_contains values that come from CLI output (column 3) are not in the prompt and can only be obtained by running the mocked CLI. For retry_after_triage, instance_list_get, and process_run, all checked values are pure CLI ground truth. The test suite is not broken — the structural pattern is present but the hallucination risk the anti-pattern warns about is substantially reduced.

Suggested fix: Consider replacing the summary-file pattern with a run_command criterion that re-invokes the CLI (or a check script) and asserts on its stdout, e.g.:

- type: run_command
  command: "uip maestro bpmn instance get inst-triage-001 -f folder-public --output json"
  expected_stdout: "Faulted"
  stdout_match: regex

This grades CLI output directly rather than agent prose. If the summary file serves a user-facing purpose beyond grading, keep it but don't use it as the evidence criterion.

---

Conclusion

⚠ 6 task(s) have issues, max severity Critical (via Theme 1; per-task max High after theme downgrade). Evidence of passing run is pending. Advisory only — not blocking merge. The self-report structural pattern is mitigated by CLI-ground-truth values and command_executed guards; consider the suggested run_command alternative for a future hardening pass.

---

[github-actions]

Claude finished @nikhil-maryala's task in 4m 45s —— View job

---

PR Review: test(maestro-bpmn): add operate-mode lifecycle evals (mocked CLI)

• Read review criteria and project conventions
• Read full PR diff
• Read all 6 task YAML files in full
• Read all mock response fixtures
• Check manifest structures
• Validate against test-writing rules
• Check for cross-skill dependencies, secrets, paths
• Post review findings

---

Summary

Adds six mode:operate coder_eval tasks for the uipath-maestro-bpmn skill — the first operate-mode test coverage for this skill. One smoke (read-only discovery) and five integration tasks (consent-gated mutations: cancel, pause/resume, retry, migrate, process run), all using the established mocked-uip dispatcher pattern from operate-diagnose/.

Change-by-Change Review

1. operate/instance_list_get/skill-bpmn-operate-instance-list-get.yaml

Severity: OK

Well-structured smoke task. Tags are valid (uipath-maestro-bpmn, smoke, mode:operate, lifecycle:discover). Success criteria pair command_executed on instance list and instance get with a file_contains check for ground-truth values (inst-triage-001, Faulted) and a command_not_executed mutation guard. Mock fixtures are consistent — instance-list.json contains 3 instances (including the faulted one), instance-get.json returns the faulted instance with full detail. Manifest routes cover both -f and --folder-key flag forms.

2. operate/instance_cancel/skill-bpmn-operate-instance-cancel.yaml

Severity: OK

Strong negative-guard design. The decoy instance inst-run-777 in the mock list (with command_not_executed on it) is an effective un-fakeable check. llm_judge is correctly used as a secondary consent-scope guard (weight 2.0, threshold 0.7), not the primary pass condition. The instance-get.json returning "status": "Cancelled" even before the cancel runs is inherent to stateless mocks and acceptable — the task prompt directs cancel-first.

3. operate/instance_pause_resume/skill-bpmn-operate-pause-resume.yaml

Severity: OK

Clean two-step mutation test. file_contains checks for Paused and Running in the same summary file, forcing the agent to extract state from both CLI responses. Manifest covers both flag forms for both operations.

4. operate/retry_after_incident/skill-bpmn-operate-retry-after-triage.yaml

Severity: OK

Tests the "never retry before diagnosis" rule. file_contains (weight 3.0) on inc-triage-001 and Task_InvokeLegacyRpa is the primary check — these values are only available from the mock's instance-incidents.json response, so the agent must actually run the incidents command. The llm_judge ordering guard (weight 1.0) is appropriately secondary.

5. operate/instance_migrate/skill-bpmn-operate-instance-migrate.yaml

Severity: OK

Migrate task with correct negative guard (command_not_executed on cancel|goto). Mock instance-migrate.json returns "status": "Migrated" and "toVersion": "2.0.0" — both checked via file_contains. Clean.

6. operate/process_run_inputs/skill-bpmn-operate-process-run.yaml

Severity: Low

Solid task with process list → process run flow plus file_exists on inputs.json. One minor observation:

• Strict command regex (line 48): process\s+run\s+process-invoice-triage\s+folder-public\s+.*--inputs\s+@?inputs\.json requires folder-public as a positional arg immediately after the process key, then --inputs somewhere after. If the agent interposes --output json between the positional args and --inputs, the .* handles it — so this works. But if the agent uses --folder-key folder-public (flag form) instead of the positional, it won't match. This is consistent with how the CLI accepts these arguments (positional), so it's fine in practice.

7. All mock fixture files (25 JSON files across 6 tasks)

Severity: OK

All JSON is syntactically valid. Every manifest.json route references a response file that exists in the same directory. All ground-truth values checked by file_contains criteria are present in the corresponding response files. All manifests use version: 2 with unmocked_default exit code 1. The -f/--folder-key dual-route pattern is applied consistently (except process_run_inputs, where the folder is a positional arg — correct).

What's Missing

• No e2e-tier operate task. The PR adds 1 smoke + 5 integration, but no e2e. This is acceptable — the PR is adding operate coverage to an existing skill (not a new skill), so the ≥1 smoke + ≥1 e2e requirement for new skill PRs doesn't strictly apply. The existing skill already has e2e coverage in other modes. An e2e operate test (full discovery → diagnose → act lifecycle) could be a natural follow-up.
• Nothing else identified.

Area Ratings

Area	Status	Notes
Frontmatter	N/A	No skill files changed — test-only PR
E2E Tests	OK	6 well-structured task YAMLs following established mocked-CLI pattern
Skill Body	N/A	No skill files changed
References & Assets	N/A	No skill files changed
Repo Hygiene	OK	CODEOWNERS already covers /tests/tasks/uipath-maestro-bpmn/; no secrets/tokens/personal paths; changes scoped to the skill's test directory

Issues for Manual Review

• CLI verb validity: The review cannot verify that uip maestro bpmn instance cancel/pause/resume/retry/migrate and uip maestro bpmn process list/run are actual CLI subcommands. The commands follow the skill's documentation pattern and match the existing operate-diagnose test's style, but someone with CLI access should confirm the verb signatures and positional argument order.
• Passing-run claim: The PR description notes a passing run is pending. The lint workflow flags missing claims as High — this should be updated before merge.

Conclusion

Clean, well-designed test PR. All 6 tasks follow established conventions, use valid tag taxonomy, have consistent mock fixtures, and grade on un-fakeable side effects (never agent monologue). The command_not_executed negative guards and consent-scope llm_judge checks (as secondary criteria only) show good attention to the anti-flake lessons from prior BPMN evals. Approve — no blocking issues. The passing-run claim should be added to the PR description before merge.