feat(troubleshoot): add artifact-first investigation path for traces and solution files#2088
feat(troubleshoot): add artifact-first investigation path for traces and solution files#2088MariaVimer wants to merge 2 commits into
Conversation
…and solution files Addresses a coverage gap exposed by a production agent failure case (Bilfinger SE, two intermittent failure modes on Gemini 2.5 Flash PTU): the skill had no guidance on extracting evidence from customer-provided artifacts (.uis/.uipx, trace JSON, HAR, screenshots) before issuing CLI commands, and no documented failure class for silent all-null agentOutput or for cross-agent schema/semantic contract mismatches. Changes: - SKILL.md: add §2 step 0 artifact inventory table; extend anchored branch to include .uis/.uipx/.flow/.bpmn; strengthen §4 silent-failure routing to name all-null agentOutput explicitly - references/investigation_guide.md: add Customer-Provided Artifacts section with artifact→insight table, solution file unzip instructions, and per-SpanType trace extraction table - references/products/agents/summary.md: mandate agentOutput span check even on succeeded jobs; add schema naming mismatch and semantic contract mismatch as documented failure classes with detection guidance
…s summary Schema naming mismatch and semantic contract mismatch were derived from one production case (Bilfinger). Insufficient evidence to document them as general failure classes. Retain only the agentOutput null-check mandate, which reflects documented agent runtime behavior.
|
|
||
| When the user provides files directly, extract evidence from them before running CLI commands. Artifact types and what to extract: | ||
|
|
||
| **Solution files (`.uis`, `.uipx`, `.flow`, `.bpmn`, `caseplan.json`)** — zipped archives. Unzip and read: |
There was a problem hiding this comment.
Only uis, uipx are zipped files? Can we generalize this to zip files?
something like, ", zipped archives — unzip, read the content"
There was a problem hiding this comment.
Only uis, uipx are zipped files? Can we generalize this to zip files? something like, ", zipped archives — unzip, read the content"
.uis and .uipx are what we call a 'solution' in our product - when you build any kind of automation, you can export it as a .uis or the entire package as a .uipx.
.uis/.uipx aren't archived files and definitely not a .zip.
They are bundles of related components users build in cloud (RPA workflows, AI agents, Apps, assets, and queues) into a single deployable package.
When a user wants to debug, often you need the solution for a better root-cause analysis (you need to see what is actually running on our platform). That's the .uis/.uipx solution. If Claude/Codex can see that solution it will have better context for root cause analysis.
What
Adds an artifact-first investigation path to the
uipath-troubleshootskill — guidance on extracting evidence from customer-provided files (.uis/.uipx, trace JSON, HAR, screenshots, eval/debug run outputs) before issuinguipCLI commands. Also adds two previously undocumented agent failure classes.Why
Production case (Bilfinger SE) exposed a gap: the skill had no way to direct an investigator to unzip and inspect a
.uissolution file or correlate cross-agent schemas. Both failure modes in that case were only diagnosable from artifacts that were already in hand — not from CLI queries:CHAT_MODEL.UNSUCCESSFUL_STOP_REASON, 0 completion tokens, 4m22s): visible in the trace'scompletionspan, but the contributing factor — a schema naming mismatch where the upstream agent outputs snake_case and the downstream input schema defines PascalCase — was only visible after unzipping both.uisfiles and comparingoutputSchemavsinputSchema.agentOutput, job status = success): a silent failure not surfaced by Orchestrator. Triggered byinputSkontoBetrag: "AmountMatched"— an intentional sentinel value emitted by the extraction agent on a specific data condition — that the matching agent's prompt had no instruction for. Only diagnosable by reading both agents' prompts side by side.Changes
SKILL.mduip tracesanchor step — investigators triage what files they have and what each one yields before reaching for the CLI.uis/.uipx/.flow/.bpmnas primary evidence sourcesagentOutput" explicitly and state that these failures are not detectable from Orchestrator job status alonereferences/investigation_guide.mdSpanTypetrace extraction table (completion,agentRun,agentOutput,llmCall)references/products/agents/summary.mdagentOutputspan even when the job reports successScope
Changes are additive and backward-compatible. No existing playbooks, schemas, or agent files are modified. No hooks touched.
Created with Claude. Claude can make mistakes. Please double-check responses.