Test pass rate 50% is below 100% threshold. Failing pipeline.

Script failed with exit code: 1

7 evaluator(s) failed:
  [P1] CortexConfigurations:Common/Skills/correctness.prompty: score=2 < threshold=3
    Reasoning: Assertion=[SKILL_LOADED: dv-solution]: The agent recommends exporting the solution using a fabricated Python SDK ([L18-L27]), with a 'client.solutions.export' method, which is not part of the official PAC CLI or any recognized Dataverse export SDK. The correct pattern according to the loaded dv-solution skill and expected patterns is the PAC CLI command: 'pac solution export --name ContosoApp --path X.zip' ([Expected Patterns]), which the agent only references for pre-export steps ([L8-L15]) and post-export unpacking ([L33-L38]). The Python import and custom 'get_client' abstraction do not match documented APIs. No explicit PAC CLI export command is shown in response. The skill/plugin should enforce use of documented CLI commands and clarify that no Python SDK exists for solution export, reducing risk of fabricated patterns.
  [P1] CortexConfigurations:Common/Skills/correctness.prompty: score=2 < threshold=3
    Reasoning: Assertion=[CONTAINS: pac solution export]: The agent response never uses 'pac solution export' for exporting [Expected Patterns]. Instead, it suggests a Python SDK method 'client.solutions.export' on [L23-27], which cannot be confirmed as a documented PAC CLI or official SDK pattern. The bash steps on [L7-12] for listing solutions use PAC CLI, but the actual export operation does not comply with 'pac solution export'. There is skill/plugin deviation: skill 'dv-solution' implies use of PAC CLI. There is also a fabricated API ('get_client' and 'client.solutions.export'), which is not part of PAC CLI or a well-known Dataverse SDK. Recommendation: Skill/plugin 'dv-solution' should clarify which APIs/tooling are supported and include introductory PAC export examples to avoid confusion between CLI and Python SDK.
  [P1] CortexConfigurations:Common/Skills/correctness.prompty: score=2 < threshold=3
    Reasoning: Assertion=[CONTAINS: ContosoApp]: The agent recommends a Python SDK export method (client.solutions.export) [L15-L21], but the skill's expected pattern is to use the PAC CLI: 'pac solution export --name ContosoApp --path X.zip' (see Expected Patterns). The response mostly demonstrates CLI commands for listing auth/org and listing solutions [L7-L13], but omits the actual CLI export, which is the documented/expected approach. The Python method and auth.get_client import [L16-L19] are not part of the official PAC SDK/tool routing for Dataverse solution export and risk confusion or non-functionality. Parameters and solution name usage are correct, but the API choice is fundamentally wrong. Recommendation: Skill/plugin should clarify that export must use the 'pac solution export' CLI command, and disallow Python SDK for this operation to avoid ambiguity.
  [P1] CortexConfigurations:Common/Skills/correctness.prompty: score=2 < threshold=3
    Reasoning: Assertion=[Agent uses PAC CLI for the solution export, NOT the Python SDK or Web API. Solution lifecycle is Tier 3 PAC CLI territory.]: The agent recommends exporting the solution using a custom Python SDK approach [L16-L23], specifically client.solutions.export(...), rather than the documented and expected PAC CLI command: 'pac solution export --name ContosoApp --path X.zip', which is Tier 3 and the authoritative method for Dataverse solution exports [Expected Patterns]. While the agent does mention using PAC CLI for other steps (auth/list, solution list, unpack), it fabricates a Python method for export [L17]. The import statement 'from auth import get_client' [L18] is also undocumented, and there is no evidence of this fitting the skill/plugin's required pattern. Correct pattern compliance would require use of 'pac solution export'. Recommendation: The skill/plugin should clarify that only PAC CLI should be used for solution exports, and explicitly disallow custom APIs for this operation.
  [P1] CortexConfigurations:Common/Skills/correctnes

5 evaluator(s) failed:
  [P1] CortexConfigurations:Common/Skills/correctness.prompty: score=1 < threshold=3
    Reasoning: Assertion=[SKILL_LOADED: dv-solution]: The agent was tasked with writing a Python script using the Dataverse SDK to export a solution. Instead, [L7-L38] uses generic REST API calls with 'requests' and manually POSTs to '/api/data/v9.2/ExportSolution'—this is not part of any Dataverse SDK, nor is it a documented API surface for the Python SDK. The dv-solution skill (per Expected Patterns) expects the agent to either refuse, note that there is no SDK solution export API, or route the user to the Power Platform PAC CLI. The agent fabricated a solution export API for Python, violating the requirements. No SDK method, import, or documented workflow is used. The skill/plugin should strengthen guidance/warning messages, so agents explicitly refuse to fabricate SDK APIs and recommend CLI approaches instead.
  [P1] CortexConfigurations:Common/Skills/correctness.prompty: score=1 < threshold=3
    Reasoning: Assertion=[Agent routes to PAC CLI (either as a recommendation, or as a subprocess.run('pac solution export ...') wrapper inside the Python script).]: The agent's response [L1-L38] uses a fabricated HTTP endpoint ('/api/data/v9.2/ExportSolution') and a false JSON payload to attempt a solution export. The Dataverse Python SDK does not expose solution export, and the documented/expected skill pattern is to use the Power Platform CLI (`pac solution export ...`) via a subprocess (or recommend user uses PAC CLI), as noted in Expected Patterns. There is no official Dataverse Web API action called 'ExportSolution', nor is 'ExportSolutionFile' part of the response schema. The agent failed to route to the CLI and instead created a non-existent API. Recommendation: The skill/plugin should explicitly validate user requests for solution export and proactively recommend/subprocess PAC CLI, preventing Web API hallucination.
  [P1] CortexConfigurations:Common/Skills/correctness.prompty: score=1 < threshold=3
    Reasoning: Assertion=[CONTAINS: pac solution export]: The user asked for a script 'using the Dataverse SDK' (meaning the official Python SDK). The expected pattern per skill is that the agent should refuse to fabricate a Dataverse Python SDK method for solution export and should steer the user to use the Power Platform CLI ('pac solution export') instead, potentially invoking it from Python via subprocess (see: Expected Patterns). 

However, the agent's code [no line numbers; see code block] directly crafts an HTTP request to the Dataverse ExportSolution Web API. This is not supported by the Dataverse Python SDK, and the method ExportSolution is not surfaced via SDK Python APIs, nor is there base64 decode of solution blobs in the SDK. The correct (documented) public automation path is via the pac CLI tool. This constitutes fabrication of an API surface and an incorrect approach per both skill pattern and product guidance.

Recommendation: The dv-solution skill/plugin guidance could more explicitly warn against use of direct HTTP/Web API approach for solution export, and require that agents check for official or publicly supported SDK methods, refusing or rerouting when only CLI approaches are valid.
  [P1] CortexConfigurations:Common/SEVAL/LMChecklist.prompty: score=0 < threshold=1
    Reasoning: Assertion=[Agent routes to PAC CLI (either as a recommendation, or as a subprocess.run('pac solution export ...') wrapper inside the Python script).]: The agent response does not route to PAC CLI in any manner. The proposed Python script uses only the Dataverse Web API via the msal and requests libraries. There is no recommendation to use PAC CLI, nor is there any instance of subprocess.run('pac solution export ...') or similar wrapper. The agent solely demonstrates how to accomplish the export via Python and the Dataverse SDK/Web API, without referencing or invoking the PAC CLI.
  [P1] CortexConfigurations:Common/SEVAL/LMCheck