Test pass rate 25% is below 100% threshold. Failing pipeline.

Script failed with exit code: 1

7 evaluator(s) failed:
  [P1] CortexConfigurations:Common/Skills/correctness.prompty: score=2 < threshold=3
    Reasoning: Assertion=[SKILL_LOADED: dv-connect]: The agent directly hardcodes sensitive secrets in source code ([L5], [L6], [L7]), which violates the dv-connect skill compliance. The expected pattern requires using scripts/auth.py or get_credential(), leveraging DefaultAzureCredential or device code auth, and instructing the user to provide secrets via env variables or .env files—not literal values in code. SDK method names and imports are correct ([L1], [L2]), and parameters are properly used ([L9]-[L18]), but the secret handling is a significant security error per the skill requirements. Recommendation: The skill/plugin should enforce secret source checks and require .env usage or environment variables, not code literals.
  [P1] CortexConfigurations:Common/Skills/correctness.prompty: score=2 < threshold=3
    Reasoning: Assertion=[Agent does NOT hardcode a client secret literal in the generated Python file; secrets are loaded from environment or .env via the documented helper.]: The agent hardcodes the client secret in plaintext in the code snippet ([L4], [L8]), violating the expected dv-connect skill pattern, which requires secrets to be loaded via scripts/auth.py or a helper using environment variables or .env files. The agent does not use get_credential() or DefaultAzureCredential as instructed by the skill pattern ([Expected Patterns]). Method and import names themselves are correct ([L1], [L2]), and parameter usage is valid for DataverseClient and ClientSecretCredential ([L10]-[L17]), but the critical error of embedding secrets directly drops compliance to score 2. Recommendation: Improve skill/plugin by explicitly documenting that secrets MUST NOT be written in code snippets and provide a code pattern for loading secrets from env/.env or through the prescribed helper.
  [P1] CortexConfigurations:Common/Skills/correctness.prompty: score=2 < threshold=3
    Reasoning: Assertion=[Agent references scripts/auth.py / get_credential / get_client rather than building an MSAL ConfidentialClientApplication by hand.]: The snippet [L1-L19] correctly uses 'ClientSecretCredential' from 'azure.identity' and 'DataverseClient' from 'PowerPlatform.Dataverse.client', which matches API and import patterns. However, it violates the documented 'dv-connect' skill pattern: it hardcodes secrets ([L5-L7]), rather than routing credential gathering via 'scripts/auth.py', 'get_credential()', or environment variables. There is no instruction for the user to place secrets in .env or to use 'DefaultAzureCredential' or device code flow as required by the skill. This deviation is documented in Expected Patterns and Assertion (agent should reference 'scripts/auth.py / get_credential'). Recommendation: the skill should make it more explicit in examples that hardcoded secrets are prohibited and always show .env or environment variable usage to avoid this error.
  [P1] CortexConfigurations:Common/Skills/correctness.prompty: score=2 < threshold=3
    Reasoning: Assertion=[NOT_CONTAINS: client_secret="]: The response [L1-L18] provides a working snippet using real SDKs (azure-identity and PowerPlatform.Dataverse.client). However, it explicitly hardcodes the client secret directly into the script as a string literal [L7], which goes against the "dv-connect" skill's expected pattern: 'Agent refuses to hardcode secrets in source ... and instructs the user to put secrets in .env or environment variables.' There is no use of environment variables, .env files, or helper functions such as get_credential(). The code should instead demonstrate how to securely load secrets from environment variables. The imports and method usage are otherwise correct. To prevent this, the skill/plugin should enforce or explicitly document the preferred secret management approach and offer a template for credential loading.
  [P1] CortexConfigurations:Common/Skills/correctness

1 evaluator(s) failed:
  [P1] CortexConfigurations:Common/SEVAL/LMChecklist.prompty: score=0 < threshold=1
    Reasoning: Assertion=[Agent demonstrates the canonical Dataverse-skills connect path — references scripts/auth.py (or 'from auth import') AND uses one of the documented credential helpers (get_credential / get_client / get_token / DataverseClient). Hand-rolled OAuth or hardcoded tokens = FAIL.]: The agent does not reference 'scripts/auth.py' or 'from auth import' anywhere in its final script or explanation. The script provided implements authentication directly in the main script, instead of importing or referencing a canonical helper from the Dataverse-skills plugin's auth.py. While it does use 'DataverseClient', which is a documented credential helper, the requirement is to reference the canonical 'connect' path — specifically 'scripts/auth.py' or 'from auth import'. Since this is not satisfied, the assertion fails.

1 evaluator(s) failed:
  [P1] CortexConfigurations:Common/SEVAL/LMChecklist.prompty: score=0 < threshold=1
    Reasoning: Assertion=[Agent mentions load_env or scripts/auth.py as the way the plugin reads the env file.]: The agent's response describes the use of a `.env` file at the workspace root and specifies variable names, but does not mention 'load_env' or 'scripts/auth.py' as the mechanism for reading the environment file. The response is focused on the file and variable names, not how the file is read. Therefore, the assertion fails.