test: add SkillOpt eval sets for dv-data, dv-query, dv-overview, dv-connect#66
Open
aadharshkannan wants to merge 7 commits into
Open
test: add SkillOpt eval sets for dv-data, dv-query, dv-overview, dv-connect#66aadharshkannan wants to merge 7 commits into
aadharshkannan wants to merge 7 commits into
Conversation
- auth.py: TracingDataverseClient wraps DataverseClient.records when DATAVERSE_TRACE_FILE env var is set. Logs every SDK CRUD/get call to the trace file and writes created GUIDs to a sibling created_guids.jsonl. No-op when env var unset. Used by SkillOpt's live-verification subsystem for eval items with live blocks. - evals/skillopt/dv_data.jsonl: 3 items migrated from the existing dv_data.biceval.json (one happy_path, one antipattern_trap with bulk flag, one skill_contract). Full 15-25 item set in follow-up commits. Scope note: only SDK records-namespace calls are traced. The raw Web API path (get_token + requests) is not covered in v1.
… import, FK-ordered, DataFrame, sample data, required-field discovery) Brings dv_data.jsonl to 17 items total. Distribution: 7 happy_path, 4 antipattern_trap, 4 edge_case, 1 skill_contract, 1 cross_tool. 3 items have live blocks targeting sko_eval_ticket and sko_eval_account tables in the SkillOptEvals solution.
25 items mirroring the dv_data.jsonl format: - 17 Regression items covering existing dv-query conventions (expand casing, field casing, SQL limits, \ aggregation, cross-table, formatted values, nested \, SDK-first reads, paging, dataframe select, N:N expand, python-only). - 8 Coverage items probing harder Dataverse query traps (lookup _value filter, choice-integer filter, \ pre-filter, \-on-related-field trap, unsupported SQL DISTINCT, null-lookup filter, multi-level nested \). All 25 pass against the current production dv-query skill, so they serve as a regression/coverage benchmark proving the skill handles these conventions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds an intent-alignment regression/coverage benchmark for the dv-overview orchestration skill. Probes verify gpt-5.5 honors the user's chosen entry point (simple read, CLI-only, read-only top-N) without over-steering into forced Python/PAC/MCP setup, while preserving safety (destructive confirm) and legitimate setup/python positive guards. All 6 pass on production gpt-5.5. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ft#63) Motivated by microsoft#63: GitHub Copilot + Dataverse plugin re-prompts for sign-in every session despite valid pac auth, and a simple query hangs ~2 min then returns nothing. Adds evals/skillopt/dv_connect_auth.jsonl targeting the dv-connect skill: - 5 diagnostic probes (reauth-every-session, non-interactive device-code query hang, PAC-vs-Python separate token caches, reject 'just re-run pac auth', corrupt-cache loop) - 2 controls the skill already covers (service principal for CI, --validate preview 403 false-negative) Judged on textual diagnosis/remediation (live_enabled:false). Baseline on the production dv-connect SKILL.md scores hard=0.857 (6/7) — Goldilocks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Move evals/skillopt/dv_query_advanced.jsonl into the consolidated eval-set PR so all SkillOpt eval data lives in one place; PR microsoft#67 keeps only the dv-query SKILL.md change it ships. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
aadharshkannan
added a commit
to aadharshkannan/Dataverse-skills
that referenced
this pull request
Jun 2, 2026
…ft#66 Drop evals/skillopt/dv_query_advanced.jsonl here; it now lives in PR microsoft#66 alongside the other SkillOpt eval sets. This PR ships only the dv-query SKILL.md change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
aadharshkannan
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds SkillOpt-format evaluation items for the Dataverse plugin skills so the skills can be measured and regression-tested by the SkillOpt harness.
Contents
evals/skillopt/dv_data.jsonl— 17 dv-data items (single/bulk create, CSV import, FK-ordered loads, DataFrame loads, sample data, required-field discovery; several live-verified).evals/skillopt/dv_query.jsonl— 25 dv-query items (new):\aggregation, cross-table, formatted values, nested\, SDK-first reads, paging, dataframe select, N:N expand, python-only._valuefilter, choice-integer filter,\pre-filter,\-on-related-field trap, unsupported SQLDISTINCT, null-lookup filter, multi-level nested\.Notes
All 25 dv-query items pass against the current production dv-query skill, so they serve as a regression/coverage benchmark proving the skill handles these conventions (no skill change is proposed here). Each item uses the existing schema:
deterministic(must_contain / must_load_skill) +semanticclaims with priorities.Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com