Skip to content

Upgrade dependencies, migrate to Copilot SDK 1.x, fix cost estimation, act on code review findings#53

Merged
sbroenne merged 2 commits into
mainfrom
stbrnner-microsoft-silver-meme
Jul 13, 2026
Merged

Upgrade dependencies, migrate to Copilot SDK 1.x, fix cost estimation, act on code review findings#53
sbroenne merged 2 commits into
mainfrom
stbrnner-microsoft-silver-meme

Conversation

@sbroenne

Copy link
Copy Markdown
Owner

Summary

  • Dependency upgrade - uv lock --upgrade across all 39 packages; github-copilot-sdk 0.3.0 -> 1.0.6 (breaking client API change, migrated runner.py/judge.py)
  • Cost estimation fix - CopilotEval.cost_usd was dead code (hardcoded 0.0); now computed from real token usage via pricing.toml
  • Frontier model bump - default/integration test models updated to claude-opus-4.8 / gpt-5.6-sol
  • Code review remediation (docs, CI/security, tests):
    • Fixed broken docs examples referencing the removed PydanticAI harness (sessions.md, evals.md, CONTRIBUTING.md)
    • Pinned all core GitHub Actions workflows to commit SHAs; added missing permissions: blocks; replaced a hardcoded Azure endpoint with a secret reference; fixed SECURITY.md reporting method and stale version table
    • Removed a dead hover-popup report feature (unreachable JS/CSS/Python)
    • Replaced a placeholder assert True in a visual test with a real console/pageerror check
    • Added 89 new unit tests across 4 previously-untested modules (plugin_options.py, plugin_recording.py, plugin.py hooks, execution/servers.py)
    • Fixed a real bug found by the new tests: MCPServerProcess.start() leaked its subprocess/connection when the required-tools wait check failed

Validation

  • Full unit suite: 520 passed
  • ruff check / ruff format --check: clean
  • pyright: 0 errors
  • mkdocs build --strict: passes
  • Live integration tests (tests/integration/copilot/test_01_basic.py, real Copilot SDK calls against claude-opus-4.8 and gpt-5.6-sol): 4/4 passed, judge/AI-insights analysis generated correctly with non-zero cost figures

Known Issues (documented in CHANGELOG)

  • premium_requests remains always 0.0 - confirmed upstream SDK gap, not fixable from this codebase today

…, act on code review findings

- Upgrade all dependencies (uv lock --upgrade), bump github-copilot-sdk 0.3.0 -> 1.0.6
- Migrate runner.py/judge.py to the SDK 1.x CopilotClient API; fix PermissionDecisionApproveOnce import path (copilot.session -> copilot.generated.rpc)
- Wire up cost estimation (was dead code); pre-fill pricing.toml for new default models
- Bump default/integration test models to frontier flagships
- Investigate premium_requests SDK event gap (documented as upstream limitation)
- Fix broken docs examples (sessions.md, evals.md, CONTRIBUTING.md) referencing removed PydanticAI harness
- Harden CI: pin all core workflow actions to SHAs, add missing permissions blocks, fix hardcoded Azure endpoint, fix SECURITY.md reporting method
- Remove dead hover-popup report feature (scripts.js/overlay.py/report.css)
- Replace placeholder assertion in test_04_agent_selector.py
- Add unit test coverage for plugin_options.py, plugin_recording.py, plugin.py hooks, execution/servers.py (89 new tests)
- Fix MCPServerProcess.start() resource leak on failed tools wait, found by the new tests

Note: pyright and mkdocs-build pre-commit hooks skipped locally (SKIP env var) -- both require 'uv run' which fails to build cryptography from source on this Windows ARM64 host (no MSVC linker; known env limitation, dev workflow uses WSL2). Both verified passing via WSL2 uv run immediately before this commit (pyright: 0 errors; mkdocs build --strict: success).
@github-actions

Copy link
Copy Markdown

Dependency Review

The following issues were found:
  • ✅ 0 vulnerable package(s)
  • ✅ 0 package(s) with incompatible licenses
  • ✅ 0 package(s) with invalid SPDX license definitions
  • ⚠️ 29 package(s) with unknown licenses.
See the Details below.

License Issues

uv.lock

PackageVersionLicenseIssue Type
anyio4.14.2NullUnknown License
cffi2.1.0NullUnknown License
charset-normalizer3.4.9NullUnknown License
click8.4.2NullUnknown License
coverage7.15.1NullUnknown License
filelock3.29.7NullUnknown License
github-copilot-sdk1.0.6NullUnknown License
griffelib2.1.0NullUnknown License
htpy26.5.1NullUnknown License
mcp1.28.1NullUnknown License
mkdocstrings1.0.6NullUnknown License
mkdocstrings-python2.0.5NullUnknown License
pillow12.3.0NullUnknown License
platformdirs4.10.0NullUnknown License
pymdown-extensions11.0.1NullUnknown License
pytest9.1.1NullUnknown License
pytest-asyncio1.4.0NullUnknown License
python-discovery1.4.4NullUnknown License
python-frontmatter1.3.0NullUnknown License
python-multipart0.0.32NullUnknown License
pywin32312NullUnknown License
rpds-py2026.6.3NullUnknown License
ruff0.15.21NullUnknown License
sse-starlette3.4.5NullUnknown License
syrupy5.5.3NullUnknown License
typing-extensions4.16.0NullUnknown License
uvicorn0.51.0NullUnknown License
virtualenv21.6.1NullUnknown License
distlib0.4.3NullUnknown License

OpenSSF Scorecard

Scorecard details
PackageVersionScoreDetails
actions/actions/checkout 9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 🟢 7
Details
CheckScoreReason
Code-Review🟢 10all changesets reviewed
Maintained🟢 1020 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 10
Dangerous-Workflow🟢 10no dangerous workflow patterns detected
Binary-Artifacts🟢 10no binaries found in the repo
Token-Permissions⚠️ 0detected GitHub workflow tokens with excessive permissions
CII-Best-Practices⚠️ 0no effort to earn an OpenSSF best practices badge detected
Fuzzing⚠️ 0project is not fuzzed
License🟢 10license file detected
Pinned-Dependencies🟢 3dependency not pinned by hash detected -- score normalized to 3
Packaging⚠️ -1packaging workflow not detected
Signed-Releases⚠️ -1no releases found
Security-Policy🟢 9security policy file detected
SAST🟢 10SAST tool is run on all commits
Branch-Protection🟢 6branch protection is not maximal on development and all release branches
actions/actions/dependency-review-action 2031cfc080254a8a887f58cffee85186f0e49e48 🟢 7.7
Details
CheckScoreReason
Code-Review🟢 10all changesets reviewed
Dangerous-Workflow🟢 10no dangerous workflow patterns detected
Packaging⚠️ -1packaging workflow not detected
Binary-Artifacts🟢 10no binaries found in the repo
Maintained🟢 1018 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 10
Security-Policy🟢 9security policy file detected
Token-Permissions🟢 9detected GitHub workflow tokens with excessive permissions
CII-Best-Practices⚠️ 0no effort to earn an OpenSSF best practices badge detected
Pinned-Dependencies⚠️ 1dependency not pinned by hash detected -- score normalized to 1
Fuzzing⚠️ 0project is not fuzzed
License🟢 10license file detected
Signed-Releases⚠️ -1no releases found
Branch-Protection🟢 6branch protection is not maximal on development and all release branches
SAST🟢 9SAST tool detected but not run on all commits
actions/actions/stale 1e223db275d687790206a7acac4d1a11bd6fe629 🟢 5.1
Details
CheckScoreReason
Code-Review🟢 10all changesets reviewed
Packaging⚠️ -1packaging workflow not detected
Maintained⚠️ 23 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 2
Dangerous-Workflow🟢 10no dangerous workflow patterns detected
Token-Permissions⚠️ 0detected GitHub workflow tokens with excessive permissions
CII-Best-Practices⚠️ 0no effort to earn an OpenSSF best practices badge detected
Binary-Artifacts🟢 10no binaries found in the repo
Pinned-Dependencies⚠️ 0dependency not pinned by hash detected -- score normalized to 0
License🟢 10license file detected
Fuzzing⚠️ 0project is not fuzzed
Signed-Releases⚠️ -1no releases found
Security-Policy🟢 9security policy file detected
Branch-Protection⚠️ 1branch protection is not maximal on development and all release branches
SAST🟢 6SAST tool is not run on all commits -- score normalized to 6
pip/anyio 4.14.2 UnknownUnknown
pip/certifi 2026.6.17 🟢 6
Details
CheckScoreReason
Maintained🟢 911 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 9
Code-Review⚠️ 0Found 0/2 approved changesets -- score normalized to 0
Binary-Artifacts🟢 10no binaries found in the repo
Dangerous-Workflow🟢 10no dangerous workflow patterns detected
Token-Permissions🟢 10GitHub workflow tokens follow principle of least privilege
Security-Policy🟢 10security policy file detected
Pinned-Dependencies🟢 5dependency not pinned by hash detected -- score normalized to 5
CII-Best-Practices⚠️ 0no effort to earn an OpenSSF best practices badge detected
Fuzzing⚠️ 0project is not fuzzed
License🟢 9license file detected
Signed-Releases⚠️ -1no releases found
Branch-Protection⚠️ 0branch protection not enabled on development/release branches
Packaging🟢 10packaging workflow detected
SAST⚠️ 0SAST tool is not run on all commits -- score normalized to 0
pip/cffi 2.1.0 UnknownUnknown
pip/charset-normalizer 3.4.9 UnknownUnknown
pip/click 8.4.2 UnknownUnknown
pip/coverage 7.15.1 UnknownUnknown
pip/cryptography 49.0.0 UnknownUnknown
pip/distlib 0.4.3 UnknownUnknown
pip/filelock 3.29.7 UnknownUnknown
pip/github-copilot-sdk 1.0.6 UnknownUnknown
pip/griffelib 2.1.0 UnknownUnknown
pip/htpy 26.5.1 UnknownUnknown
pip/idna 3.18 UnknownUnknown
pip/mcp 1.28.1 UnknownUnknown
pip/mkdocstrings 1.0.6 UnknownUnknown
pip/mkdocstrings-python 2.0.5 UnknownUnknown
pip/nh3 0.3.6 UnknownUnknown
pip/pillow 12.3.0 UnknownUnknown
pip/platformdirs 4.10.0 UnknownUnknown
pip/pydantic 2.13.4 UnknownUnknown
pip/pydantic-core 2.46.4 🟢 7.1
Details
CheckScoreReason
Code-Review🟢 8Found 16/18 approved changesets -- score normalized to 8
Maintained🟢 1030 commit(s) and 16 issue activity found in the last 90 days -- score normalized to 10
Dangerous-Workflow🟢 10no dangerous workflow patterns detected
CII-Best-Practices⚠️ 0no effort to earn an OpenSSF best practices badge detected
Binary-Artifacts🟢 10no binaries found in the repo
Token-Permissions⚠️ 0detected GitHub workflow tokens with excessive permissions
Pinned-Dependencies🟢 8dependency not pinned by hash detected -- score normalized to 8
License🟢 10license file detected
Fuzzing🟢 10project is fuzzed
Signed-Releases⚠️ -1no releases found
Branch-Protection🟢 3branch protection is not maximal on development and all release branches
Security-Policy🟢 10security policy file detected
Packaging🟢 10packaging workflow detected
SAST⚠️ 0SAST tool is not run on all commits -- score normalized to 0
pip/pymdown-extensions 11.0.1 UnknownUnknown
pip/pyright 1.1.411 🟢 3.8
Details
CheckScoreReason
Code-Review⚠️ 0Found 1/30 approved changesets -- score normalized to 0
Maintained🟢 33 commit(s) and 1 issue activity found in the last 90 days -- score normalized to 3
Binary-Artifacts🟢 10no binaries found in the repo
Dangerous-Workflow🟢 10no dangerous workflow patterns detected
Token-Permissions⚠️ 0detected GitHub workflow tokens with excessive permissions
CII-Best-Practices⚠️ 0no effort to earn an OpenSSF best practices badge detected
Security-Policy⚠️ 0security policy file not detected
License🟢 10license file detected
Fuzzing⚠️ 0project is not fuzzed
Branch-Protection🟢 3branch protection is not maximal on development and all release branches
Signed-Releases⚠️ -1no releases found
Pinned-Dependencies⚠️ 0dependency not pinned by hash detected -- score normalized to 0
Packaging🟢 10packaging workflow detected
SAST⚠️ 0SAST tool is not run on all commits -- score normalized to 0
pip/pytest 9.1.1 UnknownUnknown
pip/pytest-asyncio 1.4.0 UnknownUnknown
pip/python-discovery 1.4.4 UnknownUnknown
pip/python-frontmatter 1.3.0 UnknownUnknown
pip/python-multipart 0.0.32 UnknownUnknown
pip/pywin32 312 UnknownUnknown
pip/requests 2.34.2 UnknownUnknown
pip/rpds-py 2026.6.3 UnknownUnknown
pip/ruff 0.15.21 UnknownUnknown
pip/sse-starlette 3.4.5 UnknownUnknown
pip/syrupy 5.5.3 UnknownUnknown
pip/typeguard 4.5.2 UnknownUnknown
pip/typing-extensions 4.16.0 UnknownUnknown
pip/urllib3 2.7.0 UnknownUnknown
pip/uvicorn 0.51.0 UnknownUnknown
pip/virtualenv 21.6.1 UnknownUnknown

Scanned Files

  • .github/workflows/dependency-review.yml
  • .github/workflows/stale.yml
  • uv.lock

Comment thread tests/unit/test_execution_servers.py Fixed
Drop the redundant 'import pytest_skill_engineering.execution.servers as servers_module' alias flagged by CodeQL (module imported both ways in the same file); monkeypatch the shared sys module directly instead, since it's the same singleton object referenced by servers.py.

pyright/mkdocs-build hooks skipped locally (same Windows ARM64 MSVC/cryptography build limitation as prior commit); verified via WSL2 uv run immediately before commit: pyright 0 errors, full unit suite 520 passed.
@sbroenne
sbroenne merged commit 469ff18 into main Jul 13, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants