Skip to content

feat: prove Skill utility with paired GLM-5.2 evaluation - #1

Open
YZYY95K wants to merge 19 commits into
mainfrom
feat/behavioral-skill-evaluation
Open

feat: prove Skill utility with paired GLM-5.2 evaluation#1
YZYY95K wants to merge 19 commits into
mainfrom
feat/behavioral-skill-evaluation

Conversation

@YZYY95K

@YZYY95K YZYY95K commented Jul 24, 2026

Copy link
Copy Markdown
Owner

Outcome

Adds a real paired behavior gate for all six DevFlow Skills and removes deployment-model drift.

  • pins runtime, agent, and AgentTeams manifests to glm-5.2
  • adds 12 positive/adversarial cases with contract-bound exact oracles
  • compares the same GLM-5.2 model with and without each Skill
  • requires >=0.90 utility, 1.00 safety, >=0.05 uplift, and zero model/schema errors
  • makes pre-tool invocation/refusal boundaries explicit in every Skill
  • adds CI validation, tests, protocol documentation, and an auditable evidence record

Real model evidence

Final server run: baseline 0.5208, with Skill 1.0000, delta +0.4792, safety 1.0000, errors 0. The initial failing run and corrective iterations are retained in docs/evidence/SKILL_BEHAVIOR_GLM52.md.

Verification

  • 19 tests passed locally and on the deployment host
  • Ruff passed
  • strict mypy passed
  • six Skill static scores: 100/100
  • 12 behavior contracts valid
  • AgentTeams worker package built

The protected main branch still requires CI and an independent approval; this PR intentionally does not bypass that boundary.

@YZYY95K

YZYY95K commented Jul 24, 2026

Copy link
Copy Markdown
Owner Author

Follow-up deployment audit found and fixed a second provider drift: RAG still defaulted to text-embedding-3-small and the legacy endpoint while the deployed chat path uses Z.AI. Commit b948bd0 now uses separately overridable EMBEDDING_* settings and Z.AI embedding-3.

Live probe result: embedding-3 is recognized by the endpoint, but this account currently returns 429 / no embedding resource balance. The README and deployment metadata record that limitation. Chat/Skill evaluation remains operational; production semantic RAG is honestly blocked until an embedding package is enabled or a separately funded embedding provider is configured.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants