Add a social approval skill evaluation - #1
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2f343f5905
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| "expected_behavior": [ | ||
| "Name is kebab-case, lowercase+hyphens only, and under 64 characters", | ||
| "Description starts with a 3rd-person verb and includes 3+ trigger phrases", | ||
| "Front matter uses only standard fields and stays under 1024 characters total", |
There was a problem hiding this comment.
Assert side-effect skills are not auto-invoked
For this approval-gated publishing scenario, the repo's own Step 3 requires disable-model-invocation: true for workflows with controlled side effects such as sending/publishing. This fixture only checks that front matter is standard and that the prose asks for approval, so a generated social-posting skill can pass the evaluation while remaining auto-discoverable/invocable, undermining the safety condition the new eval is meant to cover. Add an expected behavior for the correct invocation flag here.
Useful? React with 👍 / 👎.
|
Pushed 7ec4701 to add disable-model-invocation: true to the social publishing eval fixture. Validation: python3 -m json.tool evaluations/eval-social-approval-skill.json, git diff --check. |
Summary
Validation
node -e "JSON.parse(require('fs').readFileSync('evaluations/eval-social-approval-skill.json','utf8')); console.log('json ok')"npx -y markdown-link-check README.md --quietfound one pre-existing transient homepage failure:https://lijinnair.comreturned 503. Re-run with only that known URL ignored passed.git diff --check