Skip to content

Add a social approval skill evaluation - #1

Open
kriptoburak wants to merge 2 commits into
lijinnair:mainfrom
kriptoburak:codex/add-social-approval-eval
Open

Add a social approval skill evaluation#1
kriptoburak wants to merge 2 commits into
lijinnair:mainfrom
kriptoburak:codex/add-social-approval-eval

Conversation

@kriptoburak

Copy link
Copy Markdown

Summary

  • Add an evaluation scenario for approval-gated social publishing through an MCP workflow.
  • Update the README evaluation count and tree so the new fixture is discoverable.
  • Assert dry-run default behavior, exact approval before publishing or scheduling, and no secret leakage.

Validation

  • node -e "JSON.parse(require('fs').readFileSync('evaluations/eval-social-approval-skill.json','utf8')); console.log('json ok')"
  • npx -y markdown-link-check README.md --quiet found one pre-existing transient homepage failure: https://lijinnair.com returned 503. Re-run with only that known URL ignored passed.
  • git diff --check

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2f343f5905

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

"expected_behavior": [
"Name is kebab-case, lowercase+hyphens only, and under 64 characters",
"Description starts with a 3rd-person verb and includes 3+ trigger phrases",
"Front matter uses only standard fields and stays under 1024 characters total",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Assert side-effect skills are not auto-invoked

For this approval-gated publishing scenario, the repo's own Step 3 requires disable-model-invocation: true for workflows with controlled side effects such as sending/publishing. This fixture only checks that front matter is standard and that the prose asks for approval, so a generated social-posting skill can pass the evaluation while remaining auto-discoverable/invocable, undermining the safety condition the new eval is meant to cover. Add an expected behavior for the correct invocation flag here.

Useful? React with 👍 / 👎.

@kriptoburak

Copy link
Copy Markdown
Author

Pushed 7ec4701 to add disable-model-invocation: true to the social publishing eval fixture. Validation: python3 -m json.tool evaluations/eval-social-approval-skill.json, git diff --check.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant