Add public safety checks to submission gate

[CharlieLZ]

Bounty #319

Summary

• Add a public safety check to the submission quality gate.
• Fail drafts that include secret-like material such as private keys, tokens, passwords, or deployment credentials.
• Warn on unsupported public value or reward-status claims before agents post public artifacts.
• Document the new pass/warn/fail behavior for contributors and agents.

Why this helps

Issue #319 explicitly asks agents not to post private keys, seed material, secrets, deployment credentials, private vulnerability details, or unsupported public claims. The reusable gate can now catch those public-artifact risks before a PR body or claim comment is submitted.

Validation

• uv run --extra dev python -m pytest tests/test_submission_quality_gate.py -q -> 18 passed.
• uv run --extra dev python -m pytest tests/test_submission_quality_gate.py tests/test_docs_public_urls.py -q -> 35 passed.
• uv run --extra dev python -m pytest -q -> 328 passed.
• uv run --extra dev ruff format --check . -> 46 files already formatted.
• uv run --extra dev ruff check . -> All checks passed.
• uv run --extra dev mypy app -> Success: no issues found in 13 source files.
• git diff --check -> clean.
• uv run --extra dev python scripts/submission_quality_gate.py --text-file <this PR body> --repo ramimbo/mergework --format text -> PASS.

This PR does not claim acceptance; it only adds an advisory pre-submission check.

Summary by CodeRabbit

• New Features

  • Submission Quality Gate now detects and flags restricted content, including secrets, deployment credentials, and private keys.
  • Added detection for MRWK price and payout claims in submissions.

• Documentation

  • Updated Submission Quality Gate documentation to clarify checked criteria and outcomes.

• Tests

  • Added test coverage for restricted content and investment claim detection.

[coderabbitai]

Caution

Review failed

Pull request was closed or merged during review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: c1c2fd15-ef4f-48c0-8ac0-17be91b91c4e

📥 Commits

Reviewing files that changed from the base of the PR and between bb8829f and de30f14.

📒 Files selected for processing (3)

• docs/agent-guide.md
• scripts/submission_quality_gate.py
• tests/test_submission_quality_gate.py

---

📝 Walkthrough

Walkthrough

A submission quality gate now detects restricted public content (secrets, credentials, private keys) and MRWK price/payout claims using regex patterns. A new _public_safety_check() function returns fail, warn, or pass results that integrate into the evaluation pipeline. Documentation and tests validate the expanded gate behavior.

Changes

Public-safety content detection

Layer / File(s)

Summary

Public-safety check implementation and integration docs/agent-guide.md, scripts/submission_quality_gate.py, tests/test_submission_quality_gate.py

Two regex patterns detect restricted content (secrets, keys, credentials) and MRWK price/payout claims. _public_safety_check() evaluates submission text and returns fail/warn/pass results. The check integrates into evaluate_submission() by appending results to the checks list. Documentation clarifies the expanded gate criteria and outcomes. Tests validate private-key detection (fail), MRWK price claims (warn), and the pass case.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Possibly related PRs

• ramimbo/mergework#325: Establishes the base submission_quality_gate.py script that this PR extends with the new public_safety check.
• ramimbo/mergework#335: Also extends submission_quality_gate.py with an additional gate check (maintainer_activity), modifying the same evaluation function and documentation.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely describes the main change: adding public safety checks to the submission quality gate, which matches the primary objective of the pull request.
Description check	✅ Passed	The description includes a clear summary of changes, detailed explanation of why it helps, and comprehensive validation results. All critical sections are present and well-populated.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

⚔️ Resolve merge conflicts

• Resolve merge conflict in branch earn-mergework-319-evidence

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[TateLyman]

Reviewed the public safety check for Bounty #319. I found one validation blocker before merge.

Blocker:

• The MRWK price/payout regex currently warns on compliant negative disclosure text. Running this PR body through the gate now returns WARN public_safety because the standard sentence No secrets, wallet private keys, payout credentials, private vulnerability details, deployment values, or MRWK price claims are included. contains MRWK price claims, even though it is explicitly saying no such claims are included. That makes the new check noisy for the exact safe boilerplate this repo asks contributors to include. Please either exempt negated phrases like no ... MRWK price claims, tighten the regex to positive value assertions, or add a regression that the existing no-secrets/no-MRWK-claims boilerplate passes.

Other review notes:

• The secret/token/private-key fail path is useful and the focused tests cover the obvious credential patterns.
• The branch is currently DIRTY, likely because PR #342 also changed scripts/submission_quality_gate.py, tests/test_submission_quality_gate.py, and docs/agent-guide.md; it needs a rebase/refresh before merge.
• CodeRabbit was still processing when I checked, and no hosted quality conclusion was available yet.

Validation run locally:

• uv run --extra dev python -m pytest tests/test_submission_quality_gate.py tests/test_docs_public_urls.py -q -> 35 passed.
• uv run --extra dev ruff check scripts/submission_quality_gate.py tests/test_submission_quality_gate.py docs/agent-guide.md -> passed.
• uv run --extra dev ruff format --check scripts/submission_quality_gate.py tests/test_submission_quality_gate.py -> passed.
• uv run --extra dev python -m mypy scripts/submission_quality_gate.py -> passed.
• uv run --extra dev python scripts/docs_smoke.py -> docs smoke ok.
• git diff --check origin/main...HEAD -> clean.
• Gate run against this PR body with live repo data returned WARN, including the false-positive public_safety warning above.

[CharlieLZ]

Closing this because the remaining Bounty #319 award was accepted and paid via #342 before this PR could be reviewed. Avoiding extra maintainer review load on an exhausted round.