🤖 codex-ci

Codex as a pre-review quality gate for GitLab Merge Requests.

Status: new. Piloting on a single repository. The mechanism — dedicated isolated runner, ChatGPT-account auth, structured-output review, inline blocking threads plus one summary comment — is proven end-to-end against real Merge Requests; it hasn't yet been rolled out org-wide the way cerberus has.

Why another AI review wrapper

Pointing an AI at a diff is easy. Trusting the result in CI is not: unstructured text can't reliably gate a pipeline, a model that silently times out looks identical to "the code is fine," and a review that comments on every line teaches people to skip it. codex-ci treats those as the actual problems to solve, not the AI review itself:

• Structured, not prose. Codex is forced into a JSON schema (codex exec --output-schema). The pipeline never parses free text to decide pass/fail.
• Infrastructure failure ≠ code passed. No credentials, a network blip, a malformed response — all of it fails the gate loudly, with its own distinct message. Silence is never read as approval.
• Severity, not vibes. Only BLOCKER/HIGH block a merge. MEDIUM/LOW are reported and ignored by the gate — a review that blocks on naming preferences gets turned off within a month.
• Targeted, not spammy. Each BLOCKER/HIGH finding gets its own inline thread, anchored to the exact file:line, so it shows up where the diff is — not a wall of comments for every finding regardless of severity. Everything else (severity counts, MEDIUM/LOW, risk areas, testing gaps) is one collapsed summary note, not one comment per item.

How it works

GitLab MR pipeline (after lint/test/security)
        │
        ▼
 dedicated codex-review runner ──▶ codex exec --output-schema
        │  (isolated: no prod/AWS/deploy       (diff vs target branch +
        │   credentials reachable, ever)         repo's AGENTS.md)
        ▼
   result.json { result, summary, findings[], testing_gaps[], risk_areas[] }
        │
        ├─▶ each BLOCKER/HIGH finding ──▶ its own UNRESOLVED inline diff thread
        │     (falls back into the summary note if it can't be anchored)
        ├─▶ one summary note (severity counts, MEDIUM/LOW collapsed, risk areas)
        └─▶ non-zero exit on FAIL or infra failure ──▶ quality-gate blocks

Codex never calls GitLab's Approve/reject API and never touches Approval Rules — a human approval is still required separately. The inline threads block merge only via the project's own "all threads must be resolved" merge check (see below), which is a different mechanism from approval.

Usage

stages:
  - lint
  - test
  - security
  - codex-review # after your deterministic checks
  - quality-gate # after codex-review

include:
  - remote: https://raw.githubusercontent.com/startmatter/codex-ci/main/templates/gitlab-ci.yml

quality-gate:
  stage: quality-gate
  image: alpine:latest
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
  needs:
    - lint
    - test
    - codex-review
  script:
    - echo "All required checks passed."

quality-gate intentionally has no logic of its own — GitLab already fails the pipeline (and therefore this job never runs) if anything in needs failed. It exists purely as one clear, stable job name to point a "required status check" / branch protection rule at, so you don't have to update branch protection every time an upstream job gets renamed.

Required CI/CD variable

Variable	Scope	Masked	Protected	Purpose
CODEX_GITLAB_BOT_TOKEN	project or group	yes	no	Posts the review note to the MR. Must be unprotected — MR pipelines run on unprotected feature branches, so a protected variable would simply not be injected. Use a dedicated group/project access token (Reporter role, api scope), never a developer's personal token.

Everything else (CI_API_V4_URL, CI_PROJECT_ID, CI_MERGE_REQUEST_IID, ...) is a GitLab-provided predefined variable — nothing else needs to be configured per project.

Required GitLab project setting: all threads must be resolved

For the inline BLOCKER/HIGH threads to actually block merge, enable, per project:

Settings → Merge requests → Merge checks → "All threads must be resolved" (API: PUT /projects/:id with only_allow_merge_if_all_discussions_are_resolved: true)

This is a deliberate manual step, not something this template flips for you automatically: it's a project-wide merge check that also applies to every human-left comment thread, not just Codex's — turning it on is a real behavior change for the whole team's review process, not just for this tool, so it belongs in each team's own hands rather than a side effect of including this template.

Without it, the inline threads are still posted and still useful context, but nothing stops someone from merging past an unresolved one — only the separate quality-gate pipeline status (driven by codex-review.sh's exit code) would still block the merge in that case.

AGENTS.md

If the consuming repository has an AGENTS.md at its root, its content is included in the Codex prompt as repository-specific review instructions (architecture rules, security/tenancy conventions, what matters in that codebase). Optional but recommended — without it, Codex reviews with generic engineering judgment only.

Codex authentication

See docs/codex-auth.md. In short: this template never sees a Codex API key or ChatGPT credential as a CI/CD variable. A dedicated, isolated GitLab Runner (tag codex-review) has a ~/.codex/auth.json credential mounted read-write from its host filesystem into every job container. Rotation happens by replacing that one file on the runner host — no consuming project's .gitlab-ci.yml or CI/CD variables ever change.

Design principles

• The gate blocks on severity, not on Codex having an opinion. MEDIUM/LOW findings are reported, never enforced — an AI review that nitpicks style gets disabled by frustrated engineers within weeks. This one only stops a merge for BLOCKER/HIGH.
• Infrastructure and code quality are different failure modes, and the pipeline never conflates them. A missing credential, a schema mismatch, a timeout — all reported as codex_infra_failure, distinct from result: FAIL, both from the caller's side and in the MR comment itself.
• Least privilege, structurally, not by convention. The runner that talks to OpenAI has never been given AWS keys, deploy tokens, or database credentials — not "shouldn't use them," cannot reach them. codex-review jobs also only ever run on unprotected merge_request_event pipelines, so GitLab itself never injects a project's protected variables into this job in the first place.
• One credential file, not a secret in every project. Auth lives once, on the dedicated runner's disk, auto-refreshing in place. No project's CI/CD variables ever hold a Codex credential; rotating it is a single scp, not a sweep across every consuming repo.
• Untrusted input stays untrusted. MR titles, descriptions, branch names, and diff content are never interpolated into a shell command or hand-built into JSON — they go through a prompt file piped via stdin and through jq when building API payloads.
• Codex blocks via discussion state, not by acting as a reviewer. It posts unresolved inline threads and a pipeline status — both of which a human still has to act on — but it never calls GitLab's Approve/reject endpoints or otherwise participates in Approval Rules. A bot's approval satisfying a "1 approval required" rule would make Codex the final authority by accident; keeping its only levers "unresolved thread" and "pipeline failed" keeps a human's approval meaningful.

Roadmap

• codex exec --output-schema for a JSON result the pipeline can act on, not parse
• Dedicated, isolated runner — no production credentials ever reachable
• ChatGPT-account auth with in-place auto-refresh (no re-provisioning per rotation)
• Single MR summary note for severity counts, MEDIUM/LOW, risk areas, testing gaps
• AGENTS.md support for repository-specific review context
• Proven end-to-end against a real Merge Request pipeline
• Inline diff comments for BLOCKER/HIGH findings with a reliable file:line, anchored via unresolved discussion threads — falls back to the summary note if a finding can't be anchored
• Roll out beyond the first pilot repository
• GitHub Actions variant (currently GitLab-only, unlike cerberus which supports both)
• Per-project severity/behavior tuning (a codex-ci.yml, mirroring cerberus's cerberus.yml)

License

Apache-2.0