Write Eval for RLS quickstart prompt

[czenko]

Problem

supabase/supabase#48710 adds an <AiPrompt id="row-level-security" /> panel to the Row Level Security guide — an 11-rule checklist telling an agent how to enable RLS, scope grants, write one policy per operation, wrap auth.uid() in a select, index the columns policies filter on, keep cross-table checks behind a security definer helper in a private schema, and prove the result with pgTAP.

Nothing measures whether an agent handed that prompt actually produces working policies and tests. The rules were assembled from the guide itself, the security-rls-basics.md / security-rls-performance.md references in supabase/agent-skills, and the content/row-level-security module of supabase/supacademy — sound sources, but the prompt has only been reviewed by inspection. A rule that reads correctly can still produce SQL that does not run, or policies that pass review and fail to isolate.

Existing coverage does not reach it. build-tests-001-rls-tenant-isolation measures whether an agent can diagnose a planted policy bug from test results; no eval measures whether an agent can author a policy set and a test suite to a specification.

Solution

Adds build-cli-005-quickstart-rls, the sibling of build-cli-004-quickstart-nextjs for the RLS guide's prompt. Same shape: the docs prompt copied verbatim, skills: [] so the prompt is measured standing alone rather than the prompt plus the agent-skills RLS references it was partly built from.

PROMPT.md — the row-level-security entry from ai-prompts.data.ts verbatim, suite: regression, services: [gotrue, kong, postgrest].

local/ — a four-table team wiki with nothing locked down: no RLS, no grants, no policies, no indexes beyond the primary keys, no supabase/tests/. The prompt says how to write policies but not what access the app wants, so the intended rules live in comments at the top of the migration and the agent has to read them. Each omission maps to a rule: documents covers 1-5, and team_documents is what makes 6 and 7 measurable — reading it needs a membership check against a table the comments say clients must never read, so the only design satisfying both is a security definer helper in a private schema. team_members has a (team_id, user_id) primary key, so rule 5's "a column that already leads a primary key does not need another one" is live: only user_id needs a new index.

EVAL.ts — 25 checks, deterministic wherever possible:

• 11 schema and grant checks against the catalog — RLS enabled on all four tables and not forced on the lookup table (rule 1 warns forcing it breaks the helper that reads it), no policy left defaulting to public, a separate policy per operation with no FOR ALL catch-all, using + with check on every update, every auth.uid()/auth.jwt() call wrapped, policy-filtered columns indexed, a search_path-pinned security definer helper in a schema the agent created, team_documents calling it rather than joining team_members, and has_table_privilege for grants in both directions.
• 11 functional checks through two signed-up clients plus anon. This is the anti-gaming half: a policy set that satisfies every catalog check and still leaks fails here. Covers cross-user reads, forged-owner inserts, and the two silent cases — a denied update and a denied ownership reassignment, both verified by the row being unchanged rather than by an error.
• 3 pgTAP checks — test files exist, supabase test db runs at least 8 assertions with none failing, and one judge() over the test SQL for allow/deny coverage per operation across both roles.

Manual testing

Verified against a hand-written reference solution on a real local stack, rather than by inspection.

1. Copy evals/build-cli-005-quickstart-rls/local/ to a scratch directory and supabase start. Confirm the migration applies and no table has RLS, grants, policies, or extra indexes.
2. Apply a compliant reference solution — private schema plus security definer membership helper, RLS on all four tables, per-operation policies scoped to authenticated with (select auth.uid()), btree indexes on documents.owner_id / team_members.user_id / team_documents.team_id, grants for authenticated only.
3. Run each of the scorer's catalog queries with psql against that database. All nine return what the scorer expects: roles renders {authenticated}, quals render ( SELECT auth.uid() AS uid) so the wrapped-call regex matches, indkey[0] finds the leading-column indexes, and the helper is the only security definer outside the stack schemas.
4. Drive the 11 functional checks with two signed-up users and an anon client. All pass; the forged-owner insert returns 42501, and both silent-denial cases leave the target rows intact.
5. Write a 17-assertion pgTAP suite covering allow and deny for all four operations plus anon. supabase test db reports Result: PASS, and the scorer's summary parse reads it correctly.

Two scorer bugs surfaced this way and are fixed in the branch:

• The pgTAP check trusted Tests= / Failed: alone. A suite whose plan does not match the assertions it ran prints Tests: 6 Failed: 0 while exiting non-zero with Result: FAIL — that would have scored as a pass. It now gates on the exit status too.
• The judge rubric penalised any expected error on a denied write. A with check violation or a missing grant genuinely does raise; only a using-filtered row is silent. The rubric now distinguishes the two, so a correct throws_ok is no longer marked wrong.

pnpm eval:dry -- --eval build-cli-005-quickstart-rls --experiment claude-code-sonnet-5,claude-code-sonnet-5-no-skills confirms discovery, frontmatter validation, mode=local-stack, and that skills: [] makes the no-skills experiment report SKIP … (skipEval). tsc --noEmit and pnpm format are clean.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
evals	Ignored		Aug 4, 2026 10:24pm