Skip to content

feat: add a "Verify before merging" checklist to the integration report#174

Merged
gewenyu99 merged 1 commit into
PostHog:mainfrom
ethangui:feat/verify-before-merging-checklist
Jun 15, 2026
Merged

feat: add a "Verify before merging" checklist to the integration report#174
gewenyu99 merged 1 commit into
PostHog:mainfrom
ethangui:feat/verify-before-merging-checklist

Conversation

@ethangui

@ethangui ethangui commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

Problem

Part of PostHog/wizard#447. After the wizard integrates PostHog there's a gap between "done" and "merged" — verification + glue steps the developer still needs. This adds them as a checklist the agent writes into posthog-setup-report.md.

Companion to the wizard PR PostHog/wizard#595, which surfaces a copy-paste handoff prompt pointing at this checklist.

Changes

llm-prompts/basic-integration/1.3-conclude.md now instructs the agent to write a ## Verify before merging checklist into the report, choosing only items relevant to the run from a curated candidate list.

Why this shape:

  • The model picks the relevant items; the wizard doesn't compute them. Rather than a deterministic, wizard-side per-integration map, the agent judges each candidate against the code it actually changed this run.
  • Curated candidate list + a "Do not invent items" guardrail — judgment within bounds, so it doesn't hallucinate steps.
  • Strategy-agnostic wording — e.g. the LLM item is "trigger the LLM call path(s) you instrumented and confirm $ai_generation…", not assuming OpenTelemetry vs the @posthog/ai wrapper.
  • It lives in the report, not the wizard — this is integration knowledge, so it belongs in context-mill, decoupled from the wizard release cycle.

Test plan

Validated via real wizard runs (real OAuth + agent + PostHog project) against a Next.js 15 SaaS app:

  • the checklist rendered with the run-relevant items and omitted the rest;
  • across runs, $ai_generation appeared only when LLM analytics was set up in that run and was correctly absent otherwise;
  • confirmed the edited skill flows through the context-mill build into the served bundle.

Opening per the repo's contribution invitation in the README. Co-authored with Claude Code (Opus 4.8); follows the PostHog AI policy.

cc @PostHog/team-docs-wizard

🤖 Generated with Claude Code

@gewenyu99 gewenyu99 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Amazing

Comment thread context/skills/integration/references/4-conclude.md
@ethangui ethangui force-pushed the feat/verify-before-merging-checklist branch from 94691b4 to e42862a Compare June 9, 2026 23:20
The basic-integration conclude step now writes a run-relevant checklist
into posthog-setup-report.md; the model selects which items apply.

Refs PostHog/wizard#447.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@ethangui ethangui force-pushed the feat/verify-before-merging-checklist branch from e42862a to 9f307ab Compare June 12, 2026 22:08
@gewenyu99 gewenyu99 merged commit 0b2af74 into PostHog:main Jun 15, 2026
8 of 9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants