Mission: Consolidate egghead.io onto Coursebuilder
Status: Video migration complete, Content migration next (before users)
Updated: December 13, 2025
For AI agents: Read AGENTS.md first - contains critical rules and context.
- Executive Summary
- The Three Systems
- Migration Phases
- Phase Details
- Key Numbers
- Schema Mapping
- Critical Gaps & Safety
- Repository Structure
- Running the Toolkit
We're killing two legacy systems and consolidating onto Coursebuilder:
┌─────────────────────────────────────────────────────────────────────────────┐
│ │
│ egghead-rails egghead-next Coursebuilder │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ PostgreSQL │ │ Next.js │ │ PlanetScale │ │
│ │ Sidekiq │ + │ GraphQL │ → │ Inngest │ │
│ │ 699K users │ │ Video player│ │ Mux │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
│ KILL KILL KEEP │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
The hard part is already done. All 3,335 active subscriptions use the modern account_subscriptions table. The legacy subscriptions table has been dead since December 2022.
- shadcn/ui centric - Use shadcn components, not complex egghead-next patterns
- Cut the cruft - Don't port complexity, rebuild with simplicity
- No 404s - Every legacy URL gets a redirect (SEO critical)
- Minimal downtime - 5-15 minute cutover window, timed for low traffic
- Build tests as you go - No speculative infrastructure
| System | Role | Database | Fate |
|---|---|---|---|
egghead-rails/ |
Subscriptions, webhooks, users, progress | PostgreSQL | KILL |
egghead-next/ |
Frontend, video player, search | Sanity + Rails API | KILL |
course-builder/ |
Target platform | PlanetScale | KEEP |
┌─────────────────────────────────────────────────────────────────────────────┐
│ MIGRATION ROADMAP │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ Phase 0 Phase 1 Phase 2 Phase 3 │
│ ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐ │
│ │ SAFETY │ → │ DATA │ → │WEBHOOKS│ → │ CRON │ │
│ │ INFRA │ │MIGRATE │ │HANDLERS│ │ JOBS │ │
│ └────────┘ └────────┘ └────────┘ └────────┘ │
│ │ │ │ │ │
│ ▼ ▼ ▼ ▼ │
│ Testing 699K users Stripe 17 Sidekiq │
│ CDC setup 3M progress Inngest → Inngest │
│ Monitoring Content Dual-write │
│ │
│ Phase 4 Phase 5 Phase 6 │
│ ┌────────┐ ┌────────┐ ┌────────┐ │
│ │EXTERNAL│ → │ UI │ → │CUTOVER │ │
│ │INTEGR. │ │COMPNTS │ │ KILL │ │
│ └────────┘ └────────┘ └────────┘ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ Customer.io Video player DNS flip │
│ 17 mailers Search UI Auth cutover │
│ Resend Pricing 🎉 Rails dead │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
Epic: phase-0
Goal: Prove the happy path works, then build safety as we go.
| Task | Bead | Purpose |
|---|---|---|
| ONE E2E test working | phase-0.1 |
User signs up → gets entitlement → can watch video |
| Inngest dev server running | phase-0.2 |
Can test webhook handlers locally |
| Idempotency column added | phase-0.3 |
stripe_event_id on relevant tables |
Not in Phase 0 (build when needed):
- Full testing pyramid - write tests for what you're building
- CDC triggers - sequential cutover eliminates need
- Reconciliation jobs - add post-cutover if drift detected
- Webhook deduplication - shadow mode is read-only, no dual-write
Done When:
- E2E test passes: user signup → entitlement → video access
- Inngest dev server processes test webhook without errors
-
stripe_event_idcolumn exists on relevant tables with unique constraint
See PHASE_DONE_CRITERIA.md for full assessment evidence.
Human Gate: phase-0.4 - Approve migration control plane
Epic: ntu | POC: ✅ Complete (j9b)
Content has no user dependencies - migrate first to enable testing.
Successfully migrated 2 courses to validate approach:
- ✅ Claude Code Essentials (17 lessons) - Modern Sanity source
- ✅ Fix Common Git Mistakes (19 lessons) - Legacy Rails source
- ✅ UI routes working:
/courses/[slug],/lessons/[slug] - ✅ Video playback working via Mux
Key learnings applied:
- Support both
type='course'ANDtype='post' + postType='course' - Use
.passthrough()on Zod schemas for migration metadata - Remove
'use server'from files exporting types/schemas - Always provide
createdById(NOT NULL constraint)
See: reports/POC_LEARNINGS.md
| Data | Records | Bead | Notes |
|---|---|---|---|
| ID mapping tables | - | ntu.1 |
Enable legacy lookups |
| Tags | 627 | ntu.2 |
→ egghead_Tag |
| Courses (series) | 420 | ntu.3 |
→ ContentResource type='course' |
| Lessons | 5,132 | ntu.4 |
→ ContentResource type='lesson' |
| Relationships | 15K+ | ntu.5 |
ContentResourceResource + taggings |
| Sanity merge | - | ntu.6 |
3-way: Rails > Sanity > defaults |
| Validation | - | ntu.7 |
Verify counts and integrity |
Done When:
- 423 courses in CB match Rails count (visibility_state='indexed')
- 11,001 lessons in CB match Rails count (including unpublished)
- All Mux playback IDs linked to videoResources and verified working
See PHASE_DONE_CRITERIA.md for full assessment evidence.
Human Gate: ntu.8 - Review content migration before users
Research: See reports/CONTENT_MIGRATION_RESEARCH.md for full schema mappings.
Epic: phase-1
Migrate users/accounts after content is in place:
| Data | Records | Bead | Status |
|---|---|---|---|
| Users | 699,318 | phase-1.1 |
Ready |
| Organizations (accounts) | 94,679 | phase-1.2 |
Ready |
| Instructors | 134 | phase-1.3 |
Links users→content via ContentContribution |
| Subscriptions | 3,335 active | phase-1.4 |
Ready |
| Progress | 2,957,917 | phase-1.5 |
Needs users + content |
| Gifts | ~500 | phase-1.6 |
Ready |
| Teams | 266 | phase-1.7 |
Ready |
Validation: phase-1.8 - Reconciliation queries
Done When:
- 699,318 users in CB match Rails count
- 3,335 active subscriptions in CB match Rails
account_subscriptionscount - Sample user can log in and see correct entitlements (test with pro + free user)
See PHASE_DONE_CRITERIA.md for full assessment evidence.
Human Gate: phase-1.9 - Approve user migration before execution
Epic: phase-2
Replace Rails Sidekiq handlers with Coursebuilder Inngest:
| Event | Rails Delay | Bead | Status |
|---|---|---|---|
checkout.session.completed |
None | ✅ | Working |
customer.subscription.created |
None | phase-2.1 |
STUB |
customer.subscription.updated |
5 sec | phase-2.2 |
STUB |
customer.subscription.deleted |
None | phase-2.3 |
MISSING |
invoice.payment_succeeded |
1 min | phase-2.4 |
STUB |
Why the delays?
- 5-sec on
subscription.updated: Race condition withcheckout.session.completed - 1-min on
invoice.payment_succeeded: Wait for Stripe to finalize charge
Done When:
- All 4 Stripe events have Inngest handlers implemented
- Test webhook successfully creates subscription in CB
- Idempotency check prevents duplicate processing (store
stripe_event_id)
See PHASE_DONE_CRITERIA.md for full assessment evidence.
Human Gate: phase-2.8 - Approve webhook handler design
Epic: phase-3
Port essential Sidekiq-Cron jobs to Inngest:
| Job | Frequency | Bead | Impact if Missing |
|---|---|---|---|
| StripeReconciler | Daily | phase-3.1 |
Missed transactions |
| GiftExpirationWorker | Daily | phase-3.2 |
Gifts never expire |
| RefreshSitemap | 4 hours | phase-3.3 |
SEO degrades |
| SignInTokenCleaner | 1 minute | phase-3.4 |
Magic links pile up |
| LessonPublishWorker | 10 minutes | phase-3.5 |
Scheduled content stuck |
| RenewalReminder | Daily | phase-3.6 |
No renewal emails |
| Revenue share calculation | Monthly | phase-3.7 |
Instructors not paid |
Done When:
- All 7 critical cron jobs ported to Inngest with schedules
- Jobs run successfully in dev environment on schedule
- No Sidekiq dependencies remain in Coursebuilder codebase
See PHASE_DONE_CRITERIA.md for full assessment evidence.
Epic: phase-4
| Integration | Bead | Notes |
|---|---|---|
| Customer.io API client | phase-4.1 |
Track subscribed/cancelled/billed events |
| Customer.io events | phase-4.2 |
Subscription event tracking |
| Magic link email | phase-4.3 |
PRIMARY auth method |
| Renewal/Welcome emails | phase-4.4 |
Revenue-affecting |
| 17 transactional mailers | phase-4.5 |
Port to Resend |
Done When:
- Customer.io receives subscription events (created/updated/deleted/billed)
- Magic link email sends and authenticates user successfully
- All 17 revenue-affecting mailers ported to Resend and tested
See PHASE_DONE_CRITERIA.md for full assessment evidence.
Human Gate: phase-4.7 - Approve Customer.io + email strategy
Epic: phase-5
| Component | Bead | Notes |
|---|---|---|
| Video player | phase-5.1 |
Mux player, NOT xstate complexity |
| Lesson view | phase-5.2 |
Player + transcript + navigation |
| Course view | phase-5.3 |
Lesson list + progress indicators |
| Search UI | phase-5.4 |
Typesense + InstantSearch, /q/[[...all]] |
| Pricing page | phase-5.5 |
Stripe checkout integration |
| URL redirects | phase-5.7 |
SEO critical |
Done When:
- Video player loads and plays Mux HLS stream with subtitles
- Search returns results and filters work (Typesense + InstantSearch)
- Pricing page completes checkout flow end-to-end in test mode
See PHASE_DONE_CRITERIA.md for full assessment evidence.
Human Gate: phase-5.9 - Approve UI architecture
Epic: axl + 04y
Target: 5-15 minutes of degraded service, not zero downtime.
┌─────────────────────────────────────────────────────────────────────────────┐
│ CUTOVER SEQUENCE │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ Week 1-2: Shadow Mode (READ-ONLY observation) │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ Stripe ──┬──► Rails (PRIMARY) ──► PostgreSQL ──► PROCESSES │ │
│ │ │ │ │ │
│ │ │ │ Compare │ │
│ │ │ ▼ │ │
│ │ └──► Coursebuilder (SHADOW) ─► LOGS ONLY (no mutations) │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ CB receives webhooks but only logs "would have done X" │
│ Target: 7+ days, fix any divergences │
│ │
│ Cutover Day: The Flip (~5 minutes) │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ T-48h: Lower DNS TTL to 60s │ │
│ │ T-1h: Announce maintenance window │ │
│ │ T-0: Remove Rails webhook from Stripe (30 sec) │ │
│ │ Enable CB webhook in Stripe (30 sec) │ │
│ │ Update DNS to Vercel (instant, propagation varies) │ │
│ │ Rails goes read-only │ │
│ │ T+5m: Verify CB processing webhooks │ │
│ │ T+15m: DNS propagated for most users │ │
│ │ T+1h: DNS fully propagated globally │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ Who gets sad during the 5-minute window: │
│ • Magic link clicks → "We just upgraded! Click for new link" (auto-send) │
│ • Mid-checkout users → Stripe handles gracefully (they retry) │
│ • Stale DNS cache → Works within 1 hour │
│ │
│ Week after: Kill Rails │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ Stripe ────────► Coursebuilder ────────► PlanetScale │ │
│ │ │ │
│ │ Rails ─────────────────────────────────► ARCHIVED │ │
│ │ egghead-next ──────────────────────────► ARCHIVED │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ 🎉 Heroku bill: $0 │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
Done When:
- 7 days shadow mode with <0.1% divergence between Rails and CB
- DNS flip completes in <5 minutes with successful webhook processing
- Rails scaled to 0 dynos, Heroku bill $0, PostgreSQL archived read-only
See PHASE_DONE_CRITERIA.md for full assessment evidence.
Human Gates:
phase-6.4- Shadow mode review (7+ days stable)phase-6.7- Auth cutover plan approvalphase-6.9- DNS cutover authorizationphase-6.11- Kill Rails authorization
| Metric | Value | Notes |
|---|---|---|
| Total users | 699,318 | All need migration |
| Total accounts | 94,679 | → Organizations |
| Active subscriptions | 3,335 | All in modern model |
| Monthly new subs | ~93 | All go to modern model |
| Progress records | 2,957,917 | Largest migration |
| Teams | 266 | With 1,200+ members |
| Content | Count | Status |
|---|---|---|
| Courses | 420 | 330 pro, 90 free |
| Lessons | 5,132 | 5,051 published |
| Videos | 7,634 | 97.5% on Mux |
| Instructors | 134 | |
| Tags | 627 |
┌─────────────────────────────────────────────────────────────────────────────┐
│ MUX VIDEO MIGRATION: 81.7% RAILS COVERAGE │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ████████████████████████████████████████░░░░░░░░░░░░ 8,233 / 10,082 │
│ │
│ ✅ Bulk Rails update applied: 6,764 lessons updated │
│ ✅ Gap videos on Mux: 131/146 (15 errored - missing S3 sources) │
│ ✅ SQLite has mux_playback_id for all 6,895 videos │
│ │
│ Remaining ~1,849 without Mux URLs: │
│ • Lessons without videos (text-only, retired) │
│ • Missing S3 source files (193 identified) │
│ • Draft/unpublished lessons │
│ • Coursebuilder-native lessons (served via CB, not Rails) │
│ │
│ See: reports/VIDEO_MIGRATION_STATUS.md │
│ See: reports/ERRORED_MUX_ASSETS_INVESTIGATION.md (15 failed) │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────────────┐
│ RAILS → COURSEBUILDER │
└─────────────────────────────────────────────────────────────────────────────┘
RAILS COURSEBUILDER
═════ ═════════════
┌─────────────────┐ ┌─────────────────────────┐
│ accounts │ │ Organization │
├─────────────────┤ ├─────────────────────────┤
│ id │ ──────────────────▶ │ id (new CUID) │
│ name │ ──────────────────▶ │ name │
│ slug │ ──────────────────▶ │ fields.slug │
│ stripe_customer │ ──┐ │ fields.legacyId │
└─────────────────┘ │ └─────────────────────────┘
│
│ ┌─────────────────────────┐
└────────────────▶ │ MerchantCustomer │
│ identifier = cus_xxx │
└─────────────────────────┘
┌─────────────────┐ ┌─────────────────────────┐
│ account_users │ │ OrganizationMembership │
├─────────────────┤ ├─────────────────────────┤
│ account_id │ ──────────────────▶ │ organizationId │
│ user_id │ ──────────────────▶ │ userId │
│ roles │ ──────────────────▶ │ role │
└─────────────────┘ └─────────────────────────┘
┌─────────────────┐ ┌─────────────────────────┐
│ account_subs │ │ Subscription │
├─────────────────┤ ├─────────────────────────┤
│ account_id │ ──────────────────▶ │ organizationId │
│ status │ ──────────────────▶ │ status │
│ stripe_sub_id │ ──┐ │ merchantSubscriptionId │
└─────────────────┘ │ └─────────────────────────┘
│
│ ┌─────────────────────────┐
└────────────────▶ │ MerchantSubscription │
│ identifier = sub_xxx │
└─────────────────────────┘
┌─────────────────┐ ┌─────────────────────────┐
│ users │ │ User │
├─────────────────┤ ├─────────────────────────┤
│ id │ ──────────────────▶ │ fields.legacyId │
│ email │ ──────────────────▶ │ email │
│ first + last │ ──────────────────▶ │ name │
│ roles: [:pro] │ ──────────────────▶ │ → Entitlement │
└─────────────────┘ └─────────────────────────┘
| Gap | Risk | Mitigation |
|---|---|---|
| Missing idempotency | Duplicate processing | Store stripe_event_id (341.1-2) |
| Post-cutover drift | Data inconsistency | Add reconciliation if drift detected |
| No rollback test | Can't recover | Test flip back to Rails (341.7) |
Note: Dual-write race conditions are avoided by design - shadow mode is read-only observation, not dual processing. Only ONE system processes webhooks at any time.
| Gap | Risk | Mitigation |
|---|---|---|
| Broken URLs | Traffic loss | Pre-migration sitemap snapshot (34t.1) |
| Missing redirects | 404 errors | Comprehensive redirect map (34t.8) |
| Sitemap changes | Ranking drop | Sitemap diff tool (34t.2) |
| Gap | Risk | Mitigation |
|---|---|---|
| OAuth re-linking | Users locked out | Re-link flow (04y.1) |
| OAuth-only users (~45K) | Can't sign in | Password set flow (04y.3-4) |
| Support overload | Slow resolution | Support playbook (04y.5) |
migrate-egghead/
├── AGENTS.md # AI agent instructions - READ FIRST
├── README.md # This file
├── .beads/ # Issue tracking (git-backed)
│
├── course-builder/ # TARGET - Coursebuilder submodule
│ └── apps/egghead/ # egghead app in Coursebuilder
│
├── egghead-rails/ # KILL - Rails backend submodule
│ ├── app/controllers/stripe_events_controller.rb
│ ├── app/models/account_subscription.rb
│ └── app/workers/stripe/ # Sidekiq jobs to port
│
├── egghead-next/ # KILL - Next.js frontend submodule
│ └── src/ # UI components to reference
│
├── download-egghead/ # Media migration toolkit
│ ├── egghead_videos.db # SQLite: courses, lessons, videos
│ └── send-to-mux.mjs # Mux migration script
│
├── investigation/ # Effect-TS analysis toolkit
│ └── src/queries/ # Database exploration scripts
│
└── reports/ # Analysis documents
├── COURSEBUILDER_SCHEMA_ANALYSIS.md
├── STRIPE_WEBHOOK_MIGRATION.md
├── UI_MIGRATION_ANALYSIS.md
├── CUTOVER-RUNBOOK.md
├── DUAL-WRITE-RUNBOOK.md
└── ROLLBACK-RUNBOOK.md
cd investigation
pnpm install
cp .env.example .env
# Add DATABASE_URL (Rails) and NEW_DATABASE_URL (PlanetScale)
pnpm tsx src/queries/subscriptions.ts
pnpm tsx src/queries/table-activity.ts# See what's ready to work on
beads_ready()
# Start a task
beads_start(id="migrate-egghead-phase-0.1")
# Close when done
beads_close(id="migrate-egghead-phase-0.1", reason="E2E test passing")
# Sync to git
beads_sync()Completed:
- ✅ Video migration - 81.7% Rails coverage (8,233/10,082 lessons with Mux URLs)
- ✅ Gap video upload - 131/146 on Mux
- ✅ SQLite playback IDs - 6,895 videos tracked
- ✅ Beads restructure - Human-readable phase-aligned IDs
Current Phase: 1A (Content Migration)
POC complete - 2 courses migrated and verified. Now scaling up test data incrementally.
Scaling test dataset logarithmically to catch edge cases before full migration:
Phase Courses Lessons Coverage Status
─────────────────────────────────────────────────────
1 5 90 1.2% ✅ DONE (edge cases)
2 20 350 6.8% 🔄 IN PROGRESS
3 80 1400 27.3% Pending
4 200 3500 68.2% Pending
5 420 5132 100.0% Full content
Phase 2 adds: Instructor diversity (10+), tag variety (top 20), size extremes (1-lesson AND 50+)
# Target CLI:
bun docker:reset --phase 2 # Use phase 2 dataset
bun docker:reset --phase 3 # Use phase 3 datasetmigrate-egghead-j9b - POC epic (CLOSED)
migrate-egghead-15y - Infrastructure + validation (CLOSED)migrate-egghead-ntu.1 - Create ID mapping tables in PlanetScale
migrate-egghead-ntu.12 - videoResource type and Mux integration
migrate-egghead-ntu.8 - [HUMAN] Review content migrationPOC Artifacts:
investigation/poc-migrate-modern-course.ts- Sanity→CB migration scriptinvestigation/poc-migrate-legacy-course.ts- Rails→CB migration scriptinvestigation/src/lib/migration-utils.ts- Shared utilitiesreports/POC_LEARNINGS.md- Full synthesis of learnings
These beads require explicit human approval before proceeding:
| Gate | Phase | What Needs Review |
|---|---|---|
phase-0.4 |
0 | Migration control plane |
phase-1.9 |
1 | Data migration plan |
phase-2.8 |
2 | Webhook handler design |
phase-4.7 |
4 | Customer.io + email strategy |
phase-5.9 |
5 | UI architecture |
phase-6.4 |
6 | Shadow mode results |
phase-6.7 |
6 | Auth cutover plan |
phase-6.9 |
6 | DNS cutover |
phase-6.11 |
6 | Kill Rails authorization |
┌─────────────────────────────────────────────────────────────────────────────┐
│ │
│ 🎉 RAILS IS DEAD 🎉 │
│ │
│ • All subscriptions managed by Coursebuilder │
│ • All webhooks handled by Next.js + Inngest │
│ • All users authenticated via Coursebuilder │
│ • All content served from PlanetScale │
│ • PostgreSQL archived (read-only backup) │
│ • Heroku bill: $0 │
│ │
└─────────────────────────────────────────────────────────────────────────────┘