Disco CLI

Disco is a CLI tool for managing customer database migrations for Discourse. It handles Docker containers for source databases, converters, and Discourse staging/production environments.

Installation

# Build from source
cd disco
make build

# Install to GOPATH/bin
make install

# Or install globally
sudo make install-global

Quick Start

Prerequisites

Docker running
Go toolchain (to build disco)
On macOS: set DISCO_MIGRATIONS_DIR — the default /migrations is on a read-only volume

Example: v1 Converter Pipeline

# 0. Build disco (if not installed)
cd disco && make build

# 1. (macOS) Override migrations directory
export DISCO_MIGRATIONS_DIR="$HOME/migrations"

# 2. Set up a new customer
disco setup acme-corp --db-type mariadb

# 3. Place your database dump
cp /path/to/dump.sql.gz $DISCO_MIGRATIONS_DIR/acme-corp/data/dumps/v1/

# 4. Start the source database (wait for healthy)
disco start acme-corp --profile source-db

# 5. Set up the converter
disco setup-converter acme-corp

# 6. Build and start the converter container
disco start acme-corp --profile converter --build

# 7. (Dev only) Install disco-rb gem into the converter container
cd disco-rb && gem build disco-rb.gemspec
docker cp disco-rb-*.gem <converter-container>:/tmp/
docker exec <converter-container> gem install /tmp/disco-rb-*.gem --no-document

# 8. Import the database dump (smart import with streaming transforms)
disco db import acme-corp
# Or with turbo mode: disco db import acme-corp --turbo
# (fallback: disco exec acme-corp db restore current/dump.sql.gz)

# 9. Run the conversion
disco exec acme-corp convert <platform>

# 10. Check conversion results
disco exec acme-corp status
disco exec acme-corp validate

# 11. (Optional) Add staging environment and import
disco add-env acme-corp staging
disco start acme-corp --profile staging
disco exec acme-corp:staging import

# Cleanup
disco stop acme-corp
disco destroy acme-corp --yes

Note: Step 7 is only needed during development while the disco-rb gem is not published. Once published, the converter Dockerfile will install it automatically.

Example: v2 Converter Pipeline

The v2 converter uses the core migrations framework built into the Discourse image. No disco-rb gem is needed.

# 0. Build disco (if not installed)
cd disco && make build

# 1. (macOS) Override migrations directory
export DISCO_MIGRATIONS_DIR="$HOME/migrations"

# 2. Set up a new customer with v2 converter
disco setup acme-corp --db-type mariadb --converter-version v2

# 3. Place your database dump
cp /path/to/dump.sql.gz $DISCO_MIGRATIONS_DIR/acme-corp/data/dumps/v1/

# 4. Start the source database (wait for healthy)
disco start acme-corp --profile source-db

# 5. Set up the converter (generates settings.yml, disco-db script)
disco setup-converter acme-corp

# 6. Build and start the converter container
#    (--build needed for MySQL/MariaDB sources; PostgreSQL uses base image)
disco start acme-corp --profile converter --build

# 7. Import the database dump (smart import with streaming transforms)
disco db import acme-corp
# Or with turbo mode: disco db import acme-corp --turbo
# (fallback: disco exec acme-corp db restore current/dump.sql.gz)

# 8. Run the conversion
disco exec acme-corp convert <platform>

# 9. Check conversion results
disco exec acme-corp status
disco exec acme-corp validate

# 10. (Optional) Add staging environment and import
disco add-env acme-corp staging
disco start acme-corp --profile staging
disco exec acme-corp:staging import

# Cleanup
disco stop acme-corp
disco destroy acme-corp --yes

Commands

Project Management

Command	Description
`disco setup <customer>`	Initialize a new migration project (`--converter-version v1\|v2`, `--wizard` for interactive mode)
`disco list`	List all customer projects with status and converter version
`disco status <customer>`	Show detailed status for a customer
`disco destroy <customer>`	Remove all containers and data for a customer

Container Control

Command	Description
`disco start <customer>`	Start containers (use `--profile` for specific services)
`disco stop <customer>`	Stop all containers
`disco exec <customer> <cmd>`	Execute command in converter container
`disco exec <customer>:<env> <cmd>`	Execute command in specific environment

Extension Commands

Command	Description
`disco setup-converter <customer>`	Set up converter (v1: clone repo, v2: generate config)
`disco add-env <customer> <env>`	Add staging or production environment
`disco regenerate <customer>`	Regenerate docker-compose.yml from config

Utility Commands

Command	Description
`disco db-ports`	Show all port allocations
`disco ssh-config`	Generate SSH tunnel configuration
`disco version`	Show version information

Database Commands

Command	Description
`disco db import <customer>`	Import dump with streaming transforms (`--fast`, `--turbo`, `--fix-collations`, `--exclude-tables`)
`disco db replace <customer>`	Reset database and re-import (`--yes` to skip prompt)
`disco db reset <customer>`	Drop and recreate source database
`disco db console <customer>`	Open interactive mysql/psql console
`disco db analyze <customer\|file>`	Analyze dump: table sizes, row counts (`--full` for complete scan)

Database Import Pipeline

The disco db import command provides a smart, streaming import pipeline that replaces the basic disco exec <customer> db restore command.

Performance Modes

Default — Standard import, no modifications
--fast — Disables key checks and unique checks for faster loading
--turbo — Fast mode plus enlarged packet sizes for maximum throughput

Collation Fixing

When importing MySQL 8 dumps into MariaDB, disco db import auto-detects incompatible collations (e.g. utf8mb4_0900_ai_ci) and replaces them with MariaDB-compatible equivalents. Override with --fix-collations or --no-fix-collations.

Table Exclusion

Skip large or unnecessary tables during import:

disco db analyze acme-corp              # See table sizes and row counts
disco db import acme-corp --exclude-tables sessions,cache_entries

Fallback

The existing disco exec <customer> db restore command still works as a simple fallback for cases where the streaming pipeline is not needed.

Interactive Wizard

Use the --wizard flag for an interactive setup experience:

# Full interactive mode
disco setup --wizard

# With dump file for auto-detection
disco setup acme-corp --wizard --dump /path/to/dump.sql.gz

# Auto-detection without wizard (uses detected type directly)
disco setup acme-corp --dump /path/to/dump.sql.gz

The wizard will:

Auto-detect database type from dump file signatures
Guide you through customer name and database type selection
Highlight recommended options based on detection
Confirm configuration before creating

Docker Compose Profiles

Disco uses Docker Compose profiles to selectively start services:

source-db - Source database only
converter - Converter container only
staging - Staging Discourse environment (db, redis, discourse)
production - Production Discourse environment
all - All enabled services

Example:

disco start acme-corp --profile source-db
disco start acme-corp --profile staging
disco start acme-corp  # Starts all enabled services

Directory Structure

Each customer gets an isolated directory. The layout differs by converter version:

v1 customer:

/migrations/<customer>/
├── config.env                 # Customer configuration
├── docker-compose.yml         # Generated compose file
├── config/
│   ├── staging/settings.yml
│   └── production/settings.yml
├── data/dumps/
│   ├── v1/                   # First dump from customer
│   ├── v2/                   # Second dump (if customer sends corrections)
│   └── current -> v1/        # Symlink to active version
├── uploads/
│   ├── source/               # Raw uploads from customer
│   └── processed/            # Ready for import
├── output/                   # Converter output (intermediate.db)
├── discourse-converters/     # Cloned converter repo (v1 only)
└── logs/

v2 customer:

/migrations/<customer>/
├── config.env                 # Customer configuration (CONVERTER_VERSION=v2)
├── docker-compose.yml         # Generated compose file
├── config/
│   ├── converter/
│   │   ├── settings.yml      # Converter settings (generated)
│   │   ├── disco-db           # DB helper script (generated)
│   │   └── Dockerfile         # Custom image (MySQL/MariaDB only)
│   ├── staging/settings.yml
│   └── production/settings.yml
├── data/dumps/
│   ├── v1/                   # First dump from customer
│   └── current -> v1/        # Symlink to active version
├── uploads/
│   ├── source/               # Raw uploads from customer
│   └── processed/            # Ready for import
├── output/                   # Converter output (intermediate.db)
└── logs/

Port Allocation

Disco automatically allocates ports in the 15000-16000 range:

Each customer gets a block of 10 contiguous ports
Port offsets:
- +0: Source database
- +1: Staging web
- +2: Staging database
- +3: Staging Redis
- +4: Production web
- +5: Production database
- +6: Production Redis

View allocations with disco db-ports.

Configuration

Global Configuration

Global settings are stored in ~/.config/disco/config.json:

{
  "port_range_start": 15000,
  "port_range_end": 16000,
  "ports_per_customer": 10,
  "converters_repo": "git@github.com:discourse-org/discourse-converters.git"
}

Customer Configuration

Each customer has a config.env file:

# Converter version (v1 = discourse-converters, v2 = core migrations framework)
# Omitted or empty defaults to v1
CONVERTER_VERSION=v2

# Source database configuration
SOURCE_DB_TYPE=mariadb
SOURCE_DB_PORT=15000
SOURCE_DB_NAME=acme_source_db
SOURCE_DB_PASSWORD=<randomly-generated>

# Staging environment (optional)
STAGING_ENABLED=true
STAGING_WEB_PORT=15001
STAGING_DB_PORT=15002
STAGING_REDIS_PORT=15003

# Production environment (optional)
PRODUCTION_ENABLED=false

# Advanced options (v2 only)
# DISCOURSE_PATH=/path/to/local/discourse  # Mount local Discourse for development

Note: Passwords are randomly generated when creating new customers for improved security.

Converter Versions

Disco supports two converter versions. Both can coexist on the same server (version is per-customer).

v1 (default): Uses the discourse-converters repository cloned into each customer directory. Commands are routed through wrapper scripts and the disco-rb gem inside the converter container. Requires disco-rb installation.

v2: Uses the core migrations framework built into the discourse/discourse_dev:release Docker image. Commands are translated by Go directly to migrations/bin/cli inside the container. No disco-rb or external converter repo needed.

Aspect	v1	v2
Converter source	`discourse-converters` repo	Core Discourse migrations framework
Container image	Built from `discourse-converters/Dockerfile`	`discourse/discourse_dev:release` (+ custom Dockerfile for MySQL/MariaDB)
Command routing	Wrapper scripts -> disco-rb -> converter	Go translates -> `migrations/bin/cli`
`setup-converter` creates	Cloned git repo	`settings.yml`, `disco-db` script, optional Dockerfile
disco-rb required	Yes	No

Environment Variables

DISCO_VERBOSE - Enable verbose output (set to 1, true, yes, or on)
DISCO_MIGRATIONS_DIR - Override base migrations directory (default: /migrations)

SSH Tunnels

Generate SSH config for remote access:

disco ssh-config --host migration-server.example.com >> ~/.ssh/config

# Then connect:
ssh disco-acme-corp-db

Supported Databases

MariaDB 10.11 - Default for MySQL dumps
MySQL 8.0 - For MySQL 8+ specific dumps
PostgreSQL 16 - For PostgreSQL dumps

Shell Completion

# Bash
disco completion bash > /etc/bash_completion.d/disco

# Zsh
disco completion zsh > "${fpath[1]}/_disco"

# Fish
disco completion fish > ~/.config/fish/completions/disco.fish

Testing

Unit Tests (Go)

cd disco && make test

E2E Pipeline Test

Exercises the full converter pipeline (requires Docker and the discourse-converters repo):

bash test/e2e/run.sh

Environment Variable Contract Test

Verifies that the Go compose generator and Ruby config module agree on environment variable names:

bash test/contract/check_env_vars.sh

disco-rb Integration Tests

Status and Validate commands tested against SQLite fixtures:

cd disco-rb && bundle exec rake integration

Name		Name	Last commit message	Last commit date
Latest commit History 2 Commits
cmd/disco		cmd/disco
internal		internal
.gitignore		.gitignore
CLAUDE.md		CLAUDE.md
Makefile		Makefile
README.md		README.md
go.mod		go.mod
go.sum		go.sum

Folders and files

Latest commit

History

Repository files navigation

Disco CLI

Installation

Quick Start

Prerequisites

Example: v1 Converter Pipeline

Example: v2 Converter Pipeline

Commands

Project Management

Container Control

Extension Commands

Utility Commands

Database Commands

Database Import Pipeline

Performance Modes

Collation Fixing

Table Exclusion

Fallback

Interactive Wizard

Docker Compose Profiles

Directory Structure

Port Allocation

Configuration

Global Configuration

Customer Configuration

Converter Versions

Environment Variables

SSH Tunnels

Supported Databases

Shell Completion

Testing

Unit Tests (Go)

E2E Pipeline Test

Environment Variable Contract Test

disco-rb Integration Tests

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages