operations-troubleshooting.mdx

title	Operations & Troubleshooting
description	Keep Pullbase healthy and resolve common issues quickly.

Operating Pullbase involves monitoring service health, responding to drift, and collecting diagnostics during incidents.

Health checks

• API health: curl http://localhost:8080/api/v1/healthz
• Database (SQLite): Check the database file exists and is accessible
• Database (PostgreSQL): docker compose exec db pg_isready -U pullbaseuser -d pullbasedb
• Web UI: Sign in and confirm the dashboard loads recent activity.
• Agents: List servers and check last_timestamp in status history via CLI or UI.

Routine operations checklist

- Review drift alerts from status history - Check audit logs for unexpected actions - Ensure webhooks (if enabled) are succeeding - Rotate admin tokens or ensure expiration dates are set - Back up the database (SQLite file or PostgreSQL dump) - Verify TLS certificates (at reverse proxy) aren't approaching expiry - Test agent token rotation on a sample server - Exercise rollback on a staging environment - Review GitHub App access scopes - Run disaster recovery drill (bootstrap new server + restore DB) - Patch base OS/images used for agents and central server hosts

Troubleshooting guide

- Delete `bootstrap-admin-secret.txt` inside the config directory and restart to regenerate. - Or provide a new secret via `PULLBASE_BOOTSTRAP_SECRET` environment variable. - Create a new admin via CLI if another admin exists: `pullbasectl users create` - Token revoked? Generate a new token and update the deployment. - Clock skew? Ensure NTP is running—JWT validation is time-sensitive. - Incorrect `SERVER_ID`? Confirm environment variable matches the server record. - Check outbound network connectivity to the Git provider. - For GitHub Apps, verify installation scopes and ensure the private key matches the app. - Confirm agents trust the TLS certificate (`CACERT_PATH` points to the CA bundle). - Override auto-detection by setting `system.serviceManager` in `config.yaml`. - For Docker containers, use `supervisor` as the service manager. - Ensure service manager commands are in the agent's PATH. - Review container logs around startup for SQL errors. - Migrations run automatically for both SQLite and PostgreSQL. - Check for schema drift caused by manual database edits. - Restore from backup if migrations fail due to corruption. - Inspect container logs: `docker compose logs pullbase` - Ensure the webhook secret in GitHub matches `PULLBASE_WEBHOOK_SECRET_KEY`. - Confirm Pullbase is reachable from your Git provider (network ACLs, firewalls). When TLS is enabled, the server attempts to generate self-signed certificates on startup. If you see:

```
failed to generate self-signed certificates: mkdir certs: permission denied
```

**Option 1: Disable TLS** (recommended if behind a reverse proxy or for testing):
```bash
echo "PULLBASE_TLS_ENABLED=false" | sudo tee -a /etc/pullbase/pullbase.env
sudo systemctl restart pullbase
```

**Option 2: Create the certs directory with proper permissions**:
```bash
sudo mkdir -p /var/lib/pullbase/certs
sudo chown pullbase:pullbase /var/lib/pullbase/certs

# Update env to use this path
sudo tee -a /etc/pullbase/pullbase.env > /dev/null <<'EOF'
PULLBASE_TLS_CERT_PATH=/var/lib/pullbase/certs/server.crt
PULLBASE_TLS_KEY_PATH=/var/lib/pullbase/certs/server.key
EOF

sudo systemctl restart pullbase
```

For production, use CA-signed certificates or terminate TLS at a reverse proxy.

If you see `Failed to execute /usr/local/bin/pullbase-server: Exec format error`:

You downloaded the wrong binary for your system architecture.

```bash
# Check your architecture
uname -m
```

| Output | Binary to download |
|--------|-------------------|
| `x86_64` | `pullbase-server-linux-amd64` |
| `aarch64` or `arm64` | `pullbase-server-linux-arm64` |

Download the correct binary and reinstall:
```bash
sudo systemctl stop pullbase
curl -fsSL -o pullbase-server "https://github.com/pullbase/pullbase/releases/latest/download/pullbase-server-linux-arm64"
sudo install -m 0755 pullbase-server /usr/local/bin/pullbase-server
sudo systemctl start pullbase
```

Collect diagnostics

# Pullbase server logs
docker compose logs pullbase --since 1h

# Database logs
docker compose logs db --since 1h

# Agent logs (container)
docker logs pullbase-agent --since 1h

# Agent logs (systemd)
journalctl -u pullbase-agent --since "1 hour ago"

```bash curl -H "Authorization: Bearer $ADMIN_TOKEN" \ http://localhost:8080/api/v1/environments ``` ```bash curl -H "Authorization: Bearer $ADMIN_TOKEN" \ http://localhost:8080/api/v1/servers/web-01/status/history ``` **SQLite:** ```bash cp /data/pullbase.db pullbase-$(date +%F).db ```

PostgreSQL:

docker compose exec db \
  pg_dump -U pullbaseuser pullbasedb \
  > pullbasedb-$(date +%F).sql

Resetting the installation

If you need a clean slate (lab environments):

1. Stop services: docker compose down
2. Remove data only if you accept data loss:
   • SQLite: docker volume rm pullbase_pullbase_data or delete the database file
   • PostgreSQL: docker volume rm pullbase_postgres_data
3. Start again: docker compose up -d

Removing the database wipes environments, servers, tokens, and audit history. Always back up before destructive operations.

Getting help

• Review GitHub issues for known bugs.
• Collect the diagnostic bundle above before filing a bug report.
• Provide Pullbase version and deployment details.

Keep this playbook handy so you can respond quickly when drift occurs or when an agent experiences connectivity issues.