LocalAI Docker Compose Setup

Docker Compose configuration for running LocalAI with NVIDIA GPU support. Includes a single-machine dev setup and a distributed multi-node setup.

Prerequisites

• Docker with Compose v2
• NVIDIA GPU with Container Device Interface (CDI) configured
• The local-ai Docker image (locally built from the LocalAI repo)

Quick Start: Dev Mode

Single machine with LocalAI + Caddy (HTTPS for WebRTC):

# Set up Caddy config
cp Caddyfile.example Caddyfile.local
# Edit Caddyfile.local - replace YOUR_HOSTNAME with your machine's hostname

docker compose -f docker-compose.dev.yaml up

LocalAI is available at http://localhost:8081 (HTTP) and https://YOUR_HOSTNAME:8443 (HTTPS via Caddy).

Quick Start: Distributed Mode

Distributed mode splits the frontend API server and GPU workers, connected via NATS.

On the frontend machine

cp Caddyfile.example Caddyfile.local
# Edit Caddyfile.local

docker compose -f docker-compose.distributed.yaml --profile local up

This starts PostgreSQL, NATS, LocalAI frontend, Caddy, and a local GPU worker.

On remote worker machines

cp .env.example .env
# Edit .env - set FRONTEND_HOST to the frontend machine's hostname

docker compose -f docker-compose.distributed.yaml --profile remote up

Configuration

File	Purpose
Caddyfile.local	Caddy reverse proxy config (copy from Caddyfile.example)
.env	Environment variables for distributed mode (copy from .env.example)
volumes/configuration/runtime_settings.json	LocalAI runtime settings (threads, tracing, galleries, agent pool, etc.) - created via the LocalAI UI or manually

Volume Layout

The volumes/ directory is mounted into containers. Directory structure is preserved in git via .gitkeep files; contents are gitignored.

Directory	Container mount	Purpose
volumes/models/	/models	Model weights (GGUF), configs, gallery definitions
volumes/backends/	/backends	CUDA backend libraries
volumes/content/	/tmp/generated/content	Generated audio, images, and videos
volumes/data/	/data	Runtime data, collections, assets
volumes/configuration/	/configuration	Runtime settings JSON
volumes/images/	-	Container images
volumes/postgres/	/var/lib/postgresql	PostgreSQL data (distributed mode)
volumes/worker-models/	/models	Model weights for remote workers

Notes

• Docker image: Both compose files use image: local-ai, a locally built image. Build it from the LocalAI source or change it to an official image like localai/localai:latest-gpu-nvidia-cuda-12.
• User/group: Containers run as user: "1000:100" (typical first Linux user, users group). Adjust if your UID/GID differs.
• GPU driver: Uses CDI (driver: cdi). If your NVIDIA Container Toolkit doesn't support CDI, switch to the older runtime: nvidia approach.