Skip to content

docker.md

Latest commit

 

History

History
350 lines (241 loc) · 21.7 KB

docker.md

File metadata and controls

350 lines (241 loc) · 21.7 KB
Raw

Docker Integration

Access Method: Docker Socket

HELIOS accesses Docker via the Unix socket /var/run/docker.sock.

Why Docker Socket?

  • Industry standard (used by Portainer, Watchtower, etc.)
  • Full Docker API access
  • No additional configuration required
  • Works with Docker Compose operations

Required permissions:

  • Socket must be mounted into HELIOS container
  • HELIOS runs with access to docker group (or root)

File Access

HELIOS needs read/write access to:

  • compose.yaml – to add/modify services
  • .env – to manage environment variables

Solution: Mount the stack directory and the Docker socket into the HELIOS container (see helios.rb).

volumes:
  - .:/data # Compose resolves this relative to compose.yaml
  - /var/run/docker.sock:/var/run/docker.sock

Inside the container, /data holds compose.yaml, .env, and a helios/ subdirectory for config.yaml and HELIOS' own SQLite files.

Docker Compose Operations

HELIOS performs these operations via Docker API / CLI:

Operation How
Read running containers Docker API: GET /containers/json
Start stack docker compose up --no-build -d <services-except-helios>
Stop stack docker compose down <services-except-helios>
Recreate service docker compose pull <svc> → down <svc> → up --no-build -d
Pull new images docker compose pull
View logs docker compose logs (streamed) + Docker API for recent lines
Listen for events Docker API: GET /events (streaming)

Implementation

HELIOS uses a hybrid approach:

1. docker-api Gem – for direct Docker API access

# Gemfile
gem 'docker-api'

# Usage
Docker.url = 'unix:///var/run/docker.sock'
containers = Docker::Container.all
container.logs(stdout: true, tail: 100)
container.json['State']['Health']['Status']

Used for:

  • Reading container status and health
  • Streaming logs
  • Inspecting container details

2. CLI via Open3 – for Docker Compose operations (see Orchestration::Runner)

require 'open3'

def run_compose(*args)
  cmd = [
    'docker', 'compose',
    '-f', ::Compose.path,
    '--project-directory', host_data_path,
    '--env-file', ::Env.path,
    '--progress', 'plain',
    *args,
  ]
  output, status = Open3.capture2e(*cmd)
  raise CommandError, output unless status.success?
  output
end

Used for:

  • up --no-build -d – start stack (all services except helios itself)
  • down – stop stack
  • pull – pull new images
  • pull <svc> → down <svc> → up --no-build -d <svc> – recreate a service
  • logs -f --timestamps – live log streaming via IO.popen
  • config --hash '*' – detect services whose effective config has drifted

3. Events listener – streams GET /events from the Docker API in a background thread (see Orchestration::EventsListener and events_listener/streaming.rb) and broadcasts service status changes to the browser via Turbo Streams. Only active while at least one subscriber is connected.

Rationale: The docker-api gem provides clean Ruby access to container info and logs, but cannot execute docker compose commands — those require the CLI because they involve YAML parsing, service dependencies, and network setup that only Compose handles. The events listener complements both: it turns Docker's push-based event stream into live UI updates instead of per-request polling.

Generated Files

HELIOS generates two files in the stack directory on every configuration change (and before every compose operation). Both are fully owned by HELIOS — except for parts explicitly tracked as "unmanaged" (user-added services / env vars).

compose.yaml

Generated by Export::Builder, which iterates over all Export::Services::* classes and includes each whose enabled?(configuration) predicate returns true.

Service definitions (see app/services/export/services/):

Service Always / conditional
helios Always (except in development, where HELIOS runs natively)
dashboard Always, except in collectors-only mode
power_splitter When grid_import_power, house_power and ≥1 further consumer are mapped
forecast_collector When a forecast is configured (forecast_required? and a forecast set)
postgresql Always, except in collectors-only mode
redis Always, except in collectors-only mode
influxdb Always, except in collectors-only mode (an external InfluxDB is used)
watchtower Always
senec_collector When a sensor is mapped to a SENEC source
shelly_collector One instance per configured Shelly device
mqtt_collector When a sensor is mapped to an MQTT source
ingest When external push source (ioBroker / Home Assistant) is used
traefik When HTTPS / reverse proxy is enabled

Images come from a mix of hard-coded defaults and user-configurable values (configuration.<service>.image in config.yaml) — see each service class for details. Versioning follows the strategy in Image Versioning Strategy below.

Each service declares its own healthcheck, and dependent services use depends_on: service_healthy. Unmanaged services (imported from existing installations) are appended verbatim.

.env

Generated by Export::Env from the config.yaml singletons, on top of the low-level Env::File parser. Comments and unknown variables from an existing .env are preserved on round-trip (see ADR-0008).

Variables include timezone, installation date, admin password, all secrets (DB passwords, InfluxDB token, SECRET_KEY_BASE), INFLUX_SENSOR_* mappings, and per-service connection settings. Secrets are auto-generated via SecureRandom on first setup and persisted in config.yaml.

Installation Scenario Detection

HELIOS detects the installation scenario at startup (see product.md for details).

Detection method: Import::StackReader parses the existing compose.yaml / .env; Import::ConfigurationImporter classifies what it finds. If the stack contains no services other than HELIOS, the setup wizard runs. If a local target (dashboard / influxdb) or any known collector service is present, an import pass pre-fills config.yaml. A collectors-only install (collectors present, but no local dashboard/InfluxDB) is detected via ConfigurationImporter#collectors_only?.

Scenario A/B (Fresh install): Only HELIOS service exists → setup wizard. User configures devices and selects a data source per device (SENEC/Shelly/MQTT or ioBroker/HA). Collector services are generated only for devices with direct hardware data sources. The distinction between standalone and smart home setups is implicit — no separate wizard question.

Scenario C (Existing installation): Other services present → HELIOS automatically imports compose.yaml and .env on first access, reverse-maps configuration into internal singletons (best-effort), and shows the result to the user for review.

Error Handling

Docker Not Reachable

If Docker socket is not accessible (not mounted, daemon stopped):

  • HELIOS shows a dedicated error page
  • Message: "Cannot connect to Docker"
  • Troubleshooting hints provided (check socket mount, Docker daemon status)
  • No other functionality available until resolved

Missing or Corrupt compose.yaml

If compose.yaml is missing or invalid after initial setup:

  • HELIOS can regenerate the file from config.yaml (see ADR-0009)
  • User is prompted: "Configuration file missing. Regenerate from saved state?"
  • Regeneration restores all HELIOS-managed services
  • User-added services cannot be recovered (warning shown)

Stack Detection

HELIOS identifies containers in its stack via Docker Compose labels. Compose sets these on every container it manages:

  • com.docker.compose.project – project name
  • com.docker.compose.service – service name from compose.yaml
  • com.docker.compose.config-hash – hash of the effective service config (used to detect drift)

Fixed project name. Instead of deriving the project name at runtime, HELIOS requires a fixed value:

# app/services/orchestration.rb
PROJECT_NAME = 'solectrus'.freeze

On startup, StartupCheck#check_compose_project_name parses the top-level name: in compose.yaml and refuses to proceed unless it equals solectrus. This guarantees that container lookups by label always hit the right project, regardless of the host directory name.

Containers are then enumerated via the Docker API and filtered by com.docker.compose.project=solectrus (see Orchestration::Container.all).

Image Versioning Strategy

Defaults live in the DockerImages registry (each entry carries the recommended :current tag plus :legacy tags that surface as an "update available" hint); ConfigSchema pulls them in for fresh setups. Per-service images can be overridden via config.yaml. For own services the registry offers latest (the default) and develop as selectable variants.

Service Image Tag Rationale
SOLECTRUS Dashboard ghcr.io/solectrus/solectrus:latest Own service, always latest
HELIOS ghcr.io/solectrus/helios:latest Own service, always latest
Power-Splitter ghcr.io/solectrus/power-splitter:latest Own service, always latest
Forecast-Collector ghcr.io/solectrus/forecast-collector:latest Own service, always latest
SENEC-Collector ghcr.io/solectrus/senec-collector:latest Own service, always latest
MQTT-Collector ghcr.io/solectrus/mqtt-collector:latest Own service, always latest
Shelly-Collector ghcr.io/solectrus/shelly-collector:latest Own service, always latest
Ingest ghcr.io/solectrus/ingest:latest Own service, always latest
PostgreSQL postgres:18-alpine Major version pinned
Redis redis:8-alpine Major version pinned
InfluxDB influxdb:2.9-alpine Minor version pinned
Traefik traefik:v3.7 Minor version pinned
Watchtower nickfedor/watchtower:latest Fork with additional features

Rationale:

  • Own services default to latest – Watchtower handles updates automatically; develop is available for users who want pre-release builds
  • Third-party services pin major version – prevents breaking changes, allows minor/patch updates

Update Pause During Long-Running Operations

A Watchtower update recreates the container it updates. That is fatal in the middle of an operation that spans several containers and minutes to hours: InfluxDB disappearing halfway through a CSV import truncates it, PostgreSQL recreated between the dump and the restore of a major upgrade loses the cluster.

Orchestration::UpdatePause therefore freezes Watchtower (docker compose pause watchtower) for the duration of a CSV import, a backup, a restore, a PostgreSQL major upgrade and a single-service recreate. Nothing in compose.yaml/.env changes, so the mechanism also covers adopted stacks that were never redeployed, and it protects every service at once rather than the ones a given operation happens to touch.

The pause starts with the operation, not with its container: every runner spends minutes on HELIOS-side preparation first (pull the docker:cli image, probe an external destination, download a backup from S3, and for a restore rewrite config.yaml and compose.yaml), and an update landing there recreates HELIOS itself — the preparation dies with the process and the run never happens. pause_updates! is therefore the first step of every runner, preceded only by the lock checks that can still reject the run.

Docker's own pause rather than a stop: the container keeps its identity, its logs and its place in the stack, and docker compose unpause brings it back in a fraction of a second. It also reports itself — paused is a state Docker returns, so the row renders it through the ordinary status path (label "Paused", blue dot) instead of a special case. The one thing HELIOS still overrides is the start button, which stays disabled: starting a paused container does nothing, so offering it would only promise to undo the protection.

The recreate is included because ComposeJob rewrites compose.yaml and then runs pull → down <svc> → up <svc>: an automatic update landing in that window either kills HELIOS before the up (leaving the service down) or rebuilds a container from the spec that was just replaced. The batch actions are deliberately left out — they bring Watchtower up or down themselves — as are start/stop, which have no down-then-up window, and Watchtower's own recreate, which brings it straight back.

The failure mode to avoid is a leaked pause – a stack that silently never sees another update. Hence:

  • A marker file (helios/watchtower_pause.json) records only that HELIOS paused Watchtower, so a service the user stopped themselves is never brought back. unpause additionally errors on a container that is not paused, so the state is checked before it runs — otherwise a Watchtower the user recreated meanwhile would strand the marker forever.
  • Whether the pause may end is never read from that marker: it is derived from live state (is a sidecar container still running, is an upgrade or recreate still flagged pending), which a HELIOS crash cannot falsify — a crash drops the in-process flags along with the operation they belonged to.
  • BackupScheduler sweeps on every tick — the one thread reliably awake — and lifts the pause as soon as nothing is in flight. UpdatePause::MAX_AGE is the last-resort valve for when liveness cannot be determined at all.
  • A pause that fails is resolved from live state rather than assumed to have failed cleanly. The marker is written before the docker compose pause, so an error afterwards can mean either "never froze" or "froze and reported an error anyway"; the marker is dropped only in the first case, and kept when Docker cannot answer at all. A stale marker costs a single sweep, a missing one would leave Watchtower frozen for good.

While paused, Watchtower is left out of StackStatus entirely: a paused container is not running, so counting it would turn the whole stack amber for the duration of every nightly backup.

A "Start all" from the UI is not blocked and brings Watchtower back — an explicit user action outranks the pause. The marker stays consistent, the sweep simply finds the service already running.

Interrupted PostgreSQL Upgrades

The major-version upgrade (PostgresqlUpgrade) is the one long-running operation that cannot be handed to a detached sidecar: it rewrites config.yaml and compose.yaml between the Docker calls. It therefore dies with the HELIOS process — and it has a window between emptying the data directory and restoring the dump where dying means the database is gone and only the dump file holds the data.

Its own rollback covers an upgrade that fails; it does nothing for one that is killed. So every step that changes something is recorded first in a Journal (helios/postgresql_upgrade.json), and config/puma.rb hands an unfinished journal to PostgresqlUpgrade.recover! on the next boot:

Phase State on disk Recovery
preparing dump half-written, nothing else touched drop the dump, report that the upgrade did not happen
migrating image bumped, old cluster intact revert image and PGDATA, start the old major again
rebuilding data directory emptied, dump is the data rebuild the cluster from the dump; on failure roll back to the old major
finishing data migrated and verified reconcile the stack, close the journal

The pending operation is set synchronously during boot, so the first /services render cannot offer an upgrade (or a start) for a database the recovery is about to take over, and a fresh upgrade is refused while a journal exists. A failed recovery in the rebuilding phase keeps the journal: the preserved dump is worth another attempt on the next boot. Only a dump that is gone or truncated ends the automatic path — the user is then pointed at a backup.

The crash paths are covered against real Docker in spec/integration/orchestration/postgresql_upgrade_spec.rb.

Compose File Conflict Handling

When HELIOS needs to modify compose.yaml (e.g., adding a service), it may encounter user modifications.

Strategy: Detect and warn

  1. HELIOS tracks which services it manages (stored in config.yaml)
  2. Before modifying, compare current file with expected state
  3. If differences detected in managed services → show warning to user
  4. User decides: apply changes, skip, or review diff

What HELIOS tracks:

  • Services it created (e.g., dashboard, postgresql, influxdb)
  • Expected configuration for each service

What HELIOS preserves:

  • Services it doesn't manage (user-added, e.g., traefik, dozzle) are stored as "unmanaged" in Configuration#data
  • Unmanaged services are written back to compose.yaml verbatim (with ${VAR} references intact)
  • Unknown .env variables referenced by unmanaged services are preserved in a dedicated section
  • A future web-based editor will allow power-users to modify unmanaged services directly in HELIOS

Example conflict scenarios:

Scenario HELIOS behavior
User changed port of dashboard Warning: "Port was modified. Keep your change or reset?"
User added traefik service No warning, service is preserved as unmanaged
User removed redis Warning: "Required service missing. Re-add?"

Health Checks

Health checks are defined natively in Docker Compose. HELIOS reads the health status from Docker API.

Approach:

  • Each service defines its own healthcheck in compose.yaml
  • Docker reports status: healthy, unhealthy, starting
  • HELIOS queries: docker inspect --format '{{.State.Health.Status}}' <container>

Example health checks for compose.yaml:

postgresql:
  healthcheck:
    test: ['CMD-SHELL', 'pg_isready -U postgres']
    interval: 10s
    timeout: 5s
    retries: 5

redis:
  healthcheck:
    test: ['CMD', 'redis-cli', 'ping']
    interval: 10s
    timeout: 5s
    retries: 5

influxdb:
  healthcheck:
    test: ['CMD', 'influx', 'ping']
    interval: 10s
    timeout: 5s
    retries: 5

dashboard:
  healthcheck:
    test: ['CMD-SHELL', 'nc -z 127.0.0.1 3000 || exit 1']
    interval: 10s
    timeout: 5s
    retries: 3

HELIOS behavior:

  • Displays overall status: "All services healthy" or "Problem detected"
  • On problem: Shows which service is unhealthy
  • Status is pushed to the browser live via Orchestration::EventsListener, which streams Docker's GET /events endpoint and broadcasts via Turbo Streams — no per-request polling in the UI
  • Orchestration::StackStatus tightens the refresh cadence while services are in a transient :starting state, to catch fast health transitions the event stream may coalesce