Add beginner/expert experience and remote-compute onboarding preferences

docs/API.md

@@ -282,7 +282,7 @@ Parsing and writing live on the backend: the frontend exchanges structured `Auto
 | `config/version` | — | `{server_version, esphome_version}` | Get versions |
 | `config/serial_ports` | — | `[{port, desc}]` | List serial ports |
 | `config/get_preferences` | — | `UserPreferences` | Get user preferences |
-| `config/set_preferences` | `{theme?, dashboard_view?, ...}` | `UserPreferences` | Update preferences (partial) |
+| `config/set_preferences` | `{theme?, dashboard_view?, experience_level?, remote_compute_only?, ...}` | `UserPreferences` | Update preferences (partial). `experience_level` is `beginner` / `expert` (or `null` until chosen); `remote_compute_only` marks an install as a remote build node. |
 | `config/get_secrets` | — | `[string]` | List secret key names |
 | `config/set_secret` | `{key, value, overwrite?}` | `{created}` | Atomically set one secret in secrets.yaml under a write lock; `overwrite=false` is create-if-absent |
 
@@ -292,11 +292,11 @@ Parsing and writing live on the backend: the frontend exchanges structured `Auto
 >
 > Models: [`OnboardingState`, `OnboardingStep`, `OnboardingStepId`, `OnboardingStepStatus`](../esphome_device_builder/models/onboarding.py)
 
-First-run setup tracking. Each step's `status` is computed from live data on every `get_state` call (never persisted), so the frontend's "needs attention" indicators clear the moment the user fixes the underlying state — even via a manual `secrets.yaml` edit. `completed_version` is the last onboarding-flow version the user has explicitly acknowledged; bumping `ONBOARDING_VERSION` (server-side constant) re-prompts users at lower versions when new steps are added.
+First-run setup tracking. Each step's `status` is computed from live data on every `get_state` call (never persisted), so the frontend's "needs attention" indicators clear the moment the user fixes the underlying state — even via a manual `secrets.yaml` edit. `completed_version` is the last onboarding-flow version the user has explicitly acknowledged; bumping `ONBOARDING_VERSION` (server-side constant) re-prompts users at lower versions when new steps are added. The step list is environment- and preference-aware: `use_case` only on non-HA installs, `wifi_credentials` only when not `remote_compute_only`. A pre-existing install (prior onboarding completed, or device YAML already on disk) is migrated to the `expert` experience at startup so the wizard never auto-pops for it.
 
 | Command | Args | Response | Description |
 |---------|------|----------|-------------|
-| `onboarding/get_state` | — | `OnboardingState` | Snapshot of current vs acknowledged version + per-step `pending` / `done` status. Currently one step (`wifi_credentials`) — pending when `secrets.yaml`'s `wifi_ssid` is missing, empty, whitespace-only, or matches the bootstrap placeholder. |
+| `onboarding/get_state` | — | `OnboardingState` | Snapshot of current vs acknowledged version + per-step `pending` / `done` status. Steps: `use_case` (non-HA only — remote-compute choice) and `experience_level` track whether `experience_level` is set; `wifi_credentials` (omitted when `remote_compute_only`) is pending when `secrets.yaml`'s `wifi_ssid` is missing, empty, whitespace-only, or matches the bootstrap placeholder. |
 | `onboarding/set_wifi_credentials` | `{ssid, password?}` | `OnboardingState` | Update `wifi_ssid` / `wifi_password` in `secrets.yaml` via a line-based rewrite that preserves standalone and inline trailing comments and other secrets. Validates against ESPHome's own length limits (32 char SSID, 64 char password) plus a control-character check; empty / whitespace-only SSID, oversize values, and control characters (other than `\t`) raise `INVALID_ARGS`. `password` is optional and defaults to the empty string for open networks. |
 | `onboarding/mark_acknowledged` | — | `OnboardingState` | Record that the user has finished the current onboarding flow (sets `onboarding_completed_version` to `ONBOARDING_VERSION`). Idempotent and monotonic — never downgrades a higher stored value. Use this on save AND on explicit decline ("I don't use Wi-Fi") so the wizard stops re-popping; the per-step `pending` status stays accurate so the dedicated `Set up Wi-Fi…` kebab entry still surfaces the re-entry path until the underlying data is set. |
 
@@ -428,7 +428,7 @@ Same-subnet peers read `remote_build_port` from TXT so a `--remote-build-port` o
 
 **`subscribe_events` initial state:**
 
-Right after a client subscribes (and before any live events arrive), the server pushes one `initial_state` event carrying a snapshot of state that's accumulated server-side via background activity (mDNS browser, completed pair flows, etc.) so the frontend can paint the first frame without follow-up reads. Shape: `{preferences: UserPreferences, devices?: [...], importable?: [...], pairings?: [PairingSummary], peers?: [PeerSummary], hosts?: [RemoteBuildPeer], offloader_alerts?: [OffloaderAlertSnapshotEntry], peer_queue_status?: [PeerQueueStatusSnapshotEntry], remote_jobs?: [OffloaderRemoteJobSnapshotEntry], remote_builds_enabled?: bool, version_match_policy?: "any" | "release" | "exact" | "exact_required"}`. `preferences` is always present — it's RAM-canonical behind a `PreferencesStore` (loaded once at startup, mutations debounce a write to its own `.device-builder-preferences.json`, migrated out of the shared sidecar on first run; the same per-file `Store` pattern as the device-metadata and peer-link stores), so the client paints theme/UI state without a separate `config/get_preferences`. Undecodable preferences are preserved, never destroyed: a corrupt dedicated file is renamed to `.corrupt` and an undecodable legacy sidecar blob is left in place (not stripped), both logged, before the store falls back to defaults. The rest are present only when the corresponding controller is up; `pairings` carries both PENDING and APPROVED offloader-side rows from the `_pairings` dict, `peers` carries both PENDING (`_pending_peers`) and APPROVED (`_approved_peers`) receiver-side rows, `hosts` carries the receiver controller's mDNS-discovered peer dashboards (`self._peers`, RAM-only — never persisted), `offloader_alerts` carries the offloader-side pair alerts dict (`_offloader_alerts`, RAM-only) so a tab subscribing AFTER a `pin_mismatch` / `peer_revoked` event fired still renders the alert it would have missed on the live stream — the alert only clears via re-pair or unpair, never by an operator-driven dismiss, because the underlying state (broken pairing) doesn't fix itself. `peer_queue_status` carries the most recent `queue_status` snapshot per paired receiver so a late tab paints the per-peer queue depth without waiting for the next event. `remote_jobs` carries every offloader-submitted job that's still in flight (terminal entries drop on the matching `job_state_changed` event) so the UI can render running builds on page load. `remote_builds_enabled` and `version_match_policy` carry the current value of the offloader-wide settings so the Settings dialog renders both controls on first paint instead of waiting for the matching `offloader_remote_builds_toggled` / `offloader_version_match_policy_changed` event to fire. All sync reads, no executor hop, no disk I/O. The `PeerSummary` projection persists `peer_ip` (the source IP observed at pair_request time) on `StoredPeer` so a snapshot-loaded inbox row carries the same IP the live `remote_build_pair_request_received` event would carry; that's what the receiver Settings UI renders alongside the pin as a clone-risk sanity-check. Empty string for legacy on-disk rows from receivers that pre-date the field. Live updates that arrive after the initial state mutate against this seed via the events below.
+Right after a client subscribes (and before any live events arrive), the server pushes one `initial_state` event carrying a snapshot of state that's accumulated server-side via background activity (mDNS browser, completed pair flows, etc.) so the frontend can paint the first frame without follow-up reads. Shape: `{preferences: UserPreferences, devices?: [...], importable?: [...], pairings?: [PairingSummary], peers?: [PeerSummary], hosts?: [RemoteBuildPeer], offloader_alerts?: [OffloaderAlertSnapshotEntry], peer_queue_status?: [PeerQueueStatusSnapshotEntry], remote_jobs?: [OffloaderRemoteJobSnapshotEntry], remote_builds_enabled?: bool, version_match_policy?: "any" | "release" | "exact" | "exact_required"}`. `preferences` is always present — its `experience_level` and `remote_compute_only` fields gate first-paint UI (which editor surfaces show, whether device-creation entry points are hidden), so they ride the snapshot rather than make the client chase a separate `config/get_preferences`. The rest are present only when the corresponding controller is up; `pairings` carries both PENDING and APPROVED offloader-side rows from the `_pairings` dict, `peers` carries both PENDING (`_pending_peers`) and APPROVED (`_approved_peers`) receiver-side rows, `hosts` carries the receiver controller's mDNS-discovered peer dashboards (`self._peers`, RAM-only — never persisted), `offloader_alerts` carries the offloader-side pair alerts dict (`_offloader_alerts`, RAM-only) so a tab subscribing AFTER a `pin_mismatch` / `peer_revoked` event fired still renders the alert it would have missed on the live stream — the alert only clears via re-pair or unpair, never by an operator-driven dismiss, because the underlying state (broken pairing) doesn't fix itself. `peer_queue_status` carries the most recent `queue_status` snapshot per paired receiver so a late tab paints the per-peer queue depth without waiting for the next event. `remote_jobs` carries every offloader-submitted job that's still in flight (terminal entries drop on the matching `job_state_changed` event) so the UI can render running builds on page load. `remote_builds_enabled` and `version_match_policy` carry the current value of the offloader-wide settings so the Settings dialog renders both controls on first paint instead of waiting for the matching `offloader_remote_builds_toggled` / `offloader_version_match_policy_changed` event to fire. All sync RAM reads, no executor hop, no disk I/O — `preferences` is RAM-canonical behind a `PreferencesStore` (loaded once at startup, mutations debounce a write to its own `.device-builder-preferences.json`, migrated out of the shared sidecar on first run; the same per-file `Store` pattern as the device-metadata and peer-link stores). Undecodable preferences are preserved, never destroyed: a corrupt dedicated file is renamed to `.corrupt` and an undecodable legacy sidecar blob is left in place (not stripped), both logged, before the store falls back to defaults. The `PeerSummary` projection persists `peer_ip` (the source IP observed at pair_request time) on `StoredPeer` so a snapshot-loaded inbox row carries the same IP the live `remote_build_pair_request_received` event would carry; that's what the receiver Settings UI renders alongside the pin as a clone-risk sanity-check. Empty string for legacy on-disk rows from receivers that pre-date the field. Live updates that arrive after the initial state mutate against this seed via the events below.
 
 **`subscribe_events` events:**
 - `device_added`, `device_removed`, `device_updated`, `device_state_changed`

esphome_device_builder/controllers/onboarding.py

@@ -20,8 +20,12 @@
 from __future__ import annotations
 
 import asyncio
+import logging
+from pathlib import Path
 from typing import TYPE_CHECKING, Any
 
+from esphome.util import list_yaml_files
+
 from ..helpers.api import CommandError, api_command
 from ..helpers.secrets_state import (
     is_wifi_unconfigured,
@@ -30,18 +34,22 @@
 )
 from ..models import (
     ErrorCode,
+    ExperienceLevel,
     OnboardingState,
     OnboardingStep,
     OnboardingStepId,
     OnboardingStepStatus,
     UserPreferences,
 )
 from ..models.onboarding import ONBOARDING_VERSION
+from .config.settings import _DASHBOARD_SENTINEL_FILE
 
 if TYPE_CHECKING:
     from esphome_device_builder.controllers.config._preferences_store import PreferencesStore
     from esphome_device_builder.device_builder import DeviceBuilder
 
+_LOGGER = logging.getLogger(__name__)
+
 
 # Cap inputs at the same length ESPHome's own validators enforce —
 # ``cv.ssid`` (32 chars) and the WPA password validator (64 chars).
@@ -70,27 +78,38 @@ async def get_state(self, **kwargs: Any) -> OnboardingState:
         """
         Return the current onboarding snapshot.
 
-        Computes each step's status from live data, then reads the
-        user's last-acknowledged version from preferences. The
-        frontend combines the two to decide whether to surface the
-        wizard (any pending step OR new version available).
+        The step list is environment- and preference-aware: the
+        use-case step only on non-HA installs, the Wi-Fi step only
+        when not remote-compute-only. Each status is computed from
+        live data; the frontend surfaces the wizard on any pending
+        step or a newer version.
         """
-        loop = asyncio.get_running_loop()
-        secrets = await loop.run_in_executor(None, read_secrets_yaml, self._db.settings.config_dir)
+        settings = self._db.settings
         prefs = self._prefs.snapshot()
+        loop = asyncio.get_running_loop()
+        secrets = await loop.run_in_executor(None, read_secrets_yaml, settings.config_dir)
+        return _compute_state(secrets, prefs, on_ha_addon=settings.on_ha_addon)
 
-        return OnboardingState(
-            current_version=ONBOARDING_VERSION,
-            completed_version=prefs.onboarding_completed_version,
-            steps=[
-                OnboardingStep(
-                    id=OnboardingStepId.WIFI_CREDENTIALS,
-                    status=OnboardingStepStatus.PENDING
-                    if is_wifi_unconfigured(secrets)
-                    else OnboardingStepStatus.DONE,
-                ),
-            ],
-        )
+    async def migrate_preexisting_install(self) -> None:
+        """
+        Default a pre-existing install to the EXPERT experience, once.
+
+        Installs that completed an earlier onboarding or already hold
+        device YAMLs predate the experience picker; mark them EXPERT
+        users and acknowledge onboarding so the wizard never auto-pops.
+        Idempotent — a no-op once ``experience_level`` is set.
+        """
+        prefs = self._prefs.snapshot()
+        if prefs.experience_level is not None:
+            return
+        has_configs = False
+        if prefs.onboarding_completed_version == 0:
+            loop = asyncio.get_running_loop()
+            has_configs = await loop.run_in_executor(
+                None, _has_device_configs, self._db.settings.config_dir
+            )
+        if _should_migrate_preexisting(prefs, has_device_configs=has_configs):
+            self._prefs.mutate(_mark_preexisting)
 
     @api_command("onboarding/set_wifi_credentials")
     async def set_wifi_credentials(
@@ -167,13 +186,100 @@ async def mark_acknowledged(self, **kwargs: Any) -> OnboardingState:
         releases that add new steps bump that constant; existing
         users with a lower stored value will be re-prompted.
         """
+        self._prefs.mutate(_acknowledge_current_version)
+        return await self.get_state()
+
 
-        def _bump(prefs: UserPreferences) -> None:
-            # max(), not assign: a rollback from a future build must
-            # not downgrade a higher stored acknowledgement.
-            prefs.onboarding_completed_version = max(
-                prefs.onboarding_completed_version, ONBOARDING_VERSION
+def _compute_state(
+    secrets: dict | None, prefs: UserPreferences, *, on_ha_addon: bool
+) -> OnboardingState:
+    """
+    Assemble the environment- and preference-aware onboarding step list.
+
+    *secrets* and *prefs* are read by the caller (secrets off the loop, prefs
+    from the RAM-canonical store) so this stays pure.
+    """
+    experience_done = _status(done=prefs.experience_level is not None)
+
+    steps: list[OnboardingStep] = []
+    # Use-case (remote-compute?) is a non-HA question; HA users manage
+    # devices in Home Assistant. Its status tracks the experience pick,
+    # which the wizard always answers in the same pass.
+    if not on_ha_addon:
+        steps.append(OnboardingStep(id=OnboardingStepId.USE_CASE, status=experience_done))
+    steps.append(OnboardingStep(id=OnboardingStepId.EXPERIENCE_LEVEL, status=experience_done))
+    if not prefs.remote_compute_only:
+        steps.append(
+            OnboardingStep(
+                id=OnboardingStepId.WIFI_CREDENTIALS,
+                status=_status(done=not is_wifi_unconfigured(secrets)),
             )
+        )
 
-        self._prefs.mutate(_bump)
-        return await self.get_state()
+    return OnboardingState(
+        current_version=ONBOARDING_VERSION,
+        completed_version=prefs.onboarding_completed_version,
+        steps=steps,
+    )
+
+
+def _should_migrate_preexisting(prefs: UserPreferences, *, has_device_configs: bool) -> bool:
+    """Whether a pre-existing install should default to the EXPERT experience.
+
+    No-op once an experience is chosen; a fresh install with no device YAMLs and
+    no prior onboarding is left alone so the wizard still runs.
+    """
+    if prefs.experience_level is not None:
+        return False
+    return prefs.onboarding_completed_version > 0 or has_device_configs
+
+
+def _mark_preexisting(p: UserPreferences) -> None:
+    """Mark *p* an EXPERT user; acknowledge only if onboarding was already done."""
+    p.experience_level = ExperienceLevel.EXPERT
+    # Only acknowledge onboarding for installs that already completed it, so a
+    # prior Wi-Fi save or decline is respected. An install known only by its
+    # device YAML stays un-acknowledged, so a missing-Wi-Fi prompt still surfaces.
+    if p.onboarding_completed_version > 0:
+        _acknowledge_current_version(p)
+
+
+def _acknowledge_current_version(prefs: UserPreferences) -> None:
+    """
+    Raise the acknowledged onboarding version to current, never downgrading.
+
+    max(), not assign: a rollback from a future build must not downgrade a
+    higher stored acknowledgement.
+    """
+    prefs.onboarding_completed_version = max(prefs.onboarding_completed_version, ONBOARDING_VERSION)
+
+
+def _has_device_configs(config_dir: Path) -> bool:
+    """
+    Return True when the config dir holds any user device YAML.
+
+    Uses the canonical ``list_yaml_files`` rule (.yaml + .yml, secrets
+    and dotfiles excluded) so it can't drift from the device scanner;
+    only the dashboard sentinel needs excluding on top.
+
+    A missing dir is a genuinely fresh install (return False). A dir that
+    exists but can't be read fails *safe for existing users*: assume it
+    holds configs so a transient read error can't reclassify a real
+    install as fresh and re-pop the wizard.
+    """
+    try:
+        return any(p.name != _DASHBOARD_SENTINEL_FILE for p in list_yaml_files([config_dir]))
+    except FileNotFoundError:
+        return False
+    except OSError:
+        _LOGGER.warning(
+            "Could not scan %s for device configs; assuming pre-existing install",
+            config_dir,
+            exc_info=True,
+        )
+        return True
+
+
+def _status(*, done: bool) -> OnboardingStepStatus:
+    """Map a done-ness boolean to the step status enum."""
+    return OnboardingStepStatus.DONE if done else OnboardingStepStatus.PENDING

esphome_device_builder/device_builder.py

@@ -339,6 +339,9 @@ async def start(self) -> None:
         # Seed the RAM-canonical preferences (and migrate them out of the shared
         # sidecar on first run) before onboarding reads or mutates them.
         await self.config.async_load()
+        # Default pre-existing installs to the YAML experience before
+        # any onboarding command can be served.
+        await self.onboarding.migrate_preexisting_install()
         await self.devices.start()
         await self.firmware.start()
         await self.editor.start()