fix(mapping): endpoint hardening — actuator dupes, unknown-device fallback, type-edit guard (#58)

docs/HANDOVER.md

@@ -1,13 +1,78 @@
 # indigo-matter — Build Handover
 
-**Last updated:** 2026-06-12 18:52 UTC
-**Branch:** `fix/sensor-display-props` (PR pending)
-**Version:** `2026.2.21`
-**Tests:** 634 passing (`cd indigo-matter && /Library/Frameworks/Python.framework/Versions/Current/bin/python3 -m pytest -q`)
+**Last updated:** 2026-06-12 20:38 UTC
+**Branch:** `fix/device-mapping-hardening` (PR pending)
+**Version:** `2026.2.22`
+**Tests:** 743 passing (`cd indigo-matter && /Library/Frameworks/Python.framework/Versions/Current/bin/python3 -m pytest -q`)
 **Status:** **Wi-Fi AND Thread validated with real hardware.** Tapo P110M (Wi-Fi, energy) + Aqara FP300 (Thread, presence/lux/temp/humidity — tester CliveS). Known-open: issue #56 follow-ups (button + air-quality display states), fan brightness echo (rig flake); issues #43 (bridge-child naming), #46 (fan TurnOn FanMode mapping), #21–#24 (commission hardening).
 
 ---
 
+## 2026-06-12 session (late) — endpoint-mapping hardening (issue #58)
+
+Audit prompted by #56 ("what else is in this bug class?"). Three gaps found and
+fixed; a **device-zoo contract harness** (`tests/test_device_zoo.py`) now pins
+the whole class:
+
+1. **Duplicate actuators.** Deferral chain was only OnOff→Level→Color (+
+   Fan→Thermostat). A Matter fan (OnOff mandatory alongside FanControl since
+   1.2), a thermostat/lock/valve/covering with OnOff, or a colour light
+   *without* LevelControl (zoo caught that one immediately) all produced a
+   duplicate relay/dimmer. Fix: `ENDPOINT_OWNER_CLUSTERS` in
+   `matter_handlers/base.py` (Valve 0x0081, DoorLock 0x0101, WindowCovering
+   0x0102, Thermostat 0x0201, FanControl 0x0202); OnOff and LevelControl defer
+   to any of them, and OnOff also defers directly to ColorControl.
+2. **Unknown devices vanished.** An endpoint with only unhandled clusters
+   produced nothing; a node of such endpoints commissioned to SUCCESS with
+   `indigoDeviceIds: []` (Domio shows success, Indigo shows nothing). Fix:
+   `matterUnknown` placeholder (its `supportedClusters` ConfigUI field was
+   already in Devices.xml, never wired) for device-bearing endpoints ≥1 whose
+   clusters aren't all in `_NON_DEVICE_CLUSTERS`; INFO log invites a GitHub
+   device-support report. When a later pass finds the endpoint supported
+   (firmware update — the Tapo pattern), the real device is created and an
+   INFO says the placeholder can be deleted (never auto-deleted).
+3. **Type-edit guard.** Indigo's Edit Device dialog always offers the full
+   Type menu (platform behaviour, can't be removed). `createdTypeId` is now
+   stamped into props at creation and healed onto legacy devices by the
+   reconcile self-heal; `validateDeviceConfigUi` rejects a type change with an
+   explanatory alert; `deviceStartComm` warns as backstop for edits made while
+   the plugin was off.
+
+The zoo asserts, over every cluster-combo entry: expected device types, at
+most one actuator per endpoint, spec types exist in Devices.xml, initial
+states declared, sensor specs carry explicit display props (the #56 class).
+**Add new wild devices' cluster sets to ZOO** — invariants run automatically.
+
+**Live-rig verification (same evening, jarvis on the 2026.2.22 branch build):**
+deployed via rsync + restart with all 15 virtual devices running. First
+reconcile healed every pre-fix device in one pass — 9 value sensors + smoke
+got display props, all 20 devices got `createdTypeId`, every replace passed
+the persistence verification (zero "did not persist" warnings). Second
+restart: "reconciled 16 Matter node(s)" with NO re-asserts and **NO
+stale-display warnings** → **real Indigo DOES re-derive `displayStateId` from
+`replacePluginPropsOnServer`**. Confirmed on-device: the temp sensor's
+`on_state` went `false` → `null` (built-in onOffState removed). So the #56 fix
+is fully self-healing — users (incl. CliveS) just upgrade; delete-and-recreate
+is never needed and the stale-display warning is pure dead-man insurance.
+
+**Follow-up experiment (same evening, also live on jarvis): #56 fully closed.**
+Patched the jarvis copy to set BOTH Supports* False on the button and
+air-quality handlers + temporary displayStateId instrumentation. Result:
+button `displayStateId` → **`lastButtonEvent`**, air quality → **`airQuality`**.
+The precedence rule for API-created devices: a True Supports* prop wins; with
+both explicitly False, Indigo falls back to Devices.xml `<UiDisplayStateId>`.
+Implemented for real: `display_props = {both False}` on GenericSwitchHandler
+and AirQualityHandler (the zoo invariant immediately caught that
+generic_switch didn't merge display_props into its spec props — fixed); the
+stale-display nag now covers any type disclaiming SupportsOnState; zoo
+invariant allows (False, False) only when the XML declares a UiDisplayStateId
+pointing at a declared custom state. 747 tests. Full display census of the
+16-node fleet (from the instrumentation pass): relays/locks/valves onOffState,
+dimmers/covering/fan brightnessLevel, thermostat temperatureInputsAll, value
+sensors sensorValue, button lastButtonEvent, AQ airQuality — all correct.
+
+---
+
 ## 2026-06-12 session — THREAD VALIDATED + sensor display fix (issue #56)
 
 **Thread is real-device validated.** Tester CliveS commissioned an **Aqara FP300**

indigo-matter.indigoPlugin/Contents/Info.plist

@@ -20,7 +20,7 @@
 	<key>IwsApiVersion</key>
 	<string>1.0.0</string>
 	<key>PluginVersion</key>
-	<string>2026.2.21</string>
+	<string>2026.2.22</string>
 	<key>ServerApiVersion</key>
 	<string>3.6</string>
 </dict>

indigo-matter.indigoPlugin/Contents/Server Plugin/Devices.xml

@@ -436,10 +436,11 @@
 
     <!-- AirQuality enum sensor (cluster 0x005B).
          sensorValue = raw enum int; airQuality = human-readable string.
-         UiDisplayStateId = airQuality would be the useful display, but
-         API-created devices ignore Devices.xml statics (issue #56) — they
-         show the raw enum via SupportsSensorValue. Do NOT try to fix the
-         display here; it needs a creation-prop mechanism (#56 follow-up). -->
+         UiDisplayStateId = airQuality IS the live display — it applies to
+         API-created devices only because the handler sets BOTH
+         SupportsOnState and SupportsSensorValue False in creation props
+         (the built-in display wins whenever one of them is True; verified
+         live, issue #56). -->
     <Device id="matterAirQualitySensor" type="sensor">
         <Name>Matter Air Quality Sensor</Name>
         <ConfigUI>

indigo-matter.indigoPlugin/Contents/Server Plugin/device_sync.py

@@ -34,6 +34,7 @@
     node_id_to_str,
     parse_node,
 )
+from matter_handlers.base import IndigoDeviceSpec
 from matter_handlers.electrical import CLUSTER_ELECTRICAL_ENERGY, CLUSTER_ELECTRICAL_POWER
 from matter_handlers.power_source import CLUSTER_POWER_SOURCE
 from protocol import MatterCommand
@@ -43,6 +44,27 @@
 # (on_off → matterRelay, level_control → matterDimmer, color_control → matterColorDimmer).
 _METER_CAPABLE_TYPES = frozenset({"matterRelay", "matterDimmer", "matterColorDimmer"})
 
+#: Clusters that don't, by themselves, make an endpoint a *device*: utility /
+#: metadata clusters any endpoint may carry, plus the merge-only measurement
+#: clusters (their handlers augment a sibling device, never create one). An
+#: endpoint ≥1 whose unhandled clusters are all in this set gets no
+#: matterUnknown placeholder (issue #58).
+_NON_DEVICE_CLUSTERS = frozenset({
+    0x0003,  # Identify
+    0x0004,  # Groups
+    0x0005,  # Scenes (pre-Matter-1.3 id)
+    0x001D,  # Descriptor
+    0x001E,  # Binding
+    0x0028,  # BasicInformation
+    0x002F,  # PowerSource (node-scoped battery, merged into siblings)
+    0x0039,  # BridgedDeviceBasicInformation
+    0x0040,  # FixedLabel
+    0x0041,  # UserLabel
+    0x0062,  # ScenesManagement (Matter 1.3 id)
+    0x0090,  # ElectricalPowerMeasurement (merge-only)
+    0x0091,  # ElectricalEnergyMeasurement (merge-only)
+})
+
 
 def _kvlist(states: dict) -> list:
     return [{"key": key, "value": value} for key, value in states.items()]
@@ -245,7 +267,31 @@ def create_devices(self, node: NodeInfo, suggested_room: Optional[str] = None) -
                 endpoint.has(CLUSTER_POWER_SOURCE) for endpoint in node.endpoints
             )
             for endpoint in node.endpoints:
-                for spec in self.registry.handlers_for_endpoint(node, endpoint):
+                specs = self.registry.handlers_for_endpoint(node, endpoint)
+                if not specs:
+                    # No handler claimed this endpoint. If it carries clusters
+                    # that DO make it a device (just ones we don't support yet),
+                    # surface it as a matterUnknown placeholder instead of
+                    # silently creating nothing — a commission that "succeeds"
+                    # with no visible device is a half-happened op (issue #58).
+                    # Endpoint 0 is the Matter root (utility clusters only) and
+                    # never gets a placeholder.
+                    fallback = self._unknown_spec(node, endpoint)
+                    if fallback is not None:
+                        specs = [fallback]
+                elif self._index.get(
+                    (int(node.node_id), int(endpoint.endpoint_id)), {}
+                ).get("matterUnknown"):
+                    # The endpoint used to be unsupported (placeholder exists)
+                    # but now maps to real device(s) — e.g. a firmware update
+                    # added clusters. Never auto-delete a user's device; tell
+                    # them the placeholder is obsolete instead.
+                    self.logger.info(
+                        "node %s endpoint %s is now supported — the 'Matter Device "
+                        "(unsupported clusters)' placeholder device can be deleted",
+                        node_id_to_str(node.node_id), endpoint.endpoint_id,
+                    )
+                for spec in specs:
                     if node_has_power_source:
                         spec.props.setdefault("SupportsBatteryLevel", True)
                     plan.append((endpoint, spec))
@@ -304,6 +350,15 @@ def create_devices(self, node: NodeInfo, suggested_room: Optional[str] = None) -
                 if dev_id is None:
                     failed += 1
                     continue
+                if type_id == "matterUnknown":
+                    self.logger.info(
+                        "node %s endpoint %s exposes only clusters this plugin does not "
+                        "support yet (%s) — created placeholder device %s; please report "
+                        "the device at github.com/simons-plugins/indigo-matter/issues so "
+                        "support can be added",
+                        node_id_to_str(node.node_id), endpoint.endpoint_id,
+                        spec.props.get("supportedClusters", "?"), dev_id,
+                    )
                 self._index.setdefault(ep_key, {})[type_id] = dev_id
                 created.append(dev_id)
                 new_ids.append(dev_id)
@@ -343,7 +398,38 @@ def create_devices(self, node: NodeInfo, suggested_room: Optional[str] = None) -
             result["failedEndpoints"] = failed
         return result
 
+    @staticmethod
+    def _unknown_spec(node: NodeInfo, endpoint: Any) -> Optional[IndigoDeviceSpec]:
+        """Placeholder spec for a device-bearing endpoint no handler claims.
+
+        Returns None for endpoint 0 (the Matter root — utility clusters only)
+        and for endpoints whose clusters are all in _NON_DEVICE_CLUSTERS
+        (nothing device-like to surface).
+        """
+        if int(endpoint.endpoint_id) == 0:
+            return None
+        unmapped = sorted(set(endpoint.cluster_ids) - _NON_DEVICE_CLUSTERS)
+        if not unmapped:
+            return None
+        name = node.suggested_name or node.product_name or f"Matter {node.node_id}"
+        return IndigoDeviceSpec(
+            device_type_id="matterUnknown",
+            name=name,
+            props={
+                "nodeId": str(node.node_id),
+                "endpointId": str(endpoint.endpoint_id),
+                "vendorName": node.vendor_name,
+                "productName": node.product_name,
+                "supportedClusters": ", ".join(f"0x{c:04X}" for c in unmapped),
+            },
+            initial_states={"reachable": True},
+        )
+
     def _create_one(self, spec: Any, name: str, folder_id: int = 0) -> Optional[int]:
+        # Stamp the type the cluster pipeline chose, so the type-edit guard
+        # (validateDeviceConfigUi + deviceStartComm) can detect a manual change
+        # via Indigo's Edit Device Type menu (issue #58).
+        spec.props.setdefault("createdTypeId", spec.device_type_id)
         try:
             dev = indigo.device.create(
                 protocol=indigo.kProtocol.Plugin,
@@ -574,6 +660,16 @@ def _reassert_capability_props(self, node: NodeInfo) -> None:
                         # be written explicitly, not skipped as already-falsy.
                         if key not in current_props or bool(current_props.get(key)) != bool(value):
                             missing[key] = value
+                    # Heal the type-edit guard's stamp onto devices created
+                    # before it existed (issue #58). Records the CURRENT type as
+                    # canonical — for a pristine fleet that is the created type.
+                    if type_id and "createdTypeId" not in current_props:
+                        missing["createdTypeId"] = type_id
+                    # Heal the address (UI protocol-id column, issue #18) onto
+                    # devices created before that stamping existed — it is the
+                    # node id a user needs for decommission.
+                    if not current_props.get("address"):
+                        missing["address"] = node_id_to_str(node.node_id)
                     if not missing:
                         # Props are already correct — but Indigo caches the list
                         # display state at creation and may not re-derive it from
@@ -623,20 +719,25 @@ def _reassert_capability_props(self, node: NodeInfo) -> None:
                     )
 
     def _warn_stale_display_state(self, dev: Any, type_id: str, display_props: dict) -> None:
-        """Warn when a value sensor's cached display state is still on/off.
+        """Warn when a device's cached display state is wrongly stuck on on/off.
 
         Indigo derives ``displayStateId`` from the Supports* props at device
-        creation; replacing props afterwards rebuilds states but the display
-        may stay cached on ``onOffState`` (issue #56). The fix is user-side
-        and cheap — deleting the Indigo device and reloading the plugin lets
-        reconcile recreate it with the correct creation props — so name the
-        device and say exactly that. Deliberately checked only on passes where
-        no props needed replacing: right after a replace, Indigo may not have
-        re-derived yet, and warning there would be noise (a replace that never
-        converges is surfaced separately by the did-not-persist warning).
+        creation; replacing props afterwards rebuilds states and (verified
+        live, 2026-06-12) normally re-derives the display too — this warning
+        is dead-man insurance for the case where it doesn't. Applies to any
+        type whose display_props disclaim the on/off display: value sensors
+        (SupportsSensorValue) and UiDisplayStateId-fallback types like the
+        button and air-quality sensor (both Supports* False). The remedy is
+        user-side and cheap — deleting the Indigo device and reloading the
+        plugin lets reconcile recreate it with the correct creation props —
+        so name the device and say exactly that. Deliberately checked only on
+        passes where no props needed replacing: right after a replace, Indigo
+        may not have re-derived yet, and warning there would be noise (a
+        replace that never converges is surfaced separately by the
+        did-not-persist warning).
         """
-        if not display_props.get("SupportsSensorValue"):
-            return  # on/off IS the right display for binary sensors
+        if display_props.get("SupportsOnState", True):
+            return  # on/off IS the right display (binary sensors, non-sensor types)
         display_state = getattr(dev, "displayStateId", None)
         if display_state is None:
             # Unreadable is "unknown", not "healthy" — keep a trace, but a real

indigo-matter.indigoPlugin/Contents/Server Plugin/matter_handlers/air_quality.py

@@ -45,15 +45,20 @@ class AirQualityHandler(_SensorHandler):
 
     Attribute 0x0000 is an enum; we expose both the raw integer (sensorValue)
     and the human-readable string (airQuality) so automations can trigger on
-    the string and display it directly.  Devices.xml declares UiDisplayStateId
-    = airQuality, but API-created devices ignore it (issue #56) — the list
-    display is the raw enum via the inherited SupportsSensorValue prop; the
-    string display is a #56 follow-up pending hardware testing.
+    the string and display it directly.  The list display IS the string:
+    with both Supports* props False the Devices.xml <UiDisplayStateId>
+    applies even to API-created devices (verified live — issue #56).
     """
 
     cluster_id = 0x005B
     cluster_name = "AirQuality"
     device_type_id = "matterAirQualitySensor"
+    # Override the value-sensor pair: with BOTH Supports* False, Indigo falls
+    # back to <UiDisplayStateId>airQuality</UiDisplayStateId>, so the device
+    # list shows "good"/"poor" instead of the raw enum int. Verified live on
+    # jarvis (issue #56 follow-up). sensorValue keeps working — it is an
+    # XML-declared custom state, not the disabled built-in.
+    display_props = {"SupportsOnState": False, "SupportsSensorValue": False}
 
     measured_attr = ATTR_AIR_QUALITY
 

indigo-matter.indigoPlugin/Contents/Server Plugin/matter_handlers/base.py

@@ -17,11 +17,28 @@
 
 from protocol import MatterCommand, MatterWrite  # re-exported for handlers/tests
 
-__all__ = ["IndigoDeviceSpec", "MatterCommand", "MatterWrite", "ClusterHandler", "MatterAction"]
+__all__ = [
+    "IndigoDeviceSpec", "MatterCommand", "MatterWrite", "ClusterHandler",
+    "MatterAction", "ENDPOINT_OWNER_CLUSTERS",
+]
 
 #: A handler action is either a cluster-command invoke or an attribute write.
 MatterAction = Union[MatterCommand, MatterWrite]
 
+#: Clusters whose handler owns the whole endpoint. When one of these is present,
+#: OnOff/LevelControl on the same endpoint are subordinate controls of that
+#: device (a fan's power switch, a thermostat's enable) and must NOT produce a
+#: standalone relay/dimmer — that would duplicate the device in Indigo. The
+#: Matter spec makes such combinations normal: e.g. the Fan device type
+#: mandates OnOff alongside FanControl since Matter 1.2 (issue #58).
+ENDPOINT_OWNER_CLUSTERS = frozenset({
+    0x0081,  # ValveConfigurationAndControl
+    0x0101,  # DoorLock
+    0x0102,  # WindowCovering
+    0x0201,  # Thermostat
+    0x0202,  # FanControl
+})
+
 
 @dataclass
 class IndigoDeviceSpec:

indigo-matter.indigoPlugin/Contents/Server Plugin/matter_handlers/generic_switch.py

@@ -66,6 +66,11 @@ class GenericSwitchHandler(ClusterHandler):
     cluster_id     = CLUSTER_SWITCH
     cluster_name   = "GenericSwitch"
     device_type_id = "matterButton"
+    # With BOTH Supports* False, Indigo falls back to the Devices.xml
+    # <UiDisplayStateId> (lastButtonEvent) even for API-created devices —
+    # the props-driven built-in display only takes precedence when one of
+    # these is True. Verified live on jarvis (issue #56 follow-up).
+    display_props = {"SupportsOnState": False, "SupportsSensorValue": False}
 
     def create_indigo_devices(self, node: Any, endpoint: Any) -> list[IndigoDeviceSpec]:
         name = node.suggested_name or node.product_name or f"Matter {node.node_id}"
@@ -78,6 +83,7 @@ def create_indigo_devices(self, node: Any, endpoint: Any) -> list[IndigoDeviceSp
                     "endpointId":  str(endpoint.endpoint_id),
                     "vendorName":  node.vendor_name,
                     "productName": node.product_name,
+                    **self.display_props,
                 },
                 initial_states={
                     "lastButtonEvent": "",