Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions docs/impulse/docs/config/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -352,6 +352,40 @@ mode-resolution rules and what counts as a definition change.

---

## calculated_channels (optional)

Controls the optional `calculated_channel_metrics` output. By default a report
writes calculated channels to `calculated_channel_fact` (the derived signal) and
`calculated_channel_dimension` (the definitions). Setting `emit_channel_metrics`
adds a third table, `calculated_channel_metrics`, shaped like the silver
`channel_metrics` table so the fact + metrics pair can serve as an Impulse silver
source. See the [Channels reference](../references/report/channel.md) for the
output schema.

| Field | Type | Default | Description |
|-----------------------|-------------|--------------------------------------|---------------------------------------------------------------------------------------------------|
| `emit_channel_metrics`| `bool` | `false` | Turns on the `calculated_channel_metrics` table. |
| `attribute_columns` | `list[str]` | `[]` | Calculated-channel `attributes` keys to surface as columns on the metrics table (e.g. `["unit"]`). |
| `kpis` | `list[str]` | `["duration", "min", "max", "mean"]` | KPIs computed per `(container_id, channel_id)`, one column each. Must be registered KPI names. |

When enabled, each row of `calculated_channel_metrics` is one
`(container_id, channel_id)` pair, carrying the selected `kpis` plus dynamic
identity columns (the union of `identity` keys across the report's channels) and
the configured `attribute_columns`. A channel that omits an identity or attribute
key gets `null` for that column; an identity key wins over an attribute key of the
same name.

The available `kpis` are `duration`, `min`, `max`, and `mean` (all
duration-weighted, matching the silver ingestion semantics). An unknown KPI name is
rejected at config validation with a `ValueError` naming the valid KPIs.

:::note Off by default
When `emit_channel_metrics` is `false` (the default), no metrics table is written
and `attribute_columns` / `kpis` have no effect.
:::

---

## measurement_dimensions (optional)

List of `container_metrics` column names to surface into the gold-layer
Expand Down
15 changes: 14 additions & 1 deletion docs/impulse/docs/data_model/gold_layer_event_normalized.md
Original file line number Diff line number Diff line change
Expand Up @@ -186,6 +186,18 @@ calculated_channel_fact {
timestamp _created_at
}

calculated_channel_metrics {
int container_id
long channel_id
string type
string data_type
double duration
double min
double max
double mean
timestamp _created_at
}

histogram_fact }o--|| event_dimension: event_id
histogram2d_fact }o--|| event_dimension: event_id
stats_aggregator_fact }o--|| event_instance_fact: event_instance_id
Expand Down Expand Up @@ -221,7 +233,8 @@ guaranteed.
| `{prefix}_histogram2d_fact` | `container_id`, `visual_id`, `event_id`, `x_bin_id`, `y_bin_id` | 2D histogram bin values per container. |
| `{prefix}_stats_aggregator_fact` | `container_id`, `visual_id`, `event_instance_id`, `channel_name`, `aggregation_label` | Statistics values per signal, event instance, and container. |
| `{prefix}_event_instance_fact` | `container_id`, `event_id`, `event_instance_id` | Materialized event occurrences with start/end timestamps. |
| `{prefix}_calculated_channel_fact` | `container_id`, `channel_id`, `tstart` | Materialized derived signal — one row per sample interval, in the silver `channels` shape (`tstart`, `tend`, `value`). The channel's identity lives on `calculated_channel_dimension`, joined via `channel_id`. |
| `{prefix}_calculated_channel_fact` | `container_id`, `channel_id`, `tstart` | Materialized derived signal, one row per sample interval, in the silver `channels` shape (`tstart`, `tend`, `value`). The channel's identity lives on `calculated_channel_dimension`, joined via `channel_id`. |
| `{prefix}_calculated_channel_metrics` | `container_id`, `channel_id` | Optional per-channel metrics in the silver `channel_metrics` shape, so the fact + metrics pair can serve as an Impulse silver source. Written only when [`config.calculated_channels.emit_channel_metrics`](../config/configuration.md#calculated_channels-optional) is set. Carries the configured `kpis` plus dynamic identity/attribute columns. See [Channels](../references/report/channel.md). |

---

Expand Down
1 change: 1 addition & 0 deletions docs/impulse/docs/data_model/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@ The Gold layer uses a **star schema** with fact and dimension tables. All table
| `histogram2d_fact` | One row per (x, y) bin per container | 2D histogram bin values, duration-weighted. |
| `stats_aggregator_fact` | One row per statistic label per signal per event instance | Descriptive statistics (built-in min/max/mean/median and any custom statistics). |
| `calculated_channel_fact` | One row per sample interval per container | Materialized derived signal (a *channel*, not a summary), in the silver `channels` shape. |
| `calculated_channel_metrics` | One row per calculated channel per container | Optional per-channel metrics in the silver `channel_metrics` shape. Written only when `config.calculated_channels.emit_channel_metrics` is set. |

### Dimension tables

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,4 +20,6 @@ Enumeration of supported statistic types for aggregations.
- `MAX` (`str`): Maximum value statistic.
- `MEAN` (`str`): Mean (average) value statistic.
- `MEDIAN` (`str`): Median value statistic.
- `START` (`str`): First value in the interval.
- `END` (`str`): Last value in the interval.

Original file line number Diff line number Diff line change
Expand Up @@ -190,14 +190,16 @@ Only includes computation-affecting attributes:
- input_expressions
- statistics to be calculated
- event expression if there is any
- custom statistics (name, kind, declared input indices, and function
bytecode, so implementation or input-wiring changes invalidate cached
results; only appended when custom statistics are configured so
aggregators without them keep their previous hash)

Excludes: name, desc, signal_name, units, page_number, report_id, and the
cross-channel descriptors' channel_name (presentation metadata, like
channel_names).
- channel_names, and each cross-channel descriptor's channel_name. These
are the fact table's ``channel_name`` merge key, so a rename must force
a recompute (a changed definition recomputes and prunes all containers);
otherwise, in incremental mode, already-processed containers would keep
rows under the old name.
- custom statistics (labels, kind, declared input indices, params, and
function bytecode, so implementation or input-wiring changes invalidate
cached results; only appended when custom statistics are configured)

Excludes: name, desc, units, page_number, report_id.

**Returns**:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -178,3 +178,50 @@ Create the dimension DataFrame for the given channels.
which ``createDataFrame`` builds directly from the plain dict returned by


#### determine\_channel\_metrics

```python
def determine_channel_metrics(
cls,
spark: SparkSession,
channels: list[CalculatedChannel],
fact_df: DataFrame | None,
*,
attribute_columns: list[str] | None = None,
kpis: list[str] | None = None) -> DataFrame | None
```

Derive a silver-shaped ``channel_metrics`` DataFrame from the fact rows.

The calculated-channel fact table already matches the silver ``channels``
table; this builds its companion ``channel_metrics`` so the pair can serve
as an Impulse silver source. Metrics are aggregated **directly from the
narrow fact rows** (``container_id, channel_id, tstart, tend, value``),
grouped by ``(container_id, channel_id)``.

The output schema is **dynamic**: fixed columns ``container_id,
channel_id, type, data_type`` plus one column per configured KPI (see
``kpis``), one per identity key (the union across all ``channels``), and one
per configured attribute key. Identity/attribute values are pulled from
each channel's in-memory ``identity`` / ``attributes`` dicts (null where a
channel omits a key). On an identity/attribute key collision, identity wins
and the attribute is skipped.

**Arguments**:

- `spark` (`SparkSession`): Session used to build the per-channel metadata frame.
- `channels` (`list of CalculatedChannel`): The channels whose fact rows are in ``fact_df``; supply identity and
attributes.
- `fact_df` (`DataFrame or None`): Narrow fact DataFrame (output of :meth:`determine_calculated_channels`).
``None`` returns ``None``.
- `attribute_columns` (`list of str`): Attribute keys to surface as columns. Default/empty → no attribute
columns. A key no channel defines yields an all-null column.
- `kpis` (`list of str`): KPI names to compute (see ``calculated_channel_kpis.KPI_BUILDERS``); the
output carries one column per name, in order. ``None`` → the default
KPIs (``duration, min, max, mean``).

**Returns**:

`DataFrame or None`: The dynamic-schema metrics DataFrame, or ``None`` when ``fact_df`` is
``None``.

Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,28 @@ Get the dimension table name for the channel type.

`str`: The name of the dimension table associated with this channel type.

#### get\_metrics\_table\_name

```python
def get_metrics_table_name() -> str
```

Get the (optional) channel-metrics table name for the channel type.

This table mirrors the silver-layer ``channel_metrics`` table so the
calculated-channel fact + metrics pair can serve as an Impulse silver
source. Unlike the fact/dimension tables it has **no** fixed schema
constant: identity/attribute columns are derived dynamically per report
(see :meth:`CalculatedChannel.determine_channel_metrics`).

**Raises**:

- `ValueError`: If the channel type is not supported.

**Returns**:

`str`: The name of the channel-metrics table associated with this channel type.

#### get\_dimension\_schema

```python
Expand Down Expand Up @@ -103,6 +125,26 @@ Return the first ChannelType whose fact table name matches.

`ChannelType`:

#### get\_any\_for\_metrics\_table

```python
def get_any_for_metrics_table(cls, table_name: str) -> "ChannelType"
```

Return the first ChannelType whose metrics table name matches.

**Arguments**:

- `table_name` (`str`): Metrics table name to look up.

**Raises**:

- `ValueError`: If no ChannelType matches the given table name.

**Returns**:

`ChannelType`:

#### get\_any\_for\_dimension\_table

```python
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -237,6 +237,27 @@ Configuration for incremental processing behavior.
- `silver_last_modified_column` (`str, default="timestamp"`): Column name in the silver layer used for freshness comparison.
- `gold_last_modified_column` (`str, default="last_modified"`): Column name in the gold layer used for freshness comparison.

## CalculatedChannels

```python
class CalculatedChannels(BaseModel)
```

Configuration for calculated-channel outputs.

**Arguments**:

- `emit_channel_metrics` (`bool, default=False`): When True, also emit a ``calculated_channel_metrics`` gold table (silver
``channel_metrics`` shape) alongside the calculated-channel fact table, so
the fact + metrics pair can serve as an Impulse silver source.
- `attribute_columns` (`list of str, default=[]`): Calculated-channel attribute keys to surface as columns on the metrics
table (e.g. ``["unit"]``). Empty (the default) → no attribute columns.
Identity keys are always surfaced dynamically and win over an
attribute key of the same name.
- `kpis` (`list of str, default=["duration", "min", "max", "mean"]`): KPIs computed on the metrics table, one column per name. Each must be a
registered KPI (see ``calculated_channel_kpis.KPI_BUILDERS``); an unknown
name is rejected at validation. Duplicates are removed (order preserved).

## ImpulseConfig

```python
Expand All @@ -257,6 +278,9 @@ Attributes
Optional query engine configuration. Defaults to Solvers.DEFAULT_SOLVER.
incremental : IncrementalConfig, optional
Optional incremental processing configuration. Defaults to IncrementalConfig().
calculated_channels : CalculatedChannels, optional
Optional calculated-channel output configuration (e.g. opting in to the
``calculated_channel_metrics`` table). Defaults to CalculatedChannels().
measurement_dimensions : list of str, optional
Column names to surface from ``container_metrics`` into the
gold-layer ``measurement_dimension`` table. Names are matched
Expand Down
30 changes: 30 additions & 0 deletions docs/impulse/docs/references/report/channel.md
Original file line number Diff line number Diff line change
Expand Up @@ -138,3 +138,33 @@ run-length-encoded shape as the silver `channels` table.
Both tables carry the configurable `table_prefix` (e.g. `{prefix}_calculated_channel_fact`). See the
[gold layer schema](../../data_model/gold_layer_event_normalized.md) for how they fit the star schema, and
[incremental processing](./index.md#incremental-processing) for how definition changes are reprocessed.

---

## Optional channel metrics table

Because `calculated_channel_fact` already matches the silver `channels` shape, a calculated channel needs
only a companion `channel_metrics` table to serve as an Impulse silver source in its own right. Set
[`config.calculated_channels.emit_channel_metrics`](../../config/configuration.md#calculated_channels-optional)
to also write a `calculated_channel_metrics` table, shaped like the silver `channel_metrics` table.

### calculated_channel_metrics

One row per `(container_id, channel_id)`, derived directly from the fact rows. The schema is **dynamic**:
fixed columns plus one column per configured KPI, one per identity key (the union of `identity` keys across
the report's channels), and one per configured attribute key.

| Column | Type | Description |
|----------------|----------|-------------------------------------------------------------------------------------------------|
| `container_id` | `int` | Container identifier. Type is inherited from the silver source. |
| `channel_id` | `long` | Calculated-channel identifier (matches the fact and dimension). |
| *identity cols*| `str` | One column per identity key (e.g. `channel_name`, `data_key`); `null` where a channel omits it. |
| *attribute cols*| `str` | One column per `attribute_columns` entry (e.g. `unit`); `null` where a channel omits it. |
| `type` | `str` | `"CALC"`. |
| `data_type` | `str` | `"double"`. |
| *kpi cols* | `double` | One column per configured KPI, in order (default `duration`, `min`, `max`, `mean`). |

The KPIs are duration-weighted (matching the silver ingestion semantics): `duration` is the span
`max(tend) - min(tstart)`, `min` / `max` ignore NaN values, and `mean` is the duration-weighted average.
Select which KPIs to compute via `config.calculated_channels.kpis`; an identity key wins over an attribute
key of the same name.
28 changes: 26 additions & 2 deletions skills/impulse-channels/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ description: >
channels and materialized at the same per-sample grain. Use when the user wants to "add a calculated
channel", derive/persist a signal (e.g. "speed in km/h", "power = rpm × torque"), materialize a virtual
signal into a queryable table, or run `solve_calculated_channels`. Covers the reporting-layer
CalculatedChannel, the ad-hoc `QueryBuilder.solve_calculated_channels` endpoint, and the
calculated_channel_fact/dimension gold output.
CalculatedChannel, the ad-hoc `QueryBuilder.solve_calculated_channels` endpoint, the
calculated_channel_fact/dimension gold output, and the optional calculated_channel_metrics table.
---

# Impulse — calculated channels
Expand Down Expand Up @@ -98,6 +98,30 @@ across selections.
identity is **not** on the fact — it lives on the dimension; join on `channel_id` (and to
`measurement_dimension` on `container_id`; see `impulse-data-model`).

## Optional channel metrics table

`calculated_channel_fact` already matches the silver `channels` shape, so a calculated channel needs only a
companion `channel_metrics` table to become an Impulse silver source. Set
`config.calculated_channels.emit_channel_metrics = True` to also write `calculated_channel_metrics`, shaped
like silver `channel_metrics` (one row per `(container_id, channel_id)`, derived from the fact rows).

```python
from impulse_reporting.config.config_parser import CalculatedChannels

# in the ImpulseConfig:
calculated_channels = CalculatedChannels(
emit_channel_metrics=True,
attribute_columns=["unit"], # attribute keys to surface as columns; default []
kpis=["duration", "min", "max", "mean"], # default; each must be a registered KPI
)
```

The schema is **dynamic**: fixed `container_id`, `channel_id`, `type` (`"CALC"`), `data_type` (`"double"`),
one column per configured KPI, one per identity key (union across the report's channels), and one per
`attribute_columns` entry. `null` fills a key a channel omits; an identity key wins over an attribute key of
the same name. KPIs are duration-weighted; an unknown KPI name is rejected at config validation. Adding a
new KPI is a one-line entry in `impulse_reporting.channels.calculated_channel_kpis.KPI_BUILDERS`.

## Incremental

Calculated channels reuse the report's incremental engine (see `impulse-reporting`). A definition change —
Expand Down
19 changes: 18 additions & 1 deletion skills/impulse-config/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ description: >
"configure an Impulse report", set the source/sink tables, filter which containers are processed,
choose RLE vs RAW, turn on incremental processing, run without writing (sinkless), remap column
names, or scope by project. Covers source, unity_sink, container_filters, query_engine, solver_config,
incremental, and measurement_dimensions, all validated by Pydantic.
incremental, measurement_dimensions, and calculated_channels, all validated by Pydantic.
---

# Impulse — configuration
Expand Down Expand Up @@ -40,6 +40,7 @@ config = {
},
"incremental": {"enabled": True},
"measurement_dimensions": ["container_id", "vehicle_key", "start_ts", "stop_ts"],
"calculated_channels": {"emit_channel_metrics": True, "attribute_columns": ["unit"]}, # optional
}
```

Expand Down Expand Up @@ -172,3 +173,19 @@ List of `container_metrics` columns (post-mapping **internal** names) to surface
Default: `["container_id", "start_ts", "stop_ts"]`. Keep `container_id` — it is the incremental upsert
key and the join key to fact tables. Any column present in your post-mapping `container_metrics`
DataFrame is valid; a missing one fails the run fast with a `ValueError` naming it.

## calculated_channels (optional)

Controls the optional `calculated_channel_metrics` table. Off by default; when on, it is written alongside
`calculated_channel_fact` / `calculated_channel_dimension` in the silver `channel_metrics` shape, so the
fact + metrics pair can serve as an Impulse silver source. See `impulse-channels`.

| Field | Default | Description |
|------------------------|--------------------------------------|-----------------------------------------------------------------------------|
| `emit_channel_metrics` | `false` | Turn on the `calculated_channel_metrics` table. |
| `attribute_columns` | `[]` | Calculated-channel `attributes` keys to surface as columns (e.g. `["unit"]`). |
| `kpis` | `["duration", "min", "max", "mean"]` | KPIs computed per `(container_id, channel_id)`, one column each. |

Each metrics row is one `(container_id, channel_id)` pair with the selected `kpis` (duration-weighted) plus
dynamic identity columns (union of `identity` keys) and the configured `attribute_columns`; identity wins
over an attribute of the same name. An unknown KPI name is rejected at config validation.
Loading
Loading