-The dashboard is organized into seven tabs:
+The dashboard is organized into ten tabs:
| Tab | What it shows | When to use it |
|---|---|---|
| Health | Cluster state, data consistency, restarts, reindexing, resource limits | Stability monitoring, post-incident review |
| Autoscaling: Content | +Per content cluster: CPU peak vs. ideal load, the query/write mix and query-growth that move the ideal target, load adjustment, active nodes, rescale actions | +Understanding why and by how much a content cluster is being resized |
| Autoscaling: Container | +The same set of panels for each container cluster | +Understanding why and by how much a container cluster is being resized |
| Metric Reference | +Alphabetical index of every metric referenced by the dashboard, with the tab(s) each one appears on | +Looking up where a specific metric is graphed (browser Ctrl+F) |
| Explore | +Ad-hoc: plot any metric, aggregated per cluster (optionally per host) | +A quick look at a metric not covered by the curated tabs |
Filters at the top apply across all tabs:
@@ -64,13 +77,14 @@- Query, Feed, Content Node, Resources, and Health tabs group metrics per cluster — + Query, Feed, Content Node, Resources, Health, and both Autoscaling tabs group metrics per cluster — you see all metrics for one cluster before scrolling to the next. Container metrics are grouped per container cluster, content metrics per content cluster.
Annotations are vertical lines drawn on every chart that mark operational events. @@ -88,6 +102,9 @@
The Overview tab opens with a dedicated Health Indicators row — - five stat panels designed to surface stability issues in a single glance. + stat panels designed to surface stability and data-consistency issues at a glance. A row of green zeros is the signal to stop; a non-zero value tells you which tab to visit next.
| Container: Services Down | Active container nodes where some service isn't running | 0 during steady state; brief spikes during deployments are expected |
| Buckets Out of Sync (%) | +Percentage of data buckets not yet replicated/consistent, per content cluster (green = synced, blue = rebalancing) | +0% at steady state. Non-zero is normal and expected during scaling, node replacement, upgrades, and recovery, and should converge back to 0 — investigate only if it persists with no such event or fails to converge. See the Health tab for the trend |
- The bottom row gives a quick view of CPU, memory, and disk across all clusters. - Any resource consistently above 80% warrants attention. + CPU, memory, and disk are shown in two rows — one for content clusters and + one for container clusters — because the two cluster types have different + expectations. Each cluster is drawn in its own colour so several clusters are easy to tell apart.
+
The Query tab groups per-rank-profile metrics into four sub-rows, all filterable by @@ -298,33 +331,117 @@
Vespa supports two NNS modes:
approximate-threshold (default 0.02).
+ Approximate NNS works well in most cases, but is less suited when the filter is very restrictive,
+ as that makes the graph hard to navigate. For a restrictive filter, exact NNS works well since only
+ few documents have to be checked. Vespa automatically falls back to exact NNS when the filter hit
+ ratio is below the approximate-threshold (default 0.02). In this border region, the
+ filter-first heuristic ("Acorn-1") controlled by filter-first-threshold can avoid the
+ latency spikes of classical HNSW.
+
approximate-threshold is too high.filter-first-exploration is too high.+ The query rate is split into the queries that perform an approximate NNS and the rest + (which includes queries that fall back to exact NNS), so the two add up to the total. + This shows at a glance how much of the traffic exercises the HNSW index. +
+ ++ Latency is decomposed so you can tell where time is being spent. The total splits into setup + and matching, and setup splits again to isolate approximate NNS: +
+Key metrics:
+ +approximate-threshold is too large, making Vespa fall back to exact NNS too early.
- Tuning parameters (set per
- rank profile):
- approximate-threshold, filter-first-threshold,
- target-hits-max-adjustment-factor, exploration-slack.
- If the exact NNS ratio is high, consider increasing approximate-threshold
- or restructuring filters to be less restrictive.
+ Besides targetHits, several
+ rank-profile parameters
+ (each with a query-parameter
+ counterpart) tune HNSW behaviour:
+
approximate-threshold (default 0.02) — filter hit ratio below which a query
+ falls back to exact NNS. If set wrong, queries crossing it can spike in latency.filter-first-threshold (default 0.2) — filter hit ratio below which the filter
+ is checked before a distance computation. Improves latency at low hit ratios but may reduce recall;
+ set to 0.0 to disable the heuristic.filter-first-exploration (default 0.01) — how much of the graph the filter-first
+ heuristic explores. Higher improves recall but worsens latency — increase with caution.exploration-slack (default 0.0) — an alternative to raising targetHits
+ for better recall, at the cost of latency.post-filter-threshold, target-hits-max-adjustment-factor — configure
+ post-filtering, useful for queries with high filter hit ratios.
+ The safe way to experiment is the
+ ranking.matching.anntimebudget
+ query parameter, which caps the time spent on approximate NNS so latency cannot explode while you tune.
+ To debug a specific slow query, capture a profile with the
+ Vespa CLI query command and its
+ --profile option.
+ By default each panel shows one line per content cluster. Enable the + Per host metrics toggle to break disk and memory down per node — + useful for spotting a single node approaching its feed-block limit while the + cluster view still looks healthy, for example when data is unevenly distributed + across nodes. +
Proton uses several thread pools (executors):
@@ -458,6 +582,7 @@
The Resources tab's JVM row separates the three layers of container memory:
- After scaling events, expect buckets out of sync and pending merges. - These should converge back to zero. If they don't, investigate. + This is a progress signal, not a severity gauge — so the panels carry + no green/orange/red thresholds. It is expected to be non-zero, often high, + during normal operations: scaling resizes, node replacement, rolling upgrades, and recovery + all redistribute data, and a freshly added node can be close to 100% out of sync until it + receives its share. The value alone does not indicate a problem; what matters is its behaviour + over time. Read it alongside the + Auto-scaling, Vespa upgrade, and Data migration annotations: + a rise that lines up with one of those events and then converges back to zero + is healthy. Investigate only when it is non-zero with no corresponding event, or when + it fails to converge long after the triggering event. (The Overview stat is + coloured neutrally — green when synced, blue while rebalancing — for the same reason.)
+ The Autoscaling: Content and Autoscaling: Container tabs + explain why autoscaling resizes a cluster and roughly + how big the change is. They are most useful when a resize surprised you, or when you + are deciding whether to enable autoscaling in the first place — the panels render even + while autoscaling is disabled, so you can see what it would do. + Each tab repeats its set of panels once per cluster (content clusters on the first tab, + container clusters on the second). +
++ The autoscaler compares each cluster's peak load (the busiest point in the + recent window) against an ideal load target it maintains per resource. The + ideal target sits intentionally below 100% — it already reserves headroom for traffic + growth and for absorbing traffic if another zone becomes unavailable. Peak above ideal pushes + the cluster to scale up; peak held comfortably below ideal scales it + down. +
++ Of the per-resource targets the autoscaler tracks, only CPU peak and ideal are + exported to the Vespa Cloud metric tier (memory and disk are still considered internally; their + targets simply aren't published), so these tabs focus on CPU. +
+| Panel | What it shows |
|---|---|
| CPU load — peak vs ideal | +Peak observed CPU load against the autoscaler's ideal CPU target. The gap between + them is the scale-up (or scale-down) pressure, and its size drives how big the resize is. |
| Query vs write rate — CPU mix driver | +The query and write rates side by side. The ideal CPU target is a blend of a + query-oriented and a write-oriented target, weighted by where CPU is spent — so a + shift in this mix moves the ideal line even at steady total load. |
| Query growth (rate vs 3h average) | +Query rate divided by its own trailing 3-hour average. Above 1.0 means traffic is + trending up, which makes the autoscaler reserve growth headroom (lowering the ideal CPU + target) and can trigger a scale-up even when current load looks flat. A direction + indicator, not an exact value. |
| CPU load adjustment (peak / ideal) | +The central signal: peak CPU divided by ideal CPU. 1.0 means ideally scaled for CPU; + above 1.0 is under-provisioned, below 1.0 over-provisioned. The value roughly sets the + size of the next resize (1.5 ≈ needs about 50% more CPU). |
| Active nodes | +Active node count with the non-active fraction overlaid. Step changes mark completed + resizes; a non-zero non-active fraction means a change is in progress — for content + clusters this includes data redistribution. |
| Autoscaling actions | +Rescale decisions in the selected window. Line each bump up with the node-count and + load-adjustment panels to see what triggered it. These events also appear as the + Auto-scaling annotation on every chart. |
+ Reading the "why it scaled when load looks flat" case: the ideal target moves. + When the query/write mix shifts toward writes, when query growth rises above 1.0, or when the + autoscaler reserves capacity so the cluster can absorb traffic from a zone that becomes + unavailable, the ideal CPU target drops — and a scale-up can follow at steady observed + load. Watching the ideal line, not just peak, is the key. The zone-failover + reservation is not directly graphed, so if the ideal line moves while the query/write mix, query + growth, and node count all look flat, that reservation is the likely cause. +
++ Content clusters scale on CPU, memory and disk, and a resize can change node count, + group count, or node size; because data must be redistributed, the non-active fraction stays + above zero until rebalancing completes. Container clusters are CPU-driven and scale by node + count only — with no data to move, their resizes complete quickly. +
+ + +
+ The Metric Reference tab is a single text panel listing every Prometheus-exported metric
+ currently queried by the dashboard, paired with the tab(s) it is graphed on and a
+ Per host column marking the metrics you can break down per node with the
+ Per host metrics toggle. Open the tab
+ and use the browser’s find shortcut (Ctrl+F / Cmd+F) to look up where a metric appears.
+ The list is alphabetical and includes the aggregator suffix (_max,
+ _sum, _count, _rate, _average,
+ _min, _95percentile, _99percentile); some metrics
+ appear on multiple tabs.
+
+ The table is generated from the dashboard JSON, so it stays in sync with the panels. + No queries, panels, or filters live on this tab beyond the reference itself. +
+ + +
+ An ad-hoc explorer for metrics not covered by the curated tabs. Pick a metric from the
+ Explore metric dropdown and an aggregation from
+ Explore aggregation (sum / avg / max /
+ min); the panel plots the selected metric aggregated per cluster, with Vespa’s
+ admin components (cluster controllers and logserver) excluded. Toggle
+ Per host metrics to break it down per node when the metric carries a
+ host label.
+
+ It is deliberately generic — the Y-axis unit is not metric-specific and the same + aggregation is applied to whatever metric you pick. Use the + Metric Reference tab or the + metrics reference to find metric names; for curated, + unit-correct charts prefer the dedicated tabs. +
+ +