From 90322228deb46f4abf5da20ef214163b56f0cc54 Mon Sep 17 00:00:00 2001 From: bjormel Date: Fri, 31 Jul 2026 14:32:23 +0000 Subject: [PATCH] Updated to reflect changes in Tenant Grafana Dashboard --- en/operations/monitoring.html | 298 ++++++++++++++++++++++++++++++---- 1 file changed, 270 insertions(+), 28 deletions(-) diff --git a/en/operations/monitoring.html b/en/operations/monitoring.html index c14267ff2f..b91f3bc5be 100644 --- a/en/operations/monitoring.html +++ b/en/operations/monitoring.html @@ -27,8 +27,9 @@

The Vespa Cloud metrics dashboard

Tabs and filters

+ Dashboard tab bar -

The dashboard is organized into seven tabs:

+

The dashboard is organized into ten tabs:

@@ -55,6 +56,18 @@

Tabs and filters

+ + + + + + + + + + + +
TabWhat it showsWhen to use it
Health Cluster state, data consistency, restarts, reindexing, resource limits Stability monitoring, post-incident review
Autoscaling: ContentPer content cluster: CPU peak vs. ideal load, the query/write mix and query-growth that move the ideal target, load adjustment, active nodes, rescale actionsUnderstanding why and by how much a content cluster is being resized
Autoscaling: ContainerThe same set of panels for each container clusterUnderstanding why and by how much a container cluster is being resized
Metric ReferenceAlphabetical index of every metric referenced by the dashboard, with the tab(s) each one appears onLooking up where a specific metric is graphed (browser Ctrl+F)
ExploreAd-hoc: plot any metric, aggregated per cluster (optionally per host)A quick look at a metric not covered by the curated tabs

Filters at the top apply across all tabs:

@@ -64,13 +77,14 @@

Tabs and filters

  • Rank Profile — filter per-rank-profile panels on the Query tab (defaults to "All")

    • - Query, Feed, Content Node, Resources, and Health tabs group metrics per cluster — + Query, Feed, Content Node, Resources, Health, and both Autoscaling tabs group metrics per cluster — you see all metrics for one cluster before scrolling to the next. Container metrics are grouped per container cluster, content metrics per content cluster.

    Annotations

    + Latency chart with Service restart and Core dump annotations

    Annotations are vertical lines drawn on every chart that mark operational events. @@ -88,6 +102,9 @@

    Annotations

    Vespa upgrade A new Vespa version is rolled out Brief rolling-restart latency spikes are expected around this marker + OS upgrade + Nodes start reporting a new host OS version + Nodes are rebuilt on the new OS, so brief per-node capacity dips are expected Data migration Bucket merges pending exceed a threshold Explains elevated CPU/IO and latency during redistribution @@ -114,10 +131,11 @@

    Overview tab

    Health Indicators

    + Overview tab Health Indicators row

    The Overview tab opens with a dedicated Health Indicators row — - five stat panels designed to surface stability issues in a single glance. + stat panels designed to surface stability and data-consistency issues at a glance. A row of green zeros is the signal to stop; a non-zero value tells you which tab to visit next.

    @@ -140,6 +158,9 @@

    Health Indicators

    + + +
    Container: Services Down Active container nodes where some service isn't running 0 during steady state; brief spikes during deployments are expected
    Buckets Out of Sync (%)Percentage of data buckets not yet replicated/consistent, per content cluster (green = synced, blue = rebalancing)0% at steady state. Non-zero is normal and expected during scaling, node replacement, upgrades, and recovery, and should converge back to 0 — investigate only if it persists with no such event or fails to converge. See the Health tab for the trend
    @@ -160,9 +181,20 @@

    QoS and latency overview

    Resource utilization

    - The bottom row gives a quick view of CPU, memory, and disk across all clusters. - Any resource consistently above 80% warrants attention. + CPU, memory, and disk are shown in two rows — one for content clusters and + one for container clusters — because the two cluster types have different + expectations. Each cluster is drawn in its own colour so several clusters are easy to tell apart.

    +

    Query tab

    @@ -208,6 +240,7 @@

    Container-level metrics

    Rank profile metrics

    + Query tab rank profile sub-rows

    The Query tab groups per-rank-profile metrics into four sub-rows, all filterable by @@ -298,33 +331,117 @@

    Nearest Neighbor Search tab

    This tab only appears when the application uses approximate nearest neighbor search — it is automatically hidden when no NNS distance computations are detected. + All metrics are grouped per content cluster, with a per-host toggle.

    Vespa supports two NNS modes:

    +

    + Approximate NNS works well in most cases, but is less suited when the filter is very restrictive, + as that makes the graph hard to navigate. For a restrictive filter, exact NNS works well since only + few documents have to be checked. Vespa automatically falls back to exact NNS when the filter hit + ratio is below the approximate-threshold (default 0.02). In this border region, the + filter-first heuristic ("Acorn-1") controlled by filter-first-threshold can avoid the + latency spikes of classical HNSW. +

    + +

    Overview and detail metrics

    + + +

    Query rate

    +

    + The query rate is split into the queries that perform an approximate NNS and the rest + (which includes queries that fall back to exact NNS), so the two add up to the total. + This shows at a glance how much of the traffic exercises the HNSW index. +

    + +

    Query latency breakdown

    +

    + Latency is decomposed so you can tell where time is being spent. The total splits into setup + and matching, and setup splits again to isolate approximate NNS: +

    + -

    Key metrics:

    + +

    Timeouts

    + +

    How to act on high latencies

    +
      +
    1. Check whether the system is simply overloaded — look at Query Rate and + Thread Pool – Match Utilization.
      2. +
    2. Identify the source — approximate or exact NNS? +
        +
      • Query Approximate NNS Time high — likely unoptimized HNSW parameters; + see the tuning parameters below. A query time budget is the safe way to experiment.
        • +
      • Query Setup Time high without approximate NNS time being high — + the filter computation is slow; restructuring the query could help.
        • +
      • Query Matching Time high — this is where exact NNS happens; indicates + approximate-threshold is too large, making Vespa fall back to exact NNS too early.
        • +
      +
      3. +
    + +

    HNSW parameter tuning

    - Tuning parameters (set per - rank profile): - approximate-threshold, filter-first-threshold, - target-hits-max-adjustment-factor, exploration-slack. - If the exact NNS ratio is high, consider increasing approximate-threshold - or restructuring filters to be less restrictive. + Besides targetHits, several + rank-profile parameters + (each with a query-parameter + counterpart) tune HNSW behaviour: +

    + +

    + The safe way to experiment is the + ranking.matching.anntimebudget + query parameter, which caps the time spent on approximate NNS so latency cannot explode while you tune. + To debug a specific slow query, capture a profile with the + Vespa CLI query command and its + --profile option.

    @@ -347,6 +464,13 @@

    Proton resource usage

    metrics in the Resources tab — these are the values Vespa uses for feed-blocking decisions.

    +

    + By default each panel shows one line per content cluster. Enable the + Per host metrics toggle to break disk and memory down per node — + useful for spotting a single node approaching its feed-block limit while the + cluster view still looks healthy, for example when data is unevenly distributed + across nodes. +

    Executor utilization

    Proton uses several thread pools (executors):

    @@ -458,6 +582,7 @@

    Container thread pools

    JVM memory breakdown

    + JVM memory breakdown: heap, direct, native, GC

    The Resources tab's JVM row separates the three layers of container memory: