feat(helm): observability support (ArcadeDB 26.7.1, #4463)

[robfrank]

Summary

Adds opt-in, behavior-preserving observability support for ArcadeDB 26.7.1 (issue ArcadeData/arcadedb#4463), covering all four pillars: health probes (#4464), OTLP metrics export (#4465), structured JSON logging (#4466), and distributed tracing (#4467).

A new feature-grouped observability: values section is translated by helpers into -D JVM args, plus Kubernetes-native Prometheus discovery the raw args can't provide.

• Metrics / OTLP — observability.metrics.otlp.{enabled,endpoint} pushes metrics to an OTel collector alongside the existing /prometheus endpoint.
• Prometheus discovery — optional ServiceMonitor (Prometheus Operator) and prometheus.io/* scrape pod annotations. A new requireAuthentication knob on the prometheus plugin enables unauthenticated scraping. A template guard fails fast if scrape discovery is enabled without the prometheus plugin.
• Tracing — observability.tracing.{enabled,endpoint,samplingRate}.
• Structured logging — observability.logging.{format,includeTrace} (text|json).
• Health probes — default liveness probe switched to the dependency-free /api/v1/health (no DB I/O, never 503); readiness stays on /api/v1/ready. Optional observability.health.readinessRequiresHA.
• Version — version/appVersion bumped to 26.7.1 (the release that serves /api/v1/health), so the default image and the new liveness default move together.

The ServiceMonitor selects only the {fullname}-http Service via a new app.kubernetes.io/component: http label, avoiding double-scraping the headless Service.

Every knob defaults off / behavior-preserving except the liveness path, which is coupled to the 26.7.1 bump.

Test Plan

• make test-unit — 159/159 pass across 13 suites (new observability_test.yaml, servicemonitor_test.yaml; updated statefulset_test.yaml, helpers_test.yaml, service_test.yaml)
• make lint — clean
• bash -n ci/integration-test.sh — clean; adds a /api/v1/health → 204 smoke phase (runs green under the latest-image guard)
• helm template fully-enabled and at-defaults renders verified
• kind HA integration job (pulls appVersion image — do not release until the arcadedb:26.7.1 image is published)

Note: before release, confirm the serverMetrics.otlp.* / serverMetrics.tracing.* property names against the running 26.7.1 server — unknown -D keys silently no-op.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces comprehensive, opt-in observability support for the ArcadeDB Helm chart (bumping the version to 26.7.1), including structured JSON logging, OTLP metrics, distributed tracing, and a new Prometheus ServiceMonitor template. It also updates the default liveness probe to use the dependency-free '/api/v1/health' endpoint. The review feedback highlights critical opportunities to prevent nil pointer dereference errors during template rendering by safely defaulting nested maps under '.Values.observability' in the helper templates and the ServiceMonitor manifest.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

The helper arcadedb.observability.args directly accesses nested keys under .Values.observability (such as .Values.observability.logging.format and .Values.observability.metrics.otlp.enabled) without checking if the parent keys exist. If a user provides a custom values file that partially overrides or completely omits the observability section, Helm will throw a nil pointer evaluation error (e.g., nil pointer evaluating interface {}). Using default dict to safely navigate these nested maps prevents rendering crashes.

{{- define "arcadedb.observability.args" -}}
{{- $o := .Values.observability | default dict -}}
{{- $logging := $o.logging | default dict -}}
{{- $metrics := $o.metrics | default dict -}}
{{- $otlp := $metrics.otlp | default dict -}}
{{- $tracing := $o.tracing | default dict -}}
{{- $health := $o.health | default dict -}}
{{- if eq $logging.format "json" }}
- -Darcadedb.server.logFormat=json
{{- end }}
{{- if $logging.includeTrace }}
- -Darcadedb.server.logIncludeTrace=true
{{- end }}
{{- if $otlp.enabled }}
- -Darcadedb.serverMetrics.otlp.enabled=true
- -Darcadedb.serverMetrics.otlp.endpoint={{ $otlp.endpoint }}
{{- end }}
{{- if $tracing.enabled }}
- -Darcadedb.serverMetrics.tracing.enabled=true
- -Darcadedb.serverMetrics.tracing.endpoint={{ $tracing.endpoint }}
- -Darcadedb.serverMetrics.tracing.samplingRate={{ $tracing.samplingRate }}
{{- end }}
{{- if $health.readinessRequiresHA }}
- -Darcadedb.server.readinessRequiresHA=true
{{- end }}
{{- end -}}

[robfrank]

Verified this empirically before applying. The stated trigger — partially overriding or omitting the observability section — does not panic: Helm coalesces user values over the chart's values.yaml defaults, so omitted/partially-set keys retain their defaults (helm template -f with only observability.metrics.otlp.enabled: true renders fine, exit 0; the unit suite also sets partial observability keys and passes).

The real failure mode is explicitly nulling a map, e.g. --set observability=null or --set observability.logging=null, which does dereference nil. That's a genuine crash, so I've applied nil-safe navigation (| default dict) to this helper and the other three sites, plus regression tests covering the explicit-null case.

Fixed in 9e17bbb.

[gemini-code-assist]

The validation helper arcadedb.observability.validate accesses .Values.observability.metrics.prometheus directly. If observability or metrics is omitted or overridden partially, this will cause a nil pointer dereference error during rendering. Safely defaulting the nested maps ensures robust template rendering.

{{- define "arcadedb.observability.validate" -}}
{{- $o := .Values.observability | default dict -}}
{{- $metrics := $o.metrics | default dict -}}
{{- $p := $metrics.prometheus | default dict -}}
{{- $serviceMonitor := $p.serviceMonitor | default dict -}}
{{- $podAnnotations := $p.podAnnotations | default dict -}}
{{- if or $serviceMonitor.enabled $podAnnotations.enabled -}}
  {{- $promEnabled := false -}}
  {{- with .Values.arcadedb.plugins.prometheus -}}
    {{- if .enabled -}}{{- $promEnabled = true -}}{{- end -}}
  {{- end -}}
  {{- if not $promEnabled -}}
    {{- fail "observability.metrics.prometheus serviceMonitor/podAnnotations require arcadedb.plugins.prometheus.enabled=true (the /prometheus endpoint must be served)" -}}
  {{- end -}}
{{- end -}}
{{- end -}}

[robfrank]

Addressed in 9e17bbb — arcadedb.observability.validate now safely defaults the nested maps. (See the thread above: normal partial overrides were already safe via Helm coalescing; the real crash was explicit nulling, now guarded with a regression test.)

[gemini-code-assist]

The helper arcadedb.podAnnotations accesses .Values.observability.metrics.prometheus.podAnnotations directly. If any of the parent keys are omitted in custom values, this will cause a nil pointer dereference error. Defaulting the nested maps safely avoids this issue.

{{- define "arcadedb.podAnnotations" -}}
{{- $annotations := deepCopy (default dict .Values.podAnnotations) -}}
{{- $o := .Values.observability | default dict -}}
{{- $metrics := $o.metrics | default dict -}}
{{- $prometheus := $metrics.prometheus | default dict -}}
{{- $pa := $prometheus.podAnnotations | default dict -}}
{{- if $pa.enabled -}}
  {{- $port := int .Values.service.http.port -}}
  {{- if $pa.port -}}{{- $port = int $pa.port -}}
  {{- end -}}
  {{- $_ := set $annotations "prometheus.io/scrape" "true" -}}
  {{- $_ := set $annotations "prometheus.io/port" (printf "%d" $port) -}}
  {{- $_ := set $annotations "prometheus.io/path" (default "/prometheus" $pa.path) -}}
{{- end -}}
{{- if $annotations -}}
{{- toYaml $annotations -}}
{{- end -}}
{{- end -}}

[robfrank]

Addressed in 9e17bbb — arcadedb.podAnnotations now safely navigates the nested maps with | default dict.

[gemini-code-assist]

In servicemonitor.yaml, accessing .Values.observability.metrics.prometheus.serviceMonitor.enabled and $sm.basicAuth.enabled directly can cause nil pointer dereference errors if observability or basicAuth is omitted or partially overridden in custom values. Safely defaulting these nested maps prevents rendering failures.

{{- $o := .Values.observability | default dict -}}
{{- $metrics := $o.metrics | default dict -}}
{{- $prometheus := $metrics.prometheus | default dict -}}
{{- $sm := $prometheus.serviceMonitor | default dict -}}
{{- $basicAuth := $sm.basicAuth | default dict -}}
{{- if $sm.enabled }}
{{- include "arcadedb.observability.validate" . -}}
{{- if and $basicAuth.enabled (not $basicAuth.secretName) -}}
{{- fail "serviceMonitor.basicAuth.enabled requires basicAuth.secretName (a secret with username + password keys)" -}}
{{- end -}}

[robfrank]

Addressed in 9e17bbb — servicemonitor.yaml now resolves $sm and $basicAuth via | default dict before use, so an explicitly nulled observability/serviceMonitor/basicAuth renders cleanly instead of panicking. Regression test added.