Formalize OSS self-hosted telemetry (emitter v2 + operator Telemetry UI)

[suguanYang]

Summary

Redesign Knowhere's anonymous self-hosted telemetry into a formal OSS usage metric system for Ontos operators, with:

• Emitter schema v2 (2026-07-telemetry-v2) — real periodic heartbeats, SaaS-/usage-parity aggregates, document-type + client-type mix, success rates
• Official clients auto-attaching document_metadata.created_by_client / client_version
• knowhere-metrics-dashboard /telemetry UI matching the planned Core usage + adoption story
• User-facing transparency (knowhere + self-hosted docs + dashboard disclosure)

Primary audience: Ontos operators understanding open-source / self-hosted adoption.
Transport: PostHog (unchanged). Logfire/OTEL out of scope.

Goals / product questions

Question	Metric
How many installs are alive?	Active installs = distinct installation ID with ≥1 heartbeat in 7d / 30d
Are installs sticking?	W0→W1 / W+4 retention via started ∩ heartbeat
Same KPIs as SaaS /usage?	Fleet Core usage: created/done/failed, success rate, backlog, pages, avg/P95 duration, trends, source type
Which clients create jobs?	Allowlisted document_metadata.created_by_client
What documents do they handle?	Allowlisted extension buckets + jobs/pages/success rate (never filenames)
Version / config / health?	Last-known version & flags; real heartbeat probes

Cross-repo scope

Phase	Repo(s)	Outcome
P0	knowhere	ADR-0004 — taxonomy, privacy, allowlists, metric IDs
P1	knowhere	Emitter v2 + contract tests + README/.env.example
P2	knowhere-node-sdk, knowhere-cli, knowhere-dashboard, notebook/MCP	Auto-attach created_by_client
P3	knowhere-metrics-dashboard	/telemetry queries + UI (SaaS /usage parity)
P4	knowhere, knowhere-self-hosted, knowhere-dashboard	Transparency docs / Settings disclosure

Track follow-up PRs in those repos and link them here.

Implementation checklist

P0 — ADR

• Add docs/adr/0004-anonymous-self-hosted-telemetry.md (+ index)
• Lock schema_version = 2026-07-telemetry-v2
• Lock allowlists: document types, created_by_client, source_type
• Lock success rate: done / (done + failed) as 0–1
• Document metric IDs (oss.active_installs_*, oss.usage.*, oss.client_*, oss.document_type_*, …)

P1 — Emitter v2 (knowhere)

• Extend events.py allowlists + normalize_document_type / normalize_client_name
• Periodic real heartbeat (api/postgres/redis/uptime); shutdown includes base props
• Usage aggregate: success_rate_24h, job_duration_p95_seconds_24h, source_* counts, capability buckets
• New events: self_hosted_document_type_aggregate, self_hosted_client_aggregate
• Contract tests (PII strip, normalizers, heartbeat/shutdown)
• Document TELEMETRY_* in .env.example + README Telemetry section

P2 — Official clients

• knowhere-node-sdk — default created_by_client=node-sdk + package version
• knowhere-cli — cli
• knowhere-dashboard — dashboard
• notebook / MCP — where jobs are created
• Caller override still wins; unit tests for default merge

P3 — Operator UI (knowhere-metrics-dashboard)

• Extend PostHog queries for adoption, core usage, client mix, document types, health, retention
• Rewrite /telemetry page sections (match planned mock; Core usage Chinese labels like SaaS /usage)
• Update docs/requirements.md + docs/guide.md; expand unit tests
• Graceful empty when new events absent

P4 — Transparency

• Expand knowhere-self-hosted configuration docs (en + zh)
• Dashboard Settings/About disclosure when BILLING_ENABLED=false

Locked design notes

• Reuse existing job create document_metadata.created_by_client / client_version — do not invent a new API field (JobCreateBase).
• Keep existing PostHog event names for continuity; add the two new aggregate events only.
• Do not emit free-form client_version into PostHog aggregates in v1 (cardinality).
• Opt-out remains default-on + TELEMETRY_ENABLED=false (document clearly; do not flip to opt-in in v1).

Non-goals (v1)

• Replacing Logfire/OTEL
• Per-user / content / filename / geo telemetry
• Model/provider popularity charts; GitHub stars merge into Telemetry
• Measuring installs that disabled telemetry
• Perfect multi-pod API counters
• Customer-facing OSS adoption charts in knowhere-dashboard

Acceptance

• Operators see Core usage on /telemetry with the same KPI set as SaaS /usage (fleet aggregates)
• Client mix + document type + success rate visible
• Official clients populate created_by_client by default
• Heartbeats are periodic and reflect real health
• Opt-out and collection documented in knowhere + self-hosted + dashboard disclosure

References

• SaaS KPI reference: knowhere-metrics-dashboard /usage
• Current emitter: knowhere/packages/shared-python/shared/services/telemetry/
• Current Telemetry UI: knowhere-metrics-dashboard/app/(dashboard)/telemetry/page.tsx

[suguanYang]

OSS telemetry redesign PRs (plan implementation):

• knowhere (P0/P1 emitter + ADR): Expand OSS telemetry emitter with ADR-0004 schema v2 #214
• knowhere-node-sdk (P2): Auto-attach Node SDK and MCP client metadata for OSS telemetry knowhere-node-sdk#122
• knowhere-cli (P2): https://github.com/Ontos-AI/knowhere-cli/pull/9
• knowhere-dashboard (P2 + P4 Settings About): Attach dashboard client metadata and disclose OSS telemetry in Settings knowhere-dashboard#69
• knowhere-notebook (P2): Attach notebook client metadata on Knowhere job creates knowhere-notebook#139
• knowhere-metrics-dashboard (P3): https://github.com/Ontos-AI/knowhere-metrics-dashboard/pull/11
• knowhere-self-hosted (P4 docs): Document anonymous OSS telemetry schema and opt-out knowhere-self-hosted#18

[suguanYang]

Update: skipping `knowhere-dashboard` instrumentation. The web dashboard does not create parse jobs, so there is no `created_by_client=dashboard` client. Allowlists/docs updated to `cli|node-sdk|notebook|mcp|api|other`. Dashboard PR: Ontos-AI/knowhere-dashboard#69 (closed).