Agent Observability
Trace, cost, OTEL export, and Grafana dashboards for AI agents. Zero dependencies.

Quick Start · Modules · Exporters · Ecosystem

---

What Is Qhaway?

Qhaway (Quechua: "to observe/watch") wraps every LLM call your agent makes and captures cost, latency, token usage, and model info. Export to OpenTelemetry, Prometheus, or MLflow. Works in Cloudflare Workers, Node.js, Deno, Bun.

import { QhawayTrace, ConsoleStorage } from '@carloscortezcloud/qhaway/trace';

const trace = new QhawayTrace(new ConsoleStorage(), { agent_id: 'my-agent' });

const wrapped = trace.wrap(myLlmCall, { model: 'gpt-4o', provider: 'openai', user_id: 'abc' });
const result = await wrapped(prompt);

// Console output:
// [Qhaway] ✓ gpt-4o (openai) | $0.00063 | 150→42 tok | 1234ms | user=abc

Install

npm install @carloscortezcloud/qhaway

Quick Start

import { QhawayTrace, ConsoleStorage } from '@carloscortezcloud/qhaway/trace';

const trace = new QhawayTrace(new ConsoleStorage(), { agent_id: 'my-agent' });
const wrapped = trace.wrap(myLlmCall, { model: 'gpt-4o', provider: 'openai', user_id: 'abc' });
const result = await wrapped(prompt);

Modules

Qhaway is a single package with subpath exports — import only what you need:

Import path	What
@carloscortezcloud/qhaway	All-in-one entry
@carloscortezcloud/qhaway/trace	Span wrapper + storage (D1/KV/Console)
@carloscortezcloud/qhaway/cost	Pricing DB + cost attribution
@carloscortezcloud/qhaway/otel	OTLP/HTTP JSON exporter
@carloscortezcloud/qhaway/tinkuy	Auto-instrument TinkuyAgent
@carloscortezcloud/qhaway/mlflow	MLflow metrics exporter
@carloscortezcloud/qhaway/alerts	Threshold alerts (Slack/webhook/email/PagerDuty)
@carloscortezcloud/qhaway/eval	Eval run tagging + promptfoo import + eval metrics
@carloscortezcloud/qhaway/ui	Trace viewer tree + filters (feeds ui/index.html)

Storage

Adapter	Best for	Import
D1Storage	Production (SQL, aggregation)	@carloscortezcloud/qhaway/trace (d1)
KVStorage	High-scale write	@carloscortezcloud/qhaway/trace (kv)
ConsoleStorage	Local dev / debug	@carloscortezcloud/qhaway/trace (console)

Exporters

Exporter	Destination
OTEL	Any OTLP collector (Honeycomb, Grafana Tempo, Datadog, SigNoz)
Prometheus	GET /metrics endpoint for Grafana dashboards
MLflow	Log cost/latency metrics as MLflow experiment runs
Alerts	Threshold rules → Slack/webhook/email/PagerDuty with cooldown

Feedback Loop

Correlate user feedback (thumbs up/down) with cost, latency, and model. Attach a rating to any span:

import { QhawayTrace, MemoryStorage, aggregateRating, ratingStats } from '@carloscortezcloud/qhaway';

const trace = new QhawayTrace(new MemoryStorage());

const wrapped = trace.wrap(myLlmCall, {
  model: 'gpt-4o',
  provider: 'openai',
  session_id: 'ses-1',
  rating: 1, // thumbs up — or -1 for down, 0 for neutral
});

rating flows through every storage adapter (D1/KV/Console) and the Prometheus endpoint exposes qhaway_rating_total{model, rating} and qhaway_cost_by_rating_total{rating}. The Grafana dashboard includes a Satisfaction vs Cost scatter panel.

import { aggregateRating } from '@carloscortezcloud/qhaway';

const spans = await storage.query();
const byRating = aggregateRating(spans);
// [{ model: 'gpt-4o', rating: -1, calls: 3, costUsd: 0.15, avgLatencyMs: 412, ... }]

const stats = ratingStats(spans);
// { thumbsUp: 40, thumbsDown: 7, avgCostPerThumbsDown: 0.09, thumbsDownRate: 0.15 }

Wire it to a Tinkuy feedback hook: on thumbs down, record a span with rating: -1 (see examples/feedback-wiring.ts).

Agent Evaluations

Tag eval cases with an eval_run_id, then compare cost vs score across models and runs:

import { labelEvalRun, aggregateEvalRuns, generateEvalMetrics } from '@carloscortezcloud/qhaway/eval';

const tagged = labelEvalRun(span, 'run-7', { score: 0.9, pass: true });

const runs = aggregateEvalRuns(spans);
// [{ eval_run_id: 'run-7', model: 'gpt-4o', passRate: 0.85, costPerScorePoint: 0.02, ... }]

const prom = generateEvalMetrics(spans); // qhaway_eval_run_* metrics for Grafana

Import promptfoo output directly — parsePromptfooOutput(json) converts results into labeled spans, so you can drop eval suites from promptfoo/LangChain into the same dashboards (qhaway-eval-dashboard.json). See examples/eval-comparison.ts.

Trace Viewer UI

Standalone, dependency-free HTML viewer for agent traces — no Grafana Tempo or LangSmith required. It renders session → iteration → tool-call trees, colors expensive spans red, filters by model/agent/date/success, and shows feedback ratings.

Run locally with demo data

cd qhaway
python3 -m http.server 4173 --directory ui

Open http://localhost:4173/index.html?endpoint=/demo-data.json — demo-data.json ships realistic spans (sessions, tool calls, failures, ratings).

Point at your own spans

Any URL returning QhawaySpan[] as JSON works:

open ui/index.html?endpoint=https://my-agent.example.com/spans

On Cloudflare Workers, serve from D1/KV with a tiny route:

import { D1Storage } from '@carloscortezcloud/qhaway/trace';

export default {
  async fetch(request, env) {
    const url = new URL(request.url);
    if (url.pathname === '/spans') {
      const storage = new D1Storage(env.QHAWAY_DB);
      return Response.json(await storage.query({}, 500));
    }
  },
};

Programmatic API

Tree-building and filtering are exported from the npm package:

import { getUiApi, buildTraceTree, costColor } from '@carloscortezcloud/qhaway/ui';

const api = getUiApi(storage); // or getUiApi('https://.../spans')
const spans = await api.loadSpans({ model: 'gpt-4o' });
const tree = buildTraceTree(spans);

for (const node of tree) {
  console.log(node.label, costColor(node.costUsd));
}

Full guide (data format, inject spans without a server, features): ui/README.md.

Python SDK

pip install qhaway-trace

from qhaway import QhawayTrace, console_storage
from qhaway.integrations import OpenAIPatch

trace = QhawayTrace(storage=console_storage)
OpenAIPatch.apply(trace)  # auto-instrument all OpenAI calls

See python/README.md for OpenAI, LangChain, Anthropic, and FastAPI examples.

Architecture

Your Agent (Tinkuy / LangChain / raw)
  │
  ▼
QhawayTrace.wrap(fn)
  │
  ├── D1/KV (storage)
  ├── OTLP (Honeycomb, Grafana, Datadog)
  └── GET /metrics → Prometheus → Grafana dashboard

Grafana Dashboard

Import qhaway-dashboard.json into Grafana (Cloud or OSS) to visualize:

• Cost by model and user
• Latency P99 over time
• Token usage (input vs output)
• Recent call log
• Daily spend summary

Ecosystem

Package	Role	npm
Qhaway	Agent obs (this)	@carloscortezcloud/qhaway
Styrr	LLM router	styrr
Sayay	Cost guardrails	GitHub
Tinkuy	Agent framework	@carloscortezcloud/tinkuy-agent
TideRAG	Edge RAG pipeline	@carloscortezcloud/tiderag

License

Apache 2.0 — see LICENSE.

---

Built by engineers who got tired of blind AI spending.
Tinkuy Labs · finoptix.dev

_{If you can't see the cost, you can't control it. Qhaway opens your eyes.}