-
Notifications
You must be signed in to change notification settings - Fork 57
feat(eve): package evlog as an installable eve extension #505
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
cdd05cc
cfcc20d
acae5a6
6171c6c
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| --- | ||
| "@evlog/eve": minor | ||
| "evlog": patch | ||
| --- | ||
|
|
||
| Add `@evlog/eve`, evlog packaged as an installable eve extension. | ||
|
|
||
| Mounting it replaces writing `agent/hooks/evlog.ts` by hand: | ||
|
|
||
| ```ts | ||
| // agent/extensions/evlog.ts | ||
| import evlog from '@evlog/eve' | ||
|
|
||
| export default evlog({ adapter: 'axiom', sample: 0.1 }) | ||
| ``` | ||
|
|
||
| The mount config is declarative — an adapter by name (or several, which fan out), a sampling rate, redaction paths, batching — because eve validates extension config synchronously and it therefore cannot carry functions. Agents that need a custom `drain`, `enrich` or `keep` keep using `defineEvlogHook` from `evlog/eve`. | ||
|
|
||
| The extension also contributes what a hook cannot: an `annotate` tool and a skill that teach the agent to record its own business context on the turn's event, and to keep message content and personal data out of it. | ||
|
|
||
| `defineEvlogHook` now warns when it runs more than once in a process. Every hook accumulates into the same turn, so mounting the extension alongside a hand-written hook records token and step counts once per hook. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| --- | ||
| "evlog": minor | ||
| --- | ||
|
|
||
| Add `defineEvlogInstrumentation()` to `evlog/eve`, linking eve's OpenTelemetry spans to evlog wide events. | ||
|
|
||
| A wide event and an Agent Runs span describe the same turn, but nothing joined them: you could not jump from a trace in Braintrust, Datadog or the Vercel dashboard to the event in your drain. Export it as the default export of `agent/instrumentation.ts` and every model-call span — and its children — carries `evlog.request_id` and `evlog.session_id`, the same values the wide event reports. | ||
|
|
||
| ```ts | ||
| // agent/instrumentation.ts | ||
| import { defineEvlogInstrumentation } from 'evlog/eve' | ||
| import { registerOTel } from '@vercel/otel' | ||
|
|
||
| export default defineEvlogInstrumentation({ | ||
| setup: ({ agentName }) => registerOTel({ serviceName: agentName }), | ||
| }) | ||
| ``` | ||
|
|
||
| `setup` is optional: without it, OpenTelemetry export is untouched and eve keeps recording its local traces, with only the runtime context added. `functionId`, `recordInputs`, `recordOutputs` and `traceChannelRequests` pass straight through to eve. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -46,6 +46,7 @@ jobs: | |
| dx | ||
| elysia | ||
| eve | ||
| eve-extension | ||
| evi | ||
| express | ||
| fastify | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| node_modules | ||
| .env* | ||
| .eve | ||
| .vercel | ||
| .output | ||
| .nitro | ||
| dist | ||
| .DS_Store | ||
| *.tsbuildinfo |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,35 @@ | ||
| # @evlog/eve | ||
|
|
||
| evlog packaged as an eve extension. Consumers mount it under | ||
| `agent/extensions/` instead of writing `agent/hooks/evlog.ts` by hand. | ||
|
|
||
| ## This package does not build like the others | ||
|
|
||
| It builds with `eve extension build`, not tsdown, and the output tree | ||
| (`dist/extension`) is agent-shaped rather than a bundle. `eve extension build` | ||
| requires **Node >= 24**. | ||
|
|
||
| `prepare` and `dev:prepare` are deliberately absent. The release script already | ||
| runs `turbo run build --filter='./packages/*'` before `changeset publish`, so | ||
| the dist exists when it matters. Wiring the build into `dev:prepare` would put | ||
| it on the critical path of `lint`, `typecheck` and `test`, which then fail for | ||
| anyone on Node 22 — nothing in this repo consumes this package's dist. | ||
|
|
||
| ## Config cannot carry functions | ||
|
|
||
| The mount config is a Standard Schema, validated synchronously, so `drain`, | ||
| `enrich` and `keep` cannot cross the mount boundary. Everything consumers | ||
| configure has to be declarative — an adapter name, a sampling rate, redaction | ||
| paths. Agents that genuinely need a callback use `defineEvlogHook` from | ||
| `evlog/eve` directly; keep that escape hatch documented. | ||
|
|
||
| ## Layout | ||
|
|
||
| - `extension/extension.ts` — the config schema, and the handle contributions read | ||
| - `extension/lib/` — shared code: adapter resolution, config → hook options | ||
| - `extension/hooks/` — the wide-event hook | ||
| - `extension/tools/` — `annotate`, mounted as `<namespace>__annotate` | ||
| - `extension/skills/` — procedures taught to the host agent | ||
|
|
||
| Tool, skill and connection names come from file paths and the consumer's mount | ||
| adds the namespace prefix, so name the file `annotate`, never `evlog_annotate`. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,85 @@ | ||
| # @evlog/eve | ||
|
|
||
| [evlog](https://evlog.dev) packaged as an [eve](https://eve.dev) extension: one | ||
| wide event per agent turn, sent to the destination of your choice, without | ||
| writing a hook. | ||
|
|
||
| ## Install | ||
|
|
||
| ```bash | ||
| pnpm add @evlog/eve | ||
| ``` | ||
|
|
||
| ```ts | ||
| // agent/extensions/evlog.ts | ||
| import evlog from '@evlog/eve' | ||
|
|
||
| export default evlog({ adapter: 'axiom', sample: 0.1 }) | ||
| ``` | ||
|
|
||
| That is the whole setup. Every turn now produces an event carrying token usage, | ||
| cost, tool executions, subagent delegations, approvals, connection | ||
| authorizations, compactions and failures. | ||
|
|
||
| ## Config | ||
|
|
||
| | Option | Default | What it does | | ||
| | --- | --- | --- | | ||
| | `adapter` | `'fs'` | Destination by name, or `{ type, options }`. An array fans out. | | ||
| | `service` | agent name | Service name on every event. | | ||
| | `message` | `'omit'` | `'omit'`, `'preview'` or `'full'` user message capture. | | ||
| | `messagePreviewLength` | `500` | Characters kept in `'preview'`. | | ||
| | `redact` | `true` | Built-in PII patterns, or `{ paths }`. | | ||
| | `sample` | keep all | Fraction of routine turns to keep, 0 to 1. | | ||
| | `keepOnFailure` | `true` | Always keep failed, rejected and declined turns. | | ||
| | `sessionEvent` | `false` | One extra event per session, rolling up its turns. | | ||
| | `batch` | none | `{ size, intervalMs }` buffering before draining. | | ||
|
|
||
| Adapters read their own environment variables, so `adapter: 'axiom'` with | ||
| `AXIOM_TOKEN` and `AXIOM_DATASET` set is usually all you need. `options` | ||
| overrides what the environment cannot express. | ||
|
|
||
| Available adapters: `axiom`, `better-stack`, `clickhouse`, `datadog`, `fs`, | ||
| `hyperdx`, `loki`, `memory`, `otlp`, `posthog`, `sentry`. | ||
|
|
||
| ## What the agent can do itself | ||
|
|
||
| The mount contributes an `annotate` tool (namespaced `evlog__annotate` for a | ||
| mount named `evlog.ts`) and a skill teaching the agent when to use it. The agent | ||
| records business facts on its own turn: | ||
|
|
||
| ``` | ||
| annotate({ key: "refund", value: { amount: 89.9, reason: "damaged_on_arrival" } }) | ||
| ``` | ||
|
|
||
| Those fields are what make a turn findable later. The skill also tells the agent | ||
| what never to record: message content, credentials, personal data. | ||
|
|
||
| ## When to use the hook instead | ||
|
|
||
| The mount config is validated synchronously, so it cannot carry functions. If | ||
| you need a custom `drain`, `enrich` or `keep`, skip the extension and use the | ||
| hook directly: | ||
|
|
||
| ```ts | ||
| // agent/hooks/evlog.ts | ||
| import { defineEvlogHook } from 'evlog/eve' | ||
|
|
||
| export default defineEvlogHook({ | ||
| drain: myDrain, | ||
| enrich: ctx => void (ctx.event.region = process.env.VERCEL_REGION), | ||
| }) | ||
| ``` | ||
|
|
||
| Use one or the other. Mounting both makes every hook accumulate into the same | ||
| turn, which records token and step counts once per hook. | ||
|
|
||
| ## Correlating with OpenTelemetry | ||
|
|
||
| `evlog/eve` also exports `defineEvlogInstrumentation()`, which stamps | ||
| `evlog.request_id` onto eve's AI SDK spans so a trace joins back to the wide | ||
| event. See the [eve integration docs](https://evlog.dev/use-cases/eve). | ||
|
|
||
| ## License | ||
|
|
||
| MIT |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,62 @@ | ||
| import { defineExtension } from 'eve/extension' | ||
| import { z } from 'zod' | ||
|
|
||
| /** Drain adapters the extension can resolve by name. */ | ||
| export const ADAPTER_NAMES = [ | ||
| 'axiom', | ||
| 'better-stack', | ||
| 'clickhouse', | ||
| 'datadog', | ||
| 'fs', | ||
| 'hyperdx', | ||
| 'loki', | ||
| 'memory', | ||
| 'otlp', | ||
| 'posthog', | ||
| 'sentry', | ||
| ] as const | ||
|
|
||
| const adapter = z.union([ | ||
| z.enum(ADAPTER_NAMES), | ||
| z.object({ | ||
| type: z.enum(ADAPTER_NAMES), | ||
| /** Passed straight to the adapter factory, overriding its env defaults. */ | ||
| options: z.record(z.string(), z.unknown()).optional(), | ||
| }), | ||
| ]) | ||
|
|
||
| /** | ||
| * Consumer-facing settings. Everything is declarative: a Standard Schema is | ||
| * validated synchronously at the mount site, so it cannot carry functions. | ||
| * Agents needing a custom `drain`, `enrich` or `keep` use `defineEvlogHook` | ||
| * from `evlog/eve` in `agent/hooks/` instead. | ||
| */ | ||
| export default defineExtension({ | ||
| config: z.object({ | ||
| /** Service name on every event. Defaults to the agent name. */ | ||
| service: z.string().optional(), | ||
| /** Where events go. Several adapters fan out. */ | ||
| adapter: z.union([adapter, z.array(adapter)]).default('fs'), | ||
| /** How much of the user message to record. */ | ||
| message: z.enum(['omit', 'preview', 'full']).default('omit'), | ||
| /** Max characters kept in `preview` mode. */ | ||
| messagePreviewLength: z.number().int().positive().optional(), | ||
| /** PII auto-redaction: `true` for the built-in patterns, or explicit paths. */ | ||
| redact: z | ||
| .union([z.boolean(), z.object({ paths: z.array(z.string()) })]) | ||
| .default(true), | ||
| /** Fraction of routine turns to keep, between 0 and 1. Unset keeps all. */ | ||
| sample: z.number().min(0).max(1).optional(), | ||
| /** Always keep turns that failed, were rejected, or lost an authorization. */ | ||
| keepOnFailure: z.boolean().default(true), | ||
| /** Emit one extra wide event per session, rolling up its turns. */ | ||
| sessionEvent: z.boolean().default(false), | ||
| /** Batch events before draining them. */ | ||
| batch: z | ||
| .object({ | ||
| size: z.number().int().positive(), | ||
| intervalMs: z.number().int().positive(), | ||
| }) | ||
| .optional(), | ||
| }), | ||
| }) | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| import { defineEvlogHook } from 'evlog/eve' | ||
| import extension from '../extension' | ||
| import { toHookOptions, type ExtensionConfig } from '../lib/options' | ||
|
|
||
| /** | ||
| * One evlog wide event per agent turn, wired from the mount config. | ||
| * | ||
| * The agent name is not available to a hook file at module scope, so the | ||
| * service falls back to the mount's `service` and is otherwise resolved from | ||
| * the turn context by evlog itself. | ||
| */ | ||
| export default defineEvlogHook( | ||
| toHookOptions(extension.config as ExtensionConfig, extension.config.service ?? 'eve-agent'), | ||
|
Comment on lines
+5
to
+13
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win Correct the service fallback documentation. Line 10 says evlog resolves an omitted service from turn context. Line 13 passes Update the JSDoc to describe the fixed fallback, or change the implementation to resolve the agent name as documented. 🤖 Prompt for AI Agents |
||
| ) | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| Every turn you run is recorded as one observability event. | ||
|
|
||
| Call the `annotate` tool when you learn a business fact that would make this turn | ||
| findable later — an identifier you looked up, a decision you made, an amount you | ||
| moved. Do not pass user message content, credentials, or personal data. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,84 @@ | ||
| import type { DrainContext } from 'evlog' | ||
| import { createAxiomDrain } from 'evlog/axiom' | ||
| import { createBetterStackDrain } from 'evlog/better-stack' | ||
| import { createClickHouseDrain } from 'evlog/clickhouse' | ||
| import { createDatadogDrain } from 'evlog/datadog' | ||
| import { createFsDrain } from 'evlog/fs' | ||
| import { createHyperDXDrain } from 'evlog/hyperdx' | ||
| import { createLokiDrain } from 'evlog/loki' | ||
| import { createMemoryDrain } from 'evlog/memory' | ||
| import { createOTLPDrain } from 'evlog/otlp' | ||
| import { createPostHogDrain } from 'evlog/posthog' | ||
| import { createSentryDrain } from 'evlog/sentry' | ||
| import { createDrainPipeline } from 'evlog/pipeline' | ||
| import type { ADAPTER_NAMES } from '../extension' | ||
|
|
||
| type AdapterName = (typeof ADAPTER_NAMES)[number] | ||
| /** Adapters accept a single context or a batch, which is what lets them sit behind a pipeline. */ | ||
| type AdapterDrain = (ctx: DrainContext | DrainContext[]) => void | Promise<void> | ||
| /** What `BaseEvlogOptions.drain` expects: one event at a time. */ | ||
| type HookDrain = (ctx: DrainContext) => void | Promise<void> | ||
| type DrainFactory = (overrides?: Record<string, unknown>) => AdapterDrain | ||
|
|
||
| /** | ||
| * Every adapter factory reads its own env vars, so a bare name is enough for | ||
| * the common case and `options` only overrides what the environment cannot. | ||
| */ | ||
| const FACTORIES: Record<AdapterName, DrainFactory> = { | ||
| 'axiom': createAxiomDrain as DrainFactory, | ||
| 'better-stack': createBetterStackDrain as DrainFactory, | ||
| 'clickhouse': createClickHouseDrain as DrainFactory, | ||
| 'datadog': createDatadogDrain as DrainFactory, | ||
| 'fs': createFsDrain as DrainFactory, | ||
| 'hyperdx': createHyperDXDrain as DrainFactory, | ||
| 'loki': createLokiDrain as DrainFactory, | ||
| 'memory': createMemoryDrain as DrainFactory, | ||
| 'otlp': createOTLPDrain as DrainFactory, | ||
| 'posthog': createPostHogDrain as DrainFactory, | ||
| 'sentry': createSentryDrain as DrainFactory, | ||
| } | ||
|
|
||
| export type AdapterConfig = | ||
| | AdapterName | ||
| | { type: AdapterName, options?: Record<string, unknown> } | ||
|
|
||
| export interface ResolveDrainOptions { | ||
| adapter: AdapterConfig | AdapterConfig[] | ||
| batch?: { size: number, intervalMs: number } | ||
| } | ||
|
Comment on lines
+41
to
+48
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win Add JSDoc to exported configuration types.
As per coding guidelines, “Write JSDoc for all public TypeScript APIs.” 🤖 Prompt for AI AgentsSource: Coding guidelines |
||
|
|
||
| function createDrain(config: AdapterConfig): AdapterDrain { | ||
| const { type, options } = typeof config === 'string' ? { type: config, options: undefined } : config | ||
| return FACTORIES[type](options) | ||
| } | ||
|
|
||
| /** Each destination runs independently so a failing one cannot starve the others. */ | ||
| async function fanOut( | ||
| drains: AdapterDrain[], | ||
| payload: DrainContext | DrainContext[], | ||
| ): Promise<void> { | ||
| await Promise.all(drains.map(async (send) => { | ||
| try { | ||
| await send(payload) | ||
| } catch (err) { | ||
| console.error('[evlog] eve extension drain failed:', err) | ||
| } | ||
| })) | ||
| } | ||
|
|
||
| /** | ||
| * Build the single drain the hook receives. With `batch`, the adapters sit | ||
| * behind a pipeline that buffers events and hands them over as an array — | ||
| * which every evlog adapter accepts. | ||
| */ | ||
| export function resolveDrain(options: ResolveDrainOptions): HookDrain { | ||
| const configs = Array.isArray(options.adapter) ? options.adapter : [options.adapter] | ||
| const drains = configs.map(createDrain) | ||
|
|
||
| if (options.batch) { | ||
| return createDrainPipeline<DrainContext>({ batch: options.batch })( | ||
| (batch: DrainContext[]) => fanOut(drains, batch), | ||
| ) | ||
| } | ||
| return (ctx: DrainContext) => fanOut(drains, ctx) | ||
| } | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Reject an empty adapter list.
adapter: []passes this schema.resolveDrain()then creates no drains, and events complete without delivery or an error. Require at least one adapter.Proposed fix
📝 Committable suggestion
🤖 Prompt for AI Agents