Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions .changeset/eve-close-remaining-gaps.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
---
"evlog": minor
---

Close the remaining gaps in the eve integration, and add `defineEvlogInstrumentation()`.

A wide event and an Agent Runs span describe the same turn, but nothing joined them: you could not jump from a trace in Braintrust, Datadog or the Vercel dashboard to the event in your drain. Export the new definition from `agent/instrumentation.ts` and every model-call span carries `evlog.request_id` and `evlog.session_id`, the values the wide event reports as `requestId` and `eve.sessionId`:

```ts
// agent/instrumentation.ts
import { defineEvlogInstrumentation } from 'evlog/eve'

export default defineEvlogInstrumentation()
```

Without `setup`, OpenTelemetry export is untouched and eve keeps writing its local traces. `functionId`, `recordInputs`, `recordOutputs` and `traceChannelRequests` pass through to eve.

Three more of eve's stream events now reach the wide event:

- `eve.reasoning` — `blocks` and `chars`, the size of the model's thinking. The reasoning text itself is never recorded.
- `message.responseChars`, and `message.response` once `message` is `'preview'` or `'full'` — the agent's answer, following the same rule as the incoming message.
- `eve.result` — the structured result of an agent with an output schema.
59 changes: 53 additions & 6 deletions apps/docs/content/5.use-cases/5.eve.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ links:
[eve](https://eve.dev/docs/introduction) ships built-in observability: **Agent Runs** on Vercel and optional **OpenTelemetry** spans via `agent/instrumentation.ts`. `evlog/eve` adds a third layer — **exportable wide events** per turn with your full evlog pipeline (drains, enrichers, tail sampling, audit).

::callout{icon="i-lucide-info" color="info"}
eve is currently in beta; hook and stream event shapes may change before GA. Pin `eve` and `evlog` versions in production agents.
`evlog/eve` requires **eve 0.30 or later**. eve is still in beta and stream event shapes may change before GA, so pin `eve` and `evlog` versions in production agents.
::

::prompt
Expand All @@ -34,8 +34,9 @@ Add evlog wide events to my eve agent.
- Create agent/hooks/evlog.ts with defineEvlogHook from 'evlog/eve'
- Pass drain, enrich, and keep options (same as HTTP middleware integrations)
- In tools, import useLogger from 'evlog/eve' and call useLogger() inside execute() — the turn logger is bound via AsyncLocalStorage when defineEvlogHook() is registered; pass ctx only if ALS is unavailable in your runtime
- User message content is redacted by default (redactMessage: true); set false only after reviewing PII policy
- Keep eve Agent Runs / OTel instrumentation — evlog/eve is additive
- User message content is omitted by default (message: 'omit'); use 'preview' or 'full' only after reviewing PII policy
- Optionally add agent/instrumentation.ts with defineEvlogInstrumentation from 'evlog/eve' to join OTel spans to the wide events
- Keep eve Agent Runs — evlog/eve is additive

Docs: https://www.evlog.dev/use-cases/eve
Adapters: https://www.evlog.dev/integrate/adapters/overview
Expand All @@ -49,6 +50,7 @@ Adapters: https://www.evlog.dev/integrate/adapters/overview
| Debug a session in Vercel | eve **Agent Runs** (automatic) |
| Span-level traces in Datadog / Honeycomb | eve **`agent/instrumentation.ts`** + OTel exporter |
| Wide events to Axiom / Better Stack / FS, billing, audit, tail sampling | **`evlog/eve`** |
| Jumping from a span to its wide event, and back | **`defineEvlogInstrumentation()`** — [below](#correlate-traces-with-wide-events) |

## Quick Start

Expand Down Expand Up @@ -176,7 +178,50 @@ After approval, the next turn carries the outcome:
}
```

Token usage and tool executions are accumulated from eve stream events (`step.completed`, `actions.requested`, `action.result`). Business fields set via `useLogger()` carry across turns in the same session. Link turns in analytics with `eve.sessionId` + `eve.turnSequence` — each event stays self-contained. `eve.phase` is only set for non-routine endings (`awaiting-approval`, `rejected`, `failed`).
Token usage and tool executions are accumulated from eve stream events (`step.completed`, `actions.requested`, `action.result`). Business fields set via `useLogger()` carry across turns in the same session. Link turns in analytics with `eve.sessionId` + `eve.turnSequence` — each event stays self-contained. `eve.phase` is only set for non-routine endings: `awaiting-approval`, `awaiting-authorization`, `rejected`, `cancelled`, `failed`.

### Everything the turn can report

Beyond the identifiers above, a turn records what eve reports about it. Every field is optional — it appears only when the turn produced it.

| Field | What it tells you |
| --- | --- |
| `eve.runtime` | eve version, agent id, model, and the deployed `gitSha` / `gitBranch` / `deployedAt` |
| `eve.parent` | Parent and root session ids for a subagent run — rebuild the delegation tree with `rootSessionId` |
| `eve.authorizations` | Connection sign-ins with their `outcome`, `reason` and duration |
| `eve.compaction` | How many compactions ran, on which model, and `inputTokensAtTrigger` — how full the context was when the first one fired |
| `eve.stepFailures` / `eve.failedSteps` | Model calls that failed, including on a turn that then succeeded on retry |
| `eve.subagents` | Delegations with their status and duration |
| `eve.reasoning` | `blocks` and `chars` — the size of the model's thinking, never its content |
| `eve.result` | The structured result, for an agent with an output schema |
| `eve.contextCleared` | The durable history was wiped during this turn |
| `message.responseChars` | Length of the agent's response, recorded whatever the `message` mode |
| `ai.costUsd` | Cost as reported by eve. `ai.estimatedCost` is the fallback computed from `cost` |

## Correlate traces with wide events

A wide event and an Agent Runs span describe the same turn, but nothing joins them on its own. `defineEvlogInstrumentation()` stamps evlog's turn identity onto eve's AI SDK spans:

```typescript [agent/instrumentation.ts]
import { defineEvlogInstrumentation } from 'evlog/eve'

export default defineEvlogInstrumentation()
```

Every model-call span — and its children — then carries `evlog.request_id` and `evlog.session_id`, the same values the wide event reports as `requestId` and `eve.sessionId`. Jump from a trace in Braintrust, Datadog or Agent Runs straight to the event in your drain, and back.
Comment thread
HugoRCD marked this conversation as resolved.

Without `setup`, OpenTelemetry export is untouched: eve keeps writing its local traces, readable with `eve traces`. Pass one to export elsewhere:

```typescript [agent/instrumentation.ts]
import { defineEvlogInstrumentation } from 'evlog/eve'
import { registerOTel } from '@vercel/otel'

export default defineEvlogInstrumentation({
setup: ({ agentName }) => registerOTel({ serviceName: agentName }),
})
```

`functionId`, `recordInputs`, `recordOutputs` and `traceChannelRequests` pass straight through to eve's `defineInstrumentation`.

## Production

Expand Down Expand Up @@ -216,8 +261,10 @@ export default defineEvlogHook({
| --- | --- |
| `init` | Passed to `initLogger()` on first hook invocation |
| `drain` / `enrich` / `keep` / `plugins` | Same as HTTP integrations ([plugins](/extend/plugins)) |
| `redactMessage` | Default `true` — omits user message text from the wide event |
| `cost` / `model` | Optional token pricing (`ModelCost` from `evlog/ai`) → `ai.estimatedCost` |
| `message` | `'omit'` (default), `'preview'` or `'full'` — how much of the user message and the agent response to record |
| `messagePreviewLength` | Characters kept in `'preview'` mode (default `500`) |
| `sessionEvent` | `true` emits one extra event per session, rolling up its turns |
| `cost` / `model` | Fallback token pricing (`ModelCost` from `evlog/ai`) → `ai.estimatedCost`, used only when eve reports no cost |
| `maxSessions` | In-memory session cap for context carry-over (default `256`) |
| `include` / `exclude` | Route-style filters on turn paths (`/sessions/*/turns/*`) |

Expand Down
14 changes: 14 additions & 0 deletions examples/eve/agent/instrumentation.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
import { defineEvlogInstrumentation } from 'evlog/eve'

/**
* Stamps `evlog.request_id` onto eve's AI SDK spans so a trace joins back to
* the wide event for the same turn.
*
* No `setup` here: eve keeps writing its local traces, readable with
* `eve traces` or `/traces` in the dev TUI. Add one to export elsewhere:
*
* ```ts
* setup: ({ agentName }) => registerOTel({ serviceName: agentName })
* ```
*/
export default defineEvlogInstrumentation()
4 changes: 2 additions & 2 deletions examples/eve/agent/tools/issue_refund.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,9 @@ export default defineTool({
orderId: z.string(),
reason: z.string().describe('Why the customer is getting a refund'),
}),
needsApproval: ({ toolInput }) => {
approval: ({ toolInput }) => {
const order = findOrder(String(toolInput?.orderId ?? ''))
return orderRequiresApproval(order)
return orderRequiresApproval(order) ? 'user-approval' : 'not-applicable'
},
async execute({ orderId, reason }) {
await fakeLatency(900, 1600)
Expand Down
2 changes: 1 addition & 1 deletion examples/eve/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@
"class-variance-authority": "0.7.1",
"clsx": "2.1.1",
"cmdk": "1.1.1",
"eve": "^0.12.3",
"eve": "^0.30.8",
Comment thread
HugoRCD marked this conversation as resolved.
"evlog": "workspace:*",
"lucide-react": "1.16.0",
"motion": "12.40.0",
Expand Down
11 changes: 10 additions & 1 deletion packages/evlog/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -585,7 +585,16 @@ const log = useLogger()
log.set({ order: { id: input.orderId } })
```

`defineEvlogHook()` maps eve turn lifecycle events to one wide event per turn. Call `useLogger()` in tools — the logger is bound via AsyncLocalStorage on `turn.started`. Pass `ctx` only when ALS is unavailable (`useLogger(ctx)`). Pretty-printing follows `isDev()` by default (tree locally, JSON in production); set `init.pretty: false` explicitly if you need to override. Complements eve Agent Runs and OpenTelemetry — see the [eve use case](https://evlog.dev/use-cases/eve).
```typescript
// agent/instrumentation.ts — joins eve's OTel spans to the wide events
import { defineEvlogInstrumentation } from 'evlog/eve'

export default defineEvlogInstrumentation()
```

`defineEvlogHook()` maps eve turn lifecycle events to one wide event per turn. Call `useLogger()` in tools — the logger is bound via AsyncLocalStorage on `turn.started`. Pass `ctx` only when ALS is unavailable (`useLogger(ctx)`). Pretty-printing follows `isDev()` by default (tree locally, JSON in production); set `init.pretty: false` explicitly if you need to override.
Comment thread
HugoRCD marked this conversation as resolved.

`defineEvlogInstrumentation()` is optional: it stamps `evlog.request_id` onto eve's AI SDK spans so a trace joins back to its wide event, and back. Requires eve 0.30 or later. Complements eve Agent Runs — see the [eve use case](https://evlog.dev/use-cases/eve).

See the full [eve example](https://github.com/HugoRCD/evlog/tree/main/examples/eve) for a complete agent layout.

Expand Down
139 changes: 139 additions & 0 deletions packages/evlog/src/eve/index.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,12 @@
import { AsyncLocalStorage } from 'node:async_hooks'
import { defineHook, type HookContext, type HookDefinition } from 'eve/hooks'
import {
defineInstrumentation,
type InstrumentationDefinition,
type InstrumentationSetupContext,
type InstrumentationStepStartedEventInput,
type InstrumentationStepStartedEventResult,
} from 'eve/instrumentation'
import type { AuditableLogger } from '../audit'
import type { AIToolExecution, AIEventData, ModelCost } from '../ai/index'
import { initLogger, isLoggerInitialized, isLoggerLocked } from '../logger'
Expand Down Expand Up @@ -160,6 +167,11 @@ interface TurnAccumulator {
compactionModel?: string
compactionInputTokens?: number
contextCleared: boolean
reasoningBlocks: number
reasoningChars: number
response?: string
responseChars: number
result?: unknown
pausedForInput: boolean
stepStartedAt?: number
costMap?: Record<string, ModelCost>
Expand Down Expand Up @@ -218,6 +230,9 @@ function freshAccumulator(options: EvlogEveOptions): TurnAccumulator {
compactions: 0,
compactionsRequested: 0,
contextCleared: false,
reasoningBlocks: 0,
reasoningChars: 0,
responseChars: 0,
pausedForInput: false,
costMap: options.cost,
costModel: resolveCostModel(options),
Expand Down Expand Up @@ -688,8 +703,24 @@ function flushEveMetadata(state: TurnState): void {
}
}
if (acc.contextCleared) eve.contextCleared = true
// Reasoning size only — the reasoning text itself is never recorded.
if (acc.reasoningBlocks > 0) {
eve.reasoning = { blocks: acc.reasoningBlocks, chars: acc.reasoningChars }
}
if (acc.result !== undefined) eve.result = acc.result

if (Object.keys(eve).length > 0) state.logger.set({ eve })

// Response length is recorded in every message mode; the response text is
// only present when the mode allowed the handler to keep it.
if (acc.responseChars > 0) {
state.logger.set({
message: {
responseChars: acc.responseChars,
...(acc.response !== undefined ? { response: acc.response } : {}),
},
})
}
}

/** Wide-event view of the eve instance and the parent session, when there is one. */
Expand Down Expand Up @@ -1124,6 +1155,36 @@ export function defineEvlogHook(options: EvlogEveOptions = {}): HookDefinition {
})
},

'reasoning.completed'(event, ctx) {
runSafe(() => {
const state = getTurnState(ctx.session.id, event.data.turnId)
if (!state) return
state.accumulator.reasoningBlocks += 1
state.accumulator.reasoningChars += event.data.reasoning.length
})
},

'message.completed'(event, ctx) {
runSafe(() => {
const state = getTurnState(ctx.session.id, event.data.turnId)
if (!state || event.data.message === null) return
const acc = state.accumulator
acc.responseChars += event.data.message.length
if (messageMode === 'omit') return
acc.response = messageMode === 'full'
? event.data.message
: truncateMessage(event.data.message, previewLength)
})
},

'result.completed'(event, ctx) {
runSafe(() => {
const state = getTurnState(ctx.session.id, event.data.turnId)
if (!state) return
state.accumulator.result = event.data.result
})
},

'actions.requested'(event, ctx) {
runSafe(() => {
const state = getTurnState(ctx.session.id, event.data.turnId)
Expand Down Expand Up @@ -1333,6 +1394,84 @@ export function defineEvlogHook(options: EvlogEveOptions = {}): HookDefinition {
})
}

/** Options for {@link defineEvlogInstrumentation}. */
export interface EvlogEveInstrumentationOptions {
/** Overrides `ai.telemetry.functionId` on spans. Defaults to the agent name. */
functionId?: string
/** Whether the AI SDK records full model inputs on spans. eve defaults to `true`. */
recordInputs?: boolean
/** Whether the AI SDK records model outputs on spans. eve defaults to `true`. */
recordOutputs?: boolean
/** Whether eve emits the inbound HTTP `SERVER` span wrapping each channel request. */
traceChannelRequests?: boolean
/**
* Runs at server startup with the resolved agent name — register your OTel
* provider here, exactly as in a hand-written `defineInstrumentation`. Omit it
* and eve keeps writing its local traces.
*/
setup?: (context: InstrumentationSetupContext) => void
}

/**
* Per-model-call context linking an AI SDK span back to the evlog wide event
* for the same turn. Returns `undefined` outside a tracked turn, which
* contributes no context rather than a half-filled one.
*/
function buildInstrumentationContext(
input: InstrumentationStepStartedEventInput,
): InstrumentationStepStartedEventResult | undefined {
const state = getTurnState(input.session.id, input.turn.id)
if (!state) return undefined

return {
runtimeContext: {
'evlog.request_id': state.turnId,
'evlog.session_id': state.sessionId,
},
}
}

/**
* Create an eve instrumentation definition that stamps evlog's turn identity
* onto the AI SDK telemetry spans.
*
* Export the result as the default export of `agent/instrumentation.ts`. Every
* model-call span — and its children — then carries `evlog.request_id`, the
* same value the wide event reports as `requestId`, so a trace in Braintrust,
* Datadog or Agent Runs joins to the wide event in your drain, and back.
*
* `eve.` is reserved for framework-owned context, so evlog writes under
* `evlog.`. Passing no `setup` leaves OpenTelemetry export untouched: eve keeps
* recording its local traces and only the runtime context is added.
*
* @example
* ```ts
* // agent/instrumentation.ts
* import { defineEvlogInstrumentation } from 'evlog/eve'
* import { registerOTel } from '@vercel/otel'
*
* export default defineEvlogInstrumentation({
* setup: ({ agentName }) => registerOTel({ serviceName: agentName }),
* })
* ```
*/
export function defineEvlogInstrumentation(
options: EvlogEveInstrumentationOptions = {},
): InstrumentationDefinition {
return defineInstrumentation({
...(options.functionId !== undefined ? { functionId: options.functionId } : {}),
...(options.recordInputs !== undefined ? { recordInputs: options.recordInputs } : {}),
...(options.recordOutputs !== undefined ? { recordOutputs: options.recordOutputs } : {}),
...(options.traceChannelRequests !== undefined
? { traceChannelRequests: options.traceChannelRequests }
: {}),
...(options.setup !== undefined ? { setup: options.setup } : {}),
events: {
'step.started': buildInstrumentationContext,
},
})
}

/** @internal Simulates eve tool execution where AsyncLocalStorage did not propagate. */
export function detachActiveTurnLoggerForTests(): void {
const logger = turnLoggerStorage.getStore()
Expand Down
Loading
Loading