-
Notifications
You must be signed in to change notification settings - Fork 57
feat(evi): ground answers in real sources and add an eval suite #508
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
HugoRCD
wants to merge
7
commits into
main
Choose a base branch
from
feat/evi-grounding
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+1,325
−22
Open
Changes from all commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
6a01e71
feat(evi): ground answers in real sources and add an eval suite
HugoRCD 95ce5b1
fix(evi): normalize the channel kind and tighten write authorization
HugoRCD 99660ac
Merge remote-tracking branch 'origin/main' into feat/evi-grounding
HugoRCD a1046ea
fix(evi): drop the drain guard and broaden the injection claim check
HugoRCD 5aa86ea
docs(evi): use plain wording in the design notes
HugoRCD 9f05668
fix(evi): name a tool that exists and key the tier cache by actor
HugoRCD 780c334
Merge remote-tracking branch 'origin/main' into feat/evi-grounding
HugoRCD File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,5 +1,29 @@ | ||
| import { defineAgent } from 'eve' | ||
| import { defineAgent, defineDynamic } from 'eve' | ||
| import { gatewayRouting, sessionTags } from './lib/gateway' | ||
|
|
||
| const MODEL = 'deepseek/deepseek-v4-flash' | ||
|
|
||
| export default defineAgent({ | ||
| model: 'google/gemini-3.6-flash', | ||
| model: defineDynamic({ | ||
| fallback: MODEL, | ||
| events: { | ||
| 'session.started': (_event, ctx) => ({ | ||
| model: MODEL, | ||
| modelOptions: { | ||
| providerOptions: { | ||
| gateway: { ...gatewayRouting, tags: sessionTags(ctx.channel.kind) }, | ||
| }, | ||
| }, | ||
| }), | ||
| }, | ||
| }), | ||
| /** This model honors only `high` and `xhigh`. */ | ||
| reasoning: 'high', | ||
| limits: { | ||
| maxInputTokensPerSession: 5_000_000, | ||
| maxOutputTokensPerSession: 100_000, | ||
| }, | ||
| modelOptions: { | ||
| providerOptions: { gateway: { ...gatewayRouting, tags: sessionTags() } }, | ||
| }, | ||
| }) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| import { defineMcpClientConnection } from 'eve/connections' | ||
|
|
||
| export default defineMcpClientConnection({ | ||
| url: 'https://www.evlog.dev/mcp', | ||
| description: | ||
| 'The published evlog documentation — the authority on what evlog does today: API surface, wide events, structured errors, sampling, redaction, the CLI, framework integrations, drain adapters, and extension points. `list-pages` returns every page with its title, path and description; `get-page` returns one page\'s full markdown plus the canonical URL to cite. Use it for any question about how evlog behaves or how to configure it. It does not cover unreleased work, source-level implementation detail, or anything specific to a user\'s own project.', | ||
| tools: { | ||
| allow: [ | ||
| 'list-pages', | ||
| 'get-page' | ||
| ] | ||
| }, | ||
| }) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| import { connect } from '@vercel/connect/eve' | ||
| import { defineMcpClientConnection } from 'eve/connections' | ||
|
|
||
| export default defineMcpClientConnection({ | ||
| url: 'https://mcp.linear.app/mcp', | ||
| description: 'Linear workspace: issues, projects, cycles, and comments.', | ||
| auth: connect({ connector: 'mcp.linear.app/linear-mcp', principalType: 'app' }), | ||
| }) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,6 +1,84 @@ | ||
| import githubExtension from '@github-tools/eve-extension' | ||
|
|
||
| const TOOLS = [ | ||
| // Repository and code | ||
| 'getRepository', | ||
| 'getRepositoryTree', | ||
| 'getFileContent', | ||
| 'searchCode', | ||
| 'getBlame', | ||
| 'listBranches', | ||
| 'listCommits', | ||
| 'getCommit', | ||
| 'compareCommits', | ||
| 'createBranch', | ||
| 'createOrUpdateFile', | ||
|
|
||
| // Issues | ||
| 'searchIssues', | ||
| 'listIssues', | ||
| 'getIssueContext', | ||
| 'createIssue', | ||
| 'updateIssue', | ||
| 'closeIssue', | ||
| 'addIssueComment', | ||
| 'updateIssueComment', | ||
| 'deleteIssueComment', | ||
|
|
||
| // Triage | ||
| 'listLabels', | ||
| 'addLabels', | ||
| 'removeLabel', | ||
| 'addAssignees', | ||
| 'removeAssignees', | ||
| 'addIssueReaction', | ||
| 'addCommentReaction', | ||
|
|
||
| // Pull requests | ||
| 'listPullRequests', | ||
| 'getPullRequestContext', | ||
| 'listPullRequestFiles', | ||
| 'listPullRequestReviews', | ||
| 'createPullRequest', | ||
| 'updatePullRequest', | ||
| 'addPullRequestComment', | ||
| 'updatePullRequestComment', | ||
| 'deletePullRequestComment', | ||
| 'createPullRequestReview', | ||
| 'requestReviewers', | ||
|
|
||
| // Discussions | ||
| 'listDiscussions', | ||
| 'getDiscussion', | ||
| 'addDiscussionComment', | ||
|
|
||
| // Releases, read only: AGENTS.md forbids agents from creating one | ||
| 'listReleases', | ||
| 'getLatestRelease', | ||
| 'getReleaseContext', | ||
|
|
||
| // CI, read only — diagnose a red build, never restart or cancel one | ||
| 'listCheckRuns', | ||
| 'getCiFailureContext', | ||
| ] as const | ||
|
|
||
| export default githubExtension({ | ||
| connector: 'github/evi-github-production', | ||
| preset: 'maintainer', | ||
| connect: { | ||
| scopes: [ | ||
| 'metadata:read', | ||
| 'contents:read', | ||
| 'contents:write', | ||
| 'issues:read', | ||
| 'issues:write', | ||
| 'pull_requests:read', | ||
| 'pull_requests:write', | ||
| 'discussions:read', | ||
| 'discussions:write', | ||
| 'checks:read', | ||
| 'actions:read', | ||
| ], | ||
| }, | ||
| context: { owner: 'HugoRCD', repo: 'evlog' }, | ||
| include: [...TOOLS], | ||
| }) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| import type { DrainContext } from 'evlog' | ||
| import { defineEvlogHook } from 'evlog/eve' | ||
| import { createFsDrain } from 'evlog/fs' | ||
| import { createDrainPipeline } from 'evlog/pipeline' | ||
| import { environment } from '../lib/environment' | ||
|
|
||
| const drain = createDrainPipeline<DrainContext>({ | ||
| batch: { size: 5, intervalMs: 2000 }, | ||
| })(createFsDrain()) | ||
|
|
||
| export default defineEvlogHook({ | ||
| init: { env: { service: 'evi', environment: environment() } }, | ||
| drain, | ||
| sessionEvent: true, | ||
| }) | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| import { defineDynamic, defineInstructions } from 'eve/instructions' | ||
| import { channelName } from '../lib/channel' | ||
|
|
||
| const CHECKED_OUT = `## Workspace | ||
|
|
||
| The evlog repository is checked out at \`/workspace\`, at the ref of the thread you were summoned on. Read it with \`glob\`, \`grep\` and \`read_file\` rather than the GitHub API — it is free, it is the code under discussion, and \`grep\` takes real regular expressions. The checkout is shallow, so use \`github__getBlame\` for history. | ||
|
|
||
| Every path you pass to those tools must be absolute: \`grep "x" --glob "/workspace/packages/evlog/src/**"\`. A repo-relative path is rejected outright.` | ||
|
|
||
| const EMPTY = `## Workspace | ||
|
|
||
| \`/workspace\` is empty on this channel: there is no repository checkout, and \`glob\`, \`grep\` and \`read_file\` have nothing to find. Read repository files with \`github__searchCode\` and \`github__getFileContent\` instead.` | ||
|
|
||
| /** Only the GitHub channel checks the triggering ref out into the sandbox. */ | ||
| export default defineDynamic({ | ||
| events: { | ||
| 'turn.started': (_event, ctx) => | ||
| defineInstructions({ | ||
| markdown: channelName(ctx.channel.kind) === 'github' ? CHECKED_OUT : EMPTY, | ||
| }), | ||
| }, | ||
| }) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| import { defineInstrumentation } from 'eve/instrumentation' | ||
| import { evlogRuntimeContext } from 'evlog/eve' | ||
|
|
||
| /** | ||
| * OpenTelemetry spans for every turn, carrying evlog's correlation ids and the | ||
| * calling principal. Register an exporter through `setup` to ship them. | ||
| */ | ||
| export default defineInstrumentation({ | ||
| events: { | ||
| 'step.started': (input) => { | ||
| const caller = input.session.auth.current | ||
| return { | ||
| runtimeContext: { | ||
| ...evlogRuntimeContext(input), | ||
| // Omitted rather than blank: an empty attribute reads as an empty id. | ||
| ...(caller ? { 'caller.principal_id': caller.principalId } : {}), | ||
| ...(caller ? { 'caller.principal_type': caller.principalType } : {}), | ||
| }, | ||
| } | ||
| }, | ||
| }, | ||
| }) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| /** | ||
| * The channel name eve reports, without its prefix. | ||
| * | ||
| * Framework channels arrive bare (`http`, `schedule`, `subagent`); authored ones | ||
| * as `channel:<filename>`, so `agent/channels/github.ts` is `channel:github`. | ||
| * Comparing against the bare name without stripping never matches. | ||
| */ | ||
| export function channelName(kind?: string): string { | ||
| return (kind ?? 'unknown').replace(/^channel:/, '') | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| /** | ||
| * Where this process is running, as one label. | ||
| * | ||
| * Shared by the gateway spend tags and the evlog wide events so a run that bills | ||
| * as `eval` also logs as `eval`. `EVE_RUN_MODE` is set by the `eval` script; it | ||
| * does not reach a deployment behind `eve eval --url`. | ||
| */ | ||
| export function environment(): string { | ||
| if (process.env.EVE_RUN_MODE === 'eval') return 'eval' | ||
| return process.env.VERCEL_ENV ?? 'local' | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| import { channelName } from './channel' | ||
| import { environment } from './environment' | ||
|
|
||
| /** Routing shared by every gateway call. `sort` keeps following the cheapest deployment. */ | ||
| export const gatewayRouting = { | ||
| caching: 'auto', | ||
| sort: 'cost', | ||
| } as const | ||
|
|
||
| /** | ||
| * Tags stamped on every gateway request, read back through the spend report. | ||
| * | ||
| * One tag per dimension, not one compound string: the report groups by a single | ||
| * dimension at a time, so this yields a row per environment and a row per surface. | ||
| */ | ||
| export function sessionTags(kind?: string): string[] { | ||
| return [`evi:env:${environment()}`, `evi:surface:${channelName(kind)}`] | ||
| } | ||
|
|
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
Repository: HugoRCD/evlog
Length of output: 50372
🏁 Script executed:
Repository: HugoRCD/evlog
Length of output: 30186
🌐 Web query:
Vercel how can I use files in serverless functions filesystem is not writable storage ephemeral💡 Result:
Vercel serverless functions have a read-only filesystem, meaning you cannot persistently store files in your project directory [1][2]. Because these functions are ephemeral—they spin up to handle a request and shut down shortly after—any changes made to the local environment do not persist across different invocations [2][3]. To manage files in your Vercel Functions, use the following approaches: Temporary Scratch Space If you need to process files temporarily (e.g., during a single function execution), you can use the /tmp directory [1][3]. This is the only writable area of the filesystem, with a storage limit of up to 500 MB [1]. Keep in mind that this storage is not guaranteed to persist across different function executions; it is strictly for short-term scratch space [2][3]. Persistent Storage For files that must persist across requests or deployments, you should use an external storage solution: 1. Vercel Blob: This is Vercel's recommended object storage service for uploading and serving files [4][5]. It is designed for both public assets (like images and videos) and private files [5]. You can use the Vercel Blob SDK to upload, retrieve, and manage files from your serverless functions [5][6]. 2. Database or Redis: If you are storing state, configuration, or small amounts of data, use a database or a key-value store like Redis instead of files [2][4]. Including Static Files If you need to read static files that are part of your project (e.g., configuration files or templates), Vercel's build process (using Node File Trace) automatically includes most necessary files [4]. If specific files are not being included, you can configure them to be bundled by modifying your vercel.json or your framework's configuration (such as Next.js's outputFileTracingIncludes) [4].
Citations:
Avoid sending evlog events to a non-durable filesystem drain in deployed environments.
createFsDrain()only disables itself when the configured directory is unwritable. On Vercel,/tmpis writable, but it is temporary scratch storage and does not preserve logs across function invocations. Branch on deployed environments now, and avoid addingcreateFsDrain()when local writes are ephemeral.🤖 Prompt for AI Agents