feat(model-eval-ingest): sync promoted bench evals#3258
Open
lambertjosh wants to merge 11 commits into
Open
Conversation
![kilo-code-bot[bot]](https://avatars.githubusercontent.com/in/2193792?s=60&v=4){kind=small}
Contributor
Code Review SummaryStatus: No Issues Found | Recommendation: Merge Executive SummaryThe latest commit ( All Previous Issues Resolved (click to expand)
Files Reviewed (25 files)
Reviewed by claude-sonnet-4.6 · 315,716 tokens Review guidance: REVIEW.md from base branch |
Contributor
Author
|
Contributor
Author
|
Note that if/when this is merged, we will then need to:
|
jrf0110
reviewed
May 15, 2026
jrf0110
reviewed
May 15, 2026
jrf0110
reviewed
May 15, 2026
| scale: 6, | ||
| mode: 'number', | ||
| }), | ||
| avg_execution_ms: decimal('avg_execution_ms', { precision: 16, scale: 6, mode: 'number' }), |
Contributor
There was a problem hiding this comment.
Do we need floating point precision for a value measured in ms?
Contributor
Author
There was a problem hiding this comment.
No! Ugh. Fixing.
jrf0110
reviewed
May 15, 2026
Comment on lines
+3015
to
+3022
| avg_cost_usd: decimal('avg_cost_usd', { precision: 14, scale: 8, mode: 'number' }), | ||
| avg_input_tokens: decimal('avg_input_tokens', { precision: 16, scale: 6, mode: 'number' }), | ||
| avg_output_tokens: decimal('avg_output_tokens', { precision: 16, scale: 6, mode: 'number' }), | ||
| avg_cache_read_tokens: decimal('avg_cache_read_tokens', { | ||
| precision: 16, | ||
| scale: 6, | ||
| mode: 'number', | ||
| }), |
Contributor
There was a problem hiding this comment.
I'd argue all of these should be integers
Contributor
Author
There was a problem hiding this comment.
Yea, they are averages but we should just round. We don't really need this level of precision.
Contributor
Author
There was a problem hiding this comment.
All integers now with the exception of microdollar which is bigint, as otherwise the limit would be an avg cost of ~2000usd. Since that is only a factor of 100 off from results we have seen, I bumped it to be safe.
jrf0110
reviewed
May 15, 2026
Contributor
Author
|
@jrf0110 - ready for re-review I think. Thanks for the review so far. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
model-eval-ingestpuller that reads promoted evals from kilo-bench over a Service Binding, stores append-only audit rows, and recomputes publicmodelStats.benchmarks.kiloBenchcaches.Verification
swift-falcon-local-e2ebench eval, triggered the scheduled sync endpoint, and verified Postgres stored the ingest row and publickiloBenchcache.lastPromotedAt.Visual Changes
N/A
Reviewer Notes
BENCH_DASHBOARDService Binding plus the shared internal API secret; the web admin trigger usesMODEL_EVAL_INGEST_URLto call/internal/sync.model_eval_ingest.promoted_by_emailis retained for audit but anonymized by the account soft-delete flow.