docs(bench): TTI create-path ladder, one rung per deployed change#902
Open
stubbi wants to merge 1 commit into
Open
docs(bench): TTI create-path ladder, one rung per deployed change#902stubbi wants to merge 1 commit into
stubbi wants to merge 1 commit into
Conversation
…analysis Rung B (v1.40.0) is flat client-side because the #895 gateway cuts and a +20 ms activate_rpc regression riding the same release cancelled; the stage-log evidence is on #892. 4/100 client read timeouts are server-side invisible (#901). Rung C (the pre-claimed checkout roll) lands in this file next. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Jannes Stubbemann <jannes@openclaw.rocks>
📝 WalkthroughWalkthroughAdded a 2026-07-12 benchmark report for the “TTI create-path ladder,” covering measurement methodology, rung results, analyses of B and B’, the intended rung C test, and peer-table targets. ChangesTTI create-path ladder
Estimated code review effort: 1 (Trivial) | ~2 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@bench/results/2026-07-12-tti-create-path-ladder.md`:
- Around line 6-7: Update the rung-isolation description in the benchmark
document to distinguish release rungs, which follow one deployed change, from
repeat measurements such as B′ on the same v1.40.0 deployment. Preserve the
attribution goal while removing the claim that every rung represents a new
deployment.
- Around line 20-21: Update the report text around the reproducibility note to
include the exact harness invocation, or replace “the command above” with an
explicit reference to the command’s documented location. Ensure readers can
determine how to reproduce the per-iteration outputs without relying on an
absent preceding command.
- Around line 13-18: Add millisecond units to the latency column headers in the
benchmark table, including TTI P50, create med, and first exec med. Preserve the
existing measurements and table structure.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: a1b6e019-c72f-414c-93a6-7a28dbc96748
📒 Files selected for processing (1)
bench/results/2026-07-12-tti-create-path-ladder.md
Comment on lines
+6
to
+7
| probe, 13 s pacing. Each rung re-measures prod AFTER exactly one deployed | ||
| change, so effects attribute to a release instead of to drift. Long runs |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Clarify the rung-isolation claim.
Line 6-7 says every rung follows exactly one deployed change, but rung B′ is a higher-sample rerun of the same v1.40.0 deployment. Reword this to distinguish release rungs from repeat measurements.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@bench/results/2026-07-12-tti-create-path-ladder.md` around lines 6 - 7,
Update the rung-isolation description in the benchmark document to distinguish
release rungs, which follow one deployed change, from repeat measurements such
as B′ on the same v1.40.0 deployment. Preserve the attribution goal while
removing the claim that every rung represents a new deployment.
Comment on lines
+13
to
+18
| | rung | prod version | change under test | N | TTI P50 | create med | first exec med | success | | ||
| |---|---|---|---|---|---|---|---| | ||
| | A | v1.39.1 | baseline (2026-07-11) | 20 | 327.2 | 189.3 | 136.8 | 20/20 | | ||
| | B | v1.40.0 | gateway round-trip cuts (#895) | 20 | 349.0 | 191.4 | 166.4 | 20/20 | | ||
| | B' | v1.40.0 | same, higher power | 100 | 349.9 | 190.9 | 163.1 | 96/100 | | ||
| | C | v1.41.0 + checkout | pre-claimed checkout (#896) | | pending | | | | |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Add units to the latency columns.
Line 13-18 lists numeric latency values without units. Label the relevant columns as milliseconds (for example, TTI P50 (ms) and create med (ms)) or state the units immediately above the table.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@bench/results/2026-07-12-tti-create-path-ladder.md` around lines 13 - 18, Add
millisecond units to the latency column headers in the benchmark table,
including TTI P50, create med, and first exec med. Preserve the existing
measurements and table structure.
Comment on lines
+20
to
+21
| Full per-iteration outputs for each rung are reproducible with the command | ||
| above; the harness prints them and this table quotes its summary lines. |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Include the reproducibility command.
Line 20-21 refers to “the command above,” but no command appears above in this report. Add the exact harness invocation or change the text to point explicitly to where that command is documented.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@bench/results/2026-07-12-tti-create-path-ladder.md` around lines 20 - 21,
Update the report text around the reproducibility note to include the exact
harness invocation, or replace “the command above” with an explicit reference to
the command’s documented location. Ensure readers can determine how to reproduce
the per-iteration outputs without relying on an absent preceding command.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thinking Path
Linked Issues or Issue Description
Related: #871 (TTI gap tracking), #892 (activate_rpc observation posted there), #901 (the observability gap rung B' surfaced), #896 (rung C's change).
What Changed
Verification
Risks
Model Used
Checklist
🤖 Generated with Claude Code
Summary by CodeRabbit