M10: query remote traces through Athena#16
Merged
Merged
Conversation
Remove the remote MCP dependency on a generated trace projection by cataloging the existing immutable event chunks with Glue partition projection and folding bounded SELECT results through the existing trace model and read-scope policy. Live sie-gw-cfg validation returned four matching tool events and opened one as a paired 193 ms span. Kind pushdown reduced the match selection from 184 rows to 14, and metadata deduplication reduced its context selection to 2 rows. The 4.1 GiB legacy trace.json was removed from the PVC; authenticated search still succeeded afterward. Keep every request within 7 days, 50,000 events, 64 MiB of returned envelopes, 50 seconds per SELECT, and a 20 GiB workgroup scan cutoff. The workload role retains only S3 Get/List plus workgroup-scoped Athena and Glue read-query actions.
📝 WalkthroughWalkthroughAdds optional Athena-backed raw trace queries for MCP, including bounded SQL execution, Glue/Athena deployment resources, selective read-model downloads, readiness tracking, Helm configuration, and Athena-enabled CI, release builds, Docker builds, and documentation. ChangesAthena-backed MCP trace support
Sequence Diagram(s)sequenceDiagram
participant CLI as synty MCP CLI
participant Sync as Read-model sync
participant MCP as MCP Server
participant Athena as Athena Backend
participant AWS as Athena and Glue
CLI->>Sync: Select read-model files
CLI->>MCP: Start with Athena options
MCP->>Athena: Route trace tool request
Athena->>AWS: Execute bounded SELECT
AWS-->>Athena: Return event rows
Athena-->>MCP: Render trace result
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
Helm renders the workgroup as a quoted shell argument. Assert the exact safe rendering so the deploy job verifies the intended value instead of failing on the missing unquoted form.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
src/trace_athena.rs (1)
643-653: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winSkip unparseable rows instead of failing the whole query.
event_sessionspropagates an error on the first line that fails strictEventdeserialization, yet the downstream reconstruction (TraceStore::from_text→fold_text) tolerates malformed lines. A single non-conforming envelope in the immutable lake would abort an entire trace list/show/search here, before reconstruction runs. Consider skipping rows that don't parse so session discovery matches the tolerant fold.♻️ Tolerate malformed rows
fn event_sessions(lines: &[String]) -> Result<BTreeSet<String>> { let mut sessions = BTreeSet::new(); for line in lines { - let event: crate::event::Event = - serde_json::from_str(line).context("parse Athena event row")?; - if !event.session_id.is_empty() { - sessions.insert(event.session_id); - } + if let Ok(event) = serde_json::from_str::<crate::event::Event>(line) { + if !event.session_id.is_empty() { + sessions.insert(event.session_id); + } + } } Ok(sessions) }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/trace_athena.rs` around lines 643 - 653, Update event_sessions to skip rows that fail serde_json deserialization instead of propagating the parse error; continue collecting non-empty session_id values from successfully parsed Event records and return the collected sessions so malformed Athena rows do not abort discovery.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/mcp_http.rs`:
- Around line 219-234: Validate the Athena configuration at the start of run()
before starting refreshes or dispatchers: when --athena-workgroup is supplied in
a build without the athena feature, return an error immediately instead of
constructing an unavailable trace backend. Add a scenario test covering this
no-athena configuration and asserting that startup fails.
In `@src/readmodel.rs`:
- Around line 127-132: Add scenario-style unit tests for mediated_ready covering
both trace modes: with a current format-2 build containing analysis.json but no
trace.json, assert readiness when require_trace is false and unready when
require_trace is true. Use the existing readmodel test setup and
fixtures/helpers, preserving the implementation in mediated_ready.
---
Nitpick comments:
In `@src/trace_athena.rs`:
- Around line 643-653: Update event_sessions to skip rows that fail serde_json
deserialization instead of propagating the parse error; continue collecting
non-empty session_id values from successfully parsed Event records and return
the collected sessions so malformed Athena rows do not abort discovery.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 5236b8fe-4530-493d-bb2d-e2b5c3ac3bd5
⛔ Files ignored due to path filters (1)
Cargo.lockis excluded by!**/*.lock
📒 Files selected for processing (19)
.github/workflows/ci.yml.github/workflows/release.ymlAGENTS.mdCargo.tomlDockerfileREADME.mddeploy/aws/athena-trace.yamldeploy/aws/mcp-reader.yamldeploy/helm/synty/templates/mcp.yamldeploy/helm/synty/values-sie-gw-cfg.yamldeploy/helm/synty/values.yamldocs/design.mdsrc/main.rssrc/mcp.rssrc/mcp_http.rssrc/readmodel.rssrc/sync.rssrc/trace.rssrc/trace_athena.rs
Reject configured Athena mode before HTTP startup when its backend or required bucket is unavailable. Keep remote readiness independent of the retired trace snapshot, and tolerate malformed raw rows while discovering otherwise valid sessions. Adds three scenarios; the full feature suite passes 298 tests.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/readmodel.rs`:
- Around line 127-132: Align the feature gating for mediated_ready and
mediated_ready_from: either apply #[cfg(feature = "mcp-http")] to the public
mediated_ready wrapper or remove the gate from mediated_ready_from. Ensure
non-mcp-http builds do not reference an unavailable helper.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: ad5fb335-41b5-4f1c-aaff-691acc3a90f2
📒 Files selected for processing (3)
src/mcp_http.rssrc/readmodel.rssrc/trace_athena.rs
🚧 Files skipped from review as they are similar to previous changes (2)
- src/trace_athena.rs
- src/mcp_http.rs
Contributor
|
btw just to make sure you are aware @huronat this is a public repo |
svonava
approved these changes
Jul 24, 2026
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
events/<stream>/chunks/track.<day>/JSONL layout. Partition projectiondiscovers stream/day paths without a crawler or data migration.
synty_trace_*. It issues onlygenerated
SELECTstatements, folds bounded raw envelopes through theexisting Rust trace model, and reuses the existing role/scope/redaction
enforcement before rendering.
pull the semantic index and compact analysis projection but deliberately omit
trace.json.fail-closed AWS endpoint egress, release/CI feature coverage, readiness
checks, and the architecture diagram in
docs/design.md.Safety and bounds
workgroup-scoped Athena query actions, and Glue GetDatabase/GetTable/
GetPartitions.
scan cutoff.
returned envelopes, 200 sessions, 50 seconds, and bounded AWS SDK calls.
kind;
show/compareexpand only resolved sessions. Partial search/liststores cannot satisfy a later
show.Live validation
Deployed to
sie-gw-cfg, namespacesynty, as Helm revision 17:851725219920.dkr.ecr.eu-central-1.amazonaws.com/synty:athena-trace-20260723-8/ready: semantic read model, mediated analysis, and both dispatchers ready;trace_backend=athena.synty_trace_search("libxcb.so.1")returned four realtool_callevents.synty_trace_showopened the first hit as a paired 193 msexecspan withbounded surrounding evidence.
deduplicated session metadata returned 2 rows.
trace.jsonfrom the PVC. A fresh authenticatedsearch still succeeded afterward, and no
trace.jsonexists on the volume.inspected and contains no S3 Put/Delete action.
Verification
cargo test --locked— 285 passedcargo test --locked --no-default-features— 264 passedcargo test --locked --features s3,gcs,mcp-http,athena— 298 passedcargo clippy --locked --all-targets --features s3,gcs,mcp-http,athena(no branch-introduced warnings)
scripts/mcp-http-smoke.sh target/debug/syntyserver-side dry-run
Scaling note
This is deliberately the no-migration bootstrap. Athena still scans the
selected raw JSONL objects; the current broad live selection scanned about
15.3 GB. Before daily volume reaches the workgroup cutoff, a follow-up should
write a separate immutable Parquet projection partitioned for session lookup.
That derived prefix can improve scan cost without changing or deleting the
authoritative raw events.