v0.1.4: Native gRPC transport + Reddit display fix#3
Open
Conversation
The Macrocosmos API uses gRPC-Web JSON transcoding over HTTP/2, which goes through an unstable ALB layer that frequently returns 500 errors and timeouts. The Python SDK uses native gRPC and is unaffected. This migration replaces reqwest HTTP calls with tonic native gRPC: - Add vendored proto files for sn13.v1 and gravity.v1 - Add build.rs for tonic-build proto compilation - Rewrite ApiClient to use tonic channels with auth interceptor - Convert prost_types::Struct responses to serde_json::Value - Map gRPC status codes to user-friendly error messages - All command/display/config layers unchanged Tested: search x, search reddit, gravity status, dry-run all work. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Code review fixes from 3 parallel review agents: - Extract nonempty() helper — replaces 11 inline is_empty checks - Extract to_grpc_notifications() — deduplicates notification mapping - Restore SN13_SERVICE/GRAVITY_SERVICE/CLIENT_ID constants - Pre-compute Bearer header in AuthInterceptor (was formatting on every RPC) - Convert struct_to_json to use owned values (into_iter, no key cloning) - Replace 6 per-post is_reddit() calls with single PostFields::extract() - Fix truncate() double allocation for short strings - Guard negative timestamps in timestamp_to_string - Remove thinking-aloud comments Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Gravity's gRPC endpoint is broken server-side — it sends JSON over
the binary gRPC channel (compression flag 123 = '{' ASCII). SN13
search works perfectly via native gRPC.
This simplifies the client to use each transport where it works:
- SN13 (search): native gRPC via tonic (reliable, fast)
- Gravity: HTTP/JSON via reqwest (their gRPC is broken)
Removes ~400 lines of gravity proto conversion code (crawler_to_json,
proto_dataset_to_dataset_info, timestamp helpers, etc.) since HTTP
responses are already JSON and deserialize directly into our types.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
cargo installcommand in README —dataverse→dataverse-cliWhy native gRPC?
The Macrocosmos API is a gRPC service. The CLI was hitting it through an HTTP/JSON transcoding layer (ALB) that frequently goes down — the Python SDK uses native gRPC and is unaffected during these outages. This migration puts the CLI on the same transport path as the SDK.
During a 2+ hour outage on March 22, the HTTP endpoint returned 500s and timeouts while native gRPC worked the entire time.
Changes
src/api/client.rs— Rewritten to use tonic native gRPC with auth interceptorbuild.rs— New: compiles vendored proto files via tonic-buildproto/sn13/v1/sn13_validator.proto— Vendored SN13 service definitionproto/gravity/v1/gravity.proto— Vendored Gravity service definitionsrc/display/mod.rs— Reddit display fix (source-aware field mapping)Cargo.toml— Replace reqwest with tonic/prost, add tonic-build, bump to 0.1.4README.md— Fix install commandTest plan
dv status— connects via gRPCdv search x -k bitcoin -l 5— X keyword searchdv search x -u @karpathy --from 2026-03-01 -l 50— X username searchdv search reddit -k "machine learning" -l 5— Reddit table displaydv search reddit -k politics -l 3 -o csv— Reddit CSV outputdv --dry-run search x -k test— dry-run modedv gravity status— gravity task listingcargo build --release— release buildcargo test— all tests pass🤖 Generated with Claude Code