feat: datacap direct mode (in-process sidecar replacement)#168
Open
myleshorton wants to merge 3 commits intomainfrom
Open
feat: datacap direct mode (in-process sidecar replacement)#168myleshorton wants to merge 3 commits intomainfrom
myleshorton wants to merge 3 commits intomainfrom
Conversation
Replace the Docker sidecar with an in-process DeviceUsageStore that handles device state caching, batch usage uploads, and throttle decisions directly via gRPC. Both sidecar (--datacap-url) and direct (--datacap-grpc-api) modes coexist; the sidecar path is unchanged. Key changes: - DataCapAPI / ReportSink interfaces decouple connection tracking from backend - DeviceUsageStore with singleflight-deduplicated state sync - reportingCore extracted from conn.go / packet_conn.go (~150 lines removed) - mTLS client credentials for gRPC auth (CA-verified server cert) - Local proto types in dcpb/ to avoid lantern-cloud dependency - Multi-region OCI packer builds (iad, fra, nrt, sin) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds a “direct mode” datacap reporting path that runs aggregation + batch upload in-process (via gRPC to the cloud API), while preserving the existing “sidecar over HTTP” legacy mode. It also expands OCI image builds to multiple regions.
Changes:
- Refactor datacap connection reporting to use a
ReportSinkabstraction shared by HTTP sidecar and the new in-process store. - Add
DeviceUsageStore(direct mode) with background batching, sync, and a gRPC-backedDataCapAPIimplementation. - Update CLI flags and image build automation to support direct-mode configuration and multi-region OCI images.
Reviewed changes
Copilot reviewed 16 out of 16 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| tracker/datacap/tracker.go | Tracker now targets a ReportSink and adds direct-mode lifecycle wiring. |
| tracker/datacap/reporting.go | New shared reporting/throttling core used by both Conn and PacketConn. |
| tracker/datacap/conn.go | Conn refactored to embed reportingCore and report via ReportSink. |
| tracker/datacap/packet_conn.go | PacketConn refactored similarly to Conn using reportingCore. |
| tracker/datacap/api.go | Introduces ReportSink and DataCapAPI abstractions + DTOs for batching/sync. |
| tracker/datacap/client.go | HTTP client now implements ReportSink via Report(). |
| tracker/datacap/device_store.go | New in-process store implementing ReportSink, batching uploads + syncing state. |
| tracker/datacap/device_store_test.go | Unit tests for store batching, sync behavior, concurrency, and shutdown. |
| tracker/datacap/integration_test.go | Updates tests to use Sink instead of Client in configs. |
| tracker/datacap/dcpb/datacap.proto | New proto mirroring lantern-cloud DataCapService for isolated dependency. |
| tracker/datacap/dcpb/datacap.pb.go | Generated protobuf types for datacap service. |
| tracker/datacap/dcpb/datacap_grpc.pb.go | Generated gRPC client/server interfaces for datacap service. |
| cmd/grpc_datacap.go | gRPC DataCapAPI implementation + TLS/mTLS connection helper. |
| cmd/cmd_run.go | Adds direct-mode CLI flags and wiring; enforces mode exclusivity. |
| deploy/packer/lantern-box.pkr.hcl | Splits OCI builds into per-region sources with region-specific subnet/AD vars. |
| .github/workflows/build-images.yaml | Updates workflow inputs, OCI env vars, manifest parsing, and Vault updates per region. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
- Make Stop() idempotent with sync.Once (prevents double-close panic) - Wait for background loop to exit before final upload (prevents race) - Use caller's ctx deadline for shutdown, fresh context for final upload - Check all flag parsing errors instead of discarding with _ - Validate all mTLS paths are non-empty before attempting to load certs Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Take the reviewed fixes from main: legacy env var fallbacks, oracle-oci builder expansion, Vault secret merging, and missing-region warnings. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
DeviceUsageStorefor VPS deploymentsDataCapAPI/ReportSinkinterfaces to decouple connection tracking from backend (HTTP sidecar vs gRPC direct)reportingCorefromconn.go/packet_conn.goeliminating ~150 lines of duplicationdcpb/mirror lantern-cloud's DataCapService to avoid cross-repo dependencyBoth modes coexist:
--datacap-url(legacy sidecar) and--datacap-grpc-api(new direct) are mutually exclusive flags.Companion PR: lantern-cloud feat/datacap-direct-mode (mTLS listener + VPS cert provisioning)
Test plan
go build ./...passesgo test -race ./tracker/datacap/...passes (10 new tests for DeviceUsageStore)--datacap-url) still works unchanged🤖 Generated with Claude Code