Working name: Cardinal (a nod to cardinality, the central problem in time-series systems). Rename freely.
A mentor's note before the spec: the single most common way these dual-language projects fail in interviews is that the language split feels arbitrary — "I used C++ because it's fast" is not an answer. The whole design below is built so that the C++/Go boundary is load-bearing and defensible: C++ owns the bytes-and-cache-lines hot path where you control memory layout and use SIMD; Go owns the distributed, I/O-bound, operationally-complex control plane where developer velocity and concurrency ergonomics win. If you can articulate why each piece lives where it does, you've already won half the interview.
A horizontally-sharded time-series database — think a single-person, deeply-understood miniature of Prometheus/InfluxDB/VictoriaMetrics — split into two cleanly separated systems:
carde— the C++ storage and query engine. A standalone daemon that owns one or more shards. It does columnar compression (Gorilla-style), append-only ingest into in-memory head blocks, WAL-backed durability, immutable on-disk chunks viammap, background compaction/retention, and SIMD-accelerated aggregation. It exposes a narrow binary API and knows nothing about clustering.cardinal— the Go control plane. Ingest and query APIs (HTTP + gRPC), a shard router using consistent hashing with a configurable replication factor, quorum read/write logic, downsampling/retention/compaction orchestration, acardctlCLI, and full observability.
The killer detail that makes it yours: the system monitors itself with itself. Cardinal exposes Prometheus-format metrics; you scrape them into Cardinal. Dogfooding is both an elegant demo and a genuine reliability test.
L4/L5 is about production engineering judgment, not algorithmic cleverness. This project forces all of it:
- Real systems depth. Compression encoders, WAL + crash recovery,
mmaplifecycle, lock-free data structures, SIMD — none of which you can fake your way through in an interview. - Distributed-systems reasoning. Sharding, replication, quorum, consistency tradeoffs, backpressure, load shedding. You'll have a genuine CAP-flavored story.
- A defensible cross-language boundary. The Go↔C++ interface (and your decision to use a separate daemon over cgo, then optimize it with shared memory) is the staff-level conversation.
- Operability. Benchmarks with regression gates in CI, sanitizers, chaos tests, dashboards, structured logs, traces that cross the language boundary. This is the difference between "I can code" and "I can run this at 3am."
- Scoping maturity. The hard part of L5 is shipping a coherent subset of an unbounded problem. A TSDB is unbounded; your job is to draw the box well and defend what's outside it.
┌──────────────────────────────────────────┐
clients / agents │ cardinal (Go) │
(remote_write, ─────► │ │
cardctl, Grafana) │ ┌────────────┐ ┌────────────────┐ │
│ │ Ingest API │ │ Query API │ │
│ │ HTTP/gRPC │ │ HTTP/gRPC │ │
│ └─────┬──────┘ └───────┬────────┘ │
│ │ validate/batch │ plan │
│ ┌─────▼──────────────────────▼────────┐ │
│ │ Shard Router (consistent hash, │ │
│ │ replication factor R, quorum) │ │
│ └─────┬───────────────┬───────────────┘ │
│ │ fast path │ query │
│ ┌─────▼─────┐ ┌──────▼──────┐ workers: │
│ │ engine │ │ engine │ rollup, │
│ │ client #1 │ │ client #2 │ retention,│
│ └─────┬─────┘ └──────┬──────┘ compaction│
└────────┼────────────────┼───── triggers ───┘
│ UDS / shmem │ UDS
┌────────▼─────────┐ ┌─────▼────────────┐
│ carde (C++) │ │ carde (C++) │
│ shard 0,3,6… │ │ shard 1,4,7… │
│ ┌──────────────┐ │ │ │
│ │ head blocks │ │ │ (replica of │
│ │ (per series) │ │ │ shard 0 lives │
│ ├──────────────┤ │ │ here too) │
│ │ WAL │ │ │ │
│ ├──────────────┤ │ │ │
│ │ chunks(mmap) │ │ │ │
│ ├──────────────┤ │ │ │
│ │ compactor │ │ │ │
│ │ query exec │ │ │ │
│ │ (SIMD) │ │ │ │
│ └──────────────┘ │ │ │
└──────────────────┘ └──────────────────┘
Each physical node runs one Go cardinal process + one C++ carde daemon. The Go process is the only thing that talks to the network and to peers; carde only talks to its local Go process. Replication is Go-coordinated: the router writes each sample to R nodes.
The C++ engine owns everything where memory layout, cache behavior, and per-sample CPU cost dominate.
- Series model. A series = metric name + sorted label set → hashed to a 64-bit series ID. The engine indexes series ID → head block.
- Head block (write buffer). Per-series, append-only, in-memory. Stores recent samples uncompressed-ish until a chunk is cut (e.g., every 2h of data or N samples).
- Compression (the heart of it):
- Timestamps: delta-of-delta + variable-length zig-zag encoding (regular scrape intervals compress to ~1 bit).
- Float values: Gorilla XOR encoding (XOR against previous value, store leading/trailing zero counts).
- Target: ~1.3 bytes per sample, which you'll measure and report.
- WAL. Append every accepted sample to a write-ahead log before acking, so head blocks survive a crash. Segmented, with periodic checkpoints and truncation after chunk flush.
- On-disk chunks. Immutable compressed blocks,
mmap-ed for zero-copy reads. A per-shard index maps (series ID, time range) → chunk offset. - Query execution. Decode chunks for a time range, then run vectorized aggregations (sum/min/max/avg/count/rate) over decoded
float64arrays using AVX2 intrinsics (with a scalar fallback). Downsampling/bucketing happens here. - Compaction. Background merge of small chunks into larger ones; dedup overlapping samples; drop series past retention.
- Crash recovery. On boot: replay WAL into head blocks, rebuild the in-memory index from chunk headers.
What C++ explicitly does not do: networking beyond its local socket, clustering, auth, config distribution. Keep it dumb and fast.
Go owns everything I/O-bound, concurrent, and operationally complex.
- Ingest API. Accept Prometheus
remote_write(snappy-compressed protobuf) and a simple line protocol forcardctl. Validate labels, reject high-cardinality bombs, batch, push to the engine fast path. - Query API. A PromQL-lite (selectors + a handful of functions:
rate,sum,avg,max,min,histogram_quantileoptional). Parse → plan → fan out to the right shards/replicas → merge. - Shard router. Consistent hashing (hash ring with virtual nodes) maps series ID → primary + R−1 replica nodes. Owns quorum logic (
W + R > Nstyle for read-your-writes). - Replication & repair. Write to R replicas; ack on quorum. Background read-repair / anti-entropy worker reconciles divergence.
- Workers. Rollup/downsampling (raw → 5m → 1h), retention enforcement, and compaction triggering (the actual compaction runs in C++; Go decides when).
- Backpressure & load shedding. Bounded queues, semaphores, and a circuit breaker in front of each engine client. Shed writes (429) before the engine OOMs.
cardctlCLI. Query, admin, cluster status, backup/restore, hot-config reload.- Observability.
/metrics, OTel traces, structured logs (covered in §12).
Two surfaces: external (clients) and internal (Go↔C++). Keep them separate so you can evolve them independently.
service Ingest {
rpc Write(WriteRequest) returns (WriteResponse); // batched samples
}
service Query {
rpc QueryRange(RangeRequest) returns (RangeResponse); // [start,end,step]
rpc QueryInstant(InstantRequest) returns (InstantResponse);
rpc Series(SeriesRequest) returns (SeriesResponse); // label/series discovery
}
service Admin {
rpc ClusterStatus(Empty) returns (ClusterStatus);
rpc Backup(BackupRequest) returns (stream BackupChunk);
rpc Compact(CompactRequest) returns (CompactResponse);
}
message Sample { uint64 series_id = 1; int64 ts_ms = 2; double value = 3; }
message LabelPair { string name = 1; string value = 2; }
message WriteRequest { repeated TimeSeries series = 1; }
message TimeSeries { repeated LabelPair labels = 1; repeated Sample samples = 2; }Design choices to be able to defend:
- Batching is the contract. Per-sample RPCs would be murdered by overhead. The API is batch-native end to end.
- gRPC for typed internal/agent traffic; HTTP/JSON gateway for humans and Grafana. One source of truth (proto), two front doors.
remote_writecompatibility so you can point real Prometheus/Grafana Agent at it — a fantastic demo.
Phase 1 (build this first): gRPC over a Unix domain socket. Simple, observable, testable, language-agnostic. The engine is a gRPC server; Go is the client.
Phase 2 (the optimization story): a lock-free SPSC ring buffer in shared memory for the ingest fast path only. You'll benchmark Phase 1, find the serialization + syscall cost is your ingest ceiling, then bypass it for writes while keeping gRPC for queries/admin. This before/after measurement is your best interview moment — see §11.
Why a separate daemon and not cgo? Three reasons you should be ready to say out loud: (1) crash isolation — a segfault in the engine shouldn't take down your API and drop in-flight requests; (2) independent profiling/benchmarking of the engine; (3) cgo's per-call overhead and GC-interaction complexity. The tradeoff you're accepting is IPC cost, which Phase 2 addresses surgically.
Write path:
- Client → Go ingest API (
remote_write). - Go decodes, validates labels, rejects cardinality bombs, computes series IDs.
- Router determines R target nodes; batches per node.
- Local node: push batch over fast path (Phase 2: ring buffer) to
carde. cardeappends to WAL → applies to head block → acks.- Go acks the client once a quorum (W) of replicas confirm.
- Later: head block fills → cut a compressed chunk → flush to disk → truncate WAL segment.
Read path:
- Client → Go query API → parse/plan.
- Router selects shards + a quorum (R) of replicas per shard.
- Go sends planned scans to each
cardeover gRPC. cardelocates chunks (+ head block), decodes, runs SIMD aggregation, returns partial results.- Go merges partials across shards/replicas, applies final aggregation/step alignment, returns.
Background: rollup workers read raw, write downsampled series; retention workers drop expired; Go triggers C++ compaction during low load.
This is where you show you understand both runtimes' idioms.
Go side (CSP, I/O-bound):
- Goroutine per inbound connection; a bounded worker pool behind a buffered channel for ingest processing (never
go func()per sample). - Backpressure via the buffered channel depth + a
semaphore.Weightedon in-flight engine calls. context.Contextthreaded everywhere for deadline propagation and cancellation (a cancelled query must stop work in C++ too — propagate via a cancel message).- Per-replica circuit breaker so one slow node doesn't stall the pool.
errgroupfor fan-out/fan-in across shards with first-error cancellation.
C++ side (data-parallel, CPU-bound):
- Single writer thread per shard. All head-block mutation for a shard is serialized — no write-write locking, simpler reasoning, better cache locality.
- Lock-free SPSC ring buffer per shard for ingest handoff from the Go process (Phase 2). One producer (Go via shmem), one consumer (the writer thread).
- Seqlock (or RCU-style epoch) for readers of the head block: readers retry on a version mismatch instead of blocking the writer. Reads of immutable chunks need no synchronization at all (that's the point of immutability +
mmap). - A fixed thread pool for query execution and compaction, sized to cores, with compaction yielding to queries under load.
The headline concurrency story: immutability + single-writer-per-shard eliminates almost all locking on the hot path, and the only lock-free structure (the ring) is justified by a benchmark.
Be able to walk through this table cold.
- Process crash mid-write → WAL replay on restart rebuilds head blocks; ack only after WAL fsync (or grouped fsync with a documented durability window).
- WAL / chunk corruption → per-record CRC32C; on bad record, truncate WAL at the corruption point and log; chunks validated by header checksum, quarantined if bad.
- Partial / torn writes → WAL records are length-prefixed + checksummed so a torn tail is detected and discarded.
- High-cardinality explosion (OOM risk) → per-tenant active-series limits enforced in Go; reject (429) over the limit; alert. This is the #1 real-world TSDB outage cause — say so.
- Compaction stall / write amplification → compaction is rate-limited and backs off when ingest is hot; expose compaction backlog as a metric and alert on growth.
- Slow query saturating a node → query cost limits (max series, max samples scanned, deadline); kill over-budget queries; resource pool isolates query threads from ingest.
- IPC boundary failure (engine down) → circuit breaker opens, requests routed to replicas, Go surfaces a clear 503 if quorum unreachable; engine auto-restarted by supervisor.
- Replica divergence → background read-repair compares chunk digests and reconciles; document that you're eventually consistent for reads below quorum.
- Clock skew → reject samples too far in the future; document that out-of-order/old samples within a window are accepted, beyond it dropped+counted.
- Backpressure breach → bounded queues shed load deterministically (429) rather than degrading into unbounded latency.
Layered, with the riskiest C++ invariants pinned down hardest.
- C++ unit + property tests. Compression round-trip is the crown jewel:
decode(encode(xs)) == xsfor randomized timestamp/value sequences (use rapidcheck). Fuzz the decoder (libFuzzer) against arbitrary bytes — it must never crash, only error. - C++ sanitizers in CI. Build and run the suite under ASan, UBSan, and TSan. TSan on the concurrency code is non-negotiable and very interview-credible.
- Go unit tests with the race detector (
go test -race) always on. - Golden-file query tests. Fixed dataset + query → expected output checked into the repo; protects against silent aggregation regressions.
- Integration tests. Spin up
carde+cardinaltogether (testcontainers or compose), exercise full write→query round-trips over the real socket. - Chaos / recovery tests. Kill
cardemid-write; assert no acked sample is lost after WAL replay. Partition a replica; assert quorum behavior and post-heal convergence. - Cardinality & limit tests. Assert load shedding fires and the process stays alive under a cardinality bomb.
Benchmarks are the spine of the whole project's credibility. Automate them and gate CI on regressions (fail the PR if p99 query latency regresses >10%).
Metrics to publish (with a methodology section so they're trustworthy):
- Ingest throughput — samples/sec, single node and 3-node cluster.
- Query latency — p50/p99 for instant, 1h-range, and 24h-range queries.
- Compression ratio — bytes/sample (target the ~1.3B story).
- Memory per active series — the metric that actually bounds a TSDB.
- Compaction throughput and its impact on concurrent query p99.
Tooling: Google Benchmark for C++ micro-benchmarks (encoders, SIMD aggregation), Go testing.B for the API/router, a custom load generator (cardbench) for end-to-end, and flamegraphs (perf for C++, pprof for Go).
The headline experiment — your interview centerpiece: measure ingest throughput with the Phase-1 gRPC-over-UDS fast path, identify the serialization+syscall ceiling with a flamegraph, then show the Phase-2 shared-memory ring buffer result. Report the speedup and the honest cost (added complexity, single-machine-only, harder to debug). "I measured before I optimized, and I can tell you exactly what the optimization cost me" is a staff-level sentence.
Dogfood it: Cardinal monitors Cardinal.
- Metrics (RED + USE). Rate/Errors/Duration on every API; Utilization/Saturation/Errors on engine internals (head-block size, WAL fsync latency, compaction backlog, active series, ring-buffer occupancy). Expose Prometheus format from Go; the C++ engine reports its internals to Go over the admin channel.
- Tracing. OpenTelemetry spans that cross the language boundary — propagate a trace/span ID over the internal protocol so a single trace shows
ingest → router → engine write → WAL fsync. This is rare in portfolios and very impressive. - Logging. Structured JSON (zap/zerolog in Go); the C++ side emits structured lines Go can correlate by trace ID.
- Dashboards + alerts. A checked-in Grafana dashboard (JSON) and Prometheus alert rules for the failure modes in §9 (cardinality, compaction backlog, quorum loss, fsync latency).
- Exemplars linking high-latency metric points to their traces, if you want a flourish.
- Dockerfiles. Multi-stage for
carde(build toolchain → slim runtime, e.g. distroless ordebian-slim; statically link what you can). Go builds to a static binary → distroless orscratch. Keep images small and document the sizes. docker-composefor a one-command local cluster: 3× (cardinal+carde) nodes + Prometheus + Grafana +cardbenchload generator. This is your demo.- Kubernetes (optional flourish). A small Helm chart: StatefulSet for the nodes (stable identity + PVCs for WAL/chunks), Services, a ServiceMonitor. Mention but don't over-invest.
- CI (GitHub Actions). Matrix: C++ build + ASan/UBSan/TSan + Google Benchmark; Go build +
-race+ lint (golangci-lint) +testing.B; integration tests on compose; benchmark regression gate on PRs; container build + Trivy scan; release job publishing images +cardctlbinaries. - Config. Single declarative config (YAML), hot-reloadable via
cardctl, documented schema.
# Cardinal
> A sharded time-series database: C++ storage engine + Go control plane.
[badges: CI · sanitizers · coverage · benchmark trend · license]
## What it is (2–3 sentences) + an animated GIF of the Grafana dashboard
## Highlights
- ~1.3 bytes/sample (Gorilla compression), N samples/sec ingest, p99 query <Xms
- Lock-free shared-memory fast path (Yx over gRPC baseline — link to benchmark)
- Crash-safe (WAL), horizontally sharded, quorum replicated
- Traces span the Go↔C++ boundary
## Architecture (the diagram from §3)
## Why C++ and Go (the language-boundary justification — recruiters read this)
## Quickstart
docker compose up # 3-node cluster + Grafana + load generator
cardctl query 'rate(http_requests_total[5m])'
## Design deep-dives (link to /docs)
- Storage engine & compression
- The Go↔C++ boundary: gRPC → shared memory (with benchmarks)
- Sharding, replication & quorum
- Failure modes & recovery
## Benchmarks (table + flamegraphs + methodology)
## Testing & reliability (sanitizers, fuzzing, chaos)
## Roadmap / Non-goals (shows scoping judgment)
## License
Put the Why C++ and Go and Benchmarks sections high — they're what a senior reviewer skims for.
Tune the numbers to what you actually measure; don't ship a number you can't defend.
- Built Cardinal, a horizontally-sharded time-series database (C++ storage engine + Go control plane) sustaining ~N M samples/sec ingest at ~1.3 bytes/sample via Gorilla-style delta-of-delta and XOR compression.
- Designed a lock-free shared-memory ring buffer for the Go↔C++ ingest path after profiling identified gRPC serialization as the bottleneck, improving ingest throughput Y× while preserving crash isolation between processes.
- Implemented WAL-backed crash recovery, immutable
mmap-ed chunks, and single-writer-per-shard concurrency with seqlock reads, validated under ASan/UBSan/TSan, libFuzzer, and chaos tests that kill the engine mid-write with zero acked-sample loss. - Built consistent-hash sharding with quorum replication and read-repair in Go, plus backpressure/load-shedding that keeps nodes alive under cardinality-bomb load.
- Shipped full production tooling: OpenTelemetry traces that span the language boundary, Grafana dashboards, a benchmark-regression CI gate, and a one-command 3-node Docker Compose cluster.
"Cardinal is a sharded time-series database I built to deeply understand the storage-engine-plus-distributed-control-plane pattern. I split it across two languages on purpose. The C++ engine owns the hot path — it does Gorilla-style compression down to about 1.3 bytes per sample, keeps recent data in per-series head blocks backed by a WAL for crash safety, flushes immutable
mmap-ed chunks, and runs SIMD aggregations on queries. I made the concurrency cheap by using one writer thread per shard and immutable chunks, so reads need almost no locking.Everything distributed and I/O-bound lives in Go: the ingest and query APIs, consistent-hash sharding with quorum replication, backpressure, and observability. I deliberately ran the engine as a separate daemon instead of using cgo — I wanted crash isolation and independent profiling, and I was willing to pay the IPC cost. Then I measured that cost: gRPC serialization was capping ingest, so I built a lock-free shared-memory ring buffer for the write path and got a Y× improvement, while keeping gRPC for queries.
The part I'm proudest of operationally is that it monitors itself — and I can walk you through the failure modes I designed for, like cardinality explosions and compaction stalls, and how the tests prove recovery actually works."
Then let them pull any thread — compression, the ring buffer, quorum, a specific failure mode. Every one has depth behind it.
Each phase ends in something demoable. Resist building ahead — depth per phase beats breadth.
Implementation status in this repo: Phases 0–6 plus the shared-memory ingest fast path of Phase 8 are built and tested — the Gorilla engine (property + fuzz tested), WAL crash recovery, the binary protocol over UDS, the HTTP API with aggregations, concurrency hardening (TSan +
-raceclean, backpressure, cardinality limits), consistent-hash sharding with R-replica writes, write quorum, tunable read consistency, and coordinator-side read-repair, and a lock-free SPSC shared-memory ring for ingest with a measured socket-vs-shm benchmark (≈1.9× a batched socket, ≈270× unbatched, on one core). Deviation from the plan: the Phase-1 boundary is a hand-rolled framed binary protocol rather than gRPC (gRPC-C++ is a heavy build dependency; the binary protocol is lighter and made the shared-memory swap cleaner). Phase 8 is also done: AVX2-vectorized aggregation with a scalar fallback and runtime dispatch (5–11× on cache-resident reductions). Remaining next steps: OTel traces across the boundary, compaction/rollups, and wiring the shm fast path into the cluster write path with a committed-index durability ack.
Phase 0 — Skeleton (½ week). Two repos/dirs, build systems (CMake + Google Test for C++, Go modules), CI that builds and runs empty test suites, the proto definitions. Demo: green CI.
Phase 1 — Single-series engine (1–1.5 wks). In C++: head block, the Gorilla encoder/decoder with property + fuzz tests, in-memory query of a single series. No persistence yet. Demo: round-trip benchmark + compression ratio number.
Phase 2 — Durability (1 wk). WAL with CRC + crash recovery; flush head blocks to mmap-ed chunks; chunk index. Demo: kill -9 mid-write, restart, data intact.
Phase 3 — The boundary, v1 (1 wk). carde as a gRPC server over UDS; minimal Go service that writes and queries one node. Demo: cardctl write/cardctl query end to end.
Phase 4 — Real APIs (1 wk). Prometheus remote_write ingest, PromQL-lite query with rate/sum/avg, HTTP gateway, label validation + cardinality limits. Demo: point Grafana at it, see a graph.
Phase 5 — Concurrency hardening (1 wk). Single-writer-per-shard, seqlock reads, Go worker pools + backpressure + circuit breaker, context-cancellation across the boundary. Turn on TSan and -race, fix everything. Demo: load test stays stable; sanitizers clean.
Phase 6 — Sharding & replication (1.5–2 wks). Consistent hashing, R-replica writes, quorum, read-repair worker. Run a 3-node compose cluster. Demo: kill a node, queries still work; node returns, converges.
Phase 7 — Observability + compaction (1 wk). Self-metrics, OTel traces across the boundary, Grafana dashboard, alerts; background compaction + retention + rollup workers. Demo: a single trace from API to WAL fsync; dashboard of Cardinal monitoring itself.
Phase 8 — SIMD + the shared-memory fast path (1–1.5 wks). AVX2 aggregation with scalar fallback; the lock-free SPSC ring buffer; the before/after benchmark writeup. Demo: the speedup table + flamegraphs in the README.
Phase 9 — Polish (1 wk). README with diagram and benchmarks, design-doc /docs, distroless images, Trivy in CI, the benchmark-regression gate, cardctl backup/restore. Demo: docker compose up → working cluster + dashboard in one command.
Total: roughly 10–12 focused weeks solo. If you need to cut, drop K8s/Helm and PromQL breadth first — keep compression, WAL recovery, the shared-memory fast path, sharding/quorum, and the cross-boundary traces. Those five are the depth a staff interviewer will actually probe.
- The shared-memory ring buffer + lifetime/synchronization across two runtimes — by far the gnarliest code. That's exactly why it's the best story; budget for it.
- WAL recovery correctness under torn writes — easy to get 90% right and have a lurking data-loss bug. The chaos tests are how you prove it.
- Quorum + read-repair semantics — easy to hand-wave, hard to make actually converge. Write the consistency model down explicitly and test it.
Everything else is "a lot of careful work," not conceptually treacherous.