Skip to content

feat: add cfetch and vmpush packages for converge integration #26

Draft
davidbirdsong wants to merge 2 commits into
developfrom
davidbirdsong/net-146-switch-cloudflare-exporter-from-pull-based-scraping-to-push-2
Draft

feat: add cfetch and vmpush packages for converge integration #26
davidbirdsong wants to merge 2 commits into
developfrom
davidbirdsong/net-146-switch-cloudflare-exporter-from-pull-based-scraping-to-push-2

Conversation

@davidbirdsong

Copy link
Copy Markdown

Summary

Adds two packages that bridge the converge engine to Cloudflare (input) and VictoriaMetrics (output), completing the data pipeline. Both are self-contained behind package boundaries with no dependency on root package globals.

Cloudflare GraphQL ──► cfetch/ ──► converge/ ──► vmpush/ ──► VictoriaMetrics
                      (Fetcher)    (Engine)      (Sink)

cfetch/ — Cloudflare Fetcher

Implements converge.Fetcher by querying httpRequests1mGroups over a time range.

  • Owns its own GraphQL query (range-based datetime_geq/datetime_lt, not the single-bucket datetime: used by the existing scrape path)
  • Owns its own response structs (no import of root package types)
  • Takes a GQLClient interface at construction, not a global. Root package wraps its *GraphQL in a thin adapter
  • Takes []Zone (simple struct with ID/Name/Free) instead of importing cfzones
  • Chunks zones into batches of 10 (CF limit), skips failing chunks
  • Filters free-plan zones (no access to httpRequests1mGroups)
  • Flattens response into cfp_zone_* observations (gauge semantics, not counters)
  • Metrics produced: requests, bandwidth, cached, encrypted, pageviews, threats, uniques, plus breakdowns by country, status code, content type, browser, threat type

vmpush/ — VictoriaMetrics Sink

Implements converge.Sink using Prometheus remote write protocol.

  • Protobuf serialization via prompb.WriteRequest
  • Snappy compression
  • Basic Auth (per o11y team's VmUser provisioning)
  • Parses pre-serialized Sample.Key (e.g. cfp_zone_requests_total{zone="example.com"}) back into __name__ + label pairs for protobuf encoding
  • Connection pooling via http.Transport
  • Takes Config{Endpoint, Username, Password} at construction, no globals

Integration (not in this PR)

// Thin adapter for the GQL client
type gqlAdapter struct{ g *GraphQL }
func (a *gqlAdapter) RunGQL(ctx context.Context, req *cfetch.GQLRequest, dest any) error {
    r := NewGraphQLRequest(req.Query)
    for k, v := range req.Variables { r.Var(k, v) }
    return a.g.Run(ctx, r, dest)
}

// Wire up
fetcher := cfetch.New(&gqlAdapter{gql}, zones)
sink := vmpush.New(vmpush.Config{Endpoint: vmURL, Username: u, Password: p})
go converge.Run(ctx, cfg, fetcher, sink)

Test plan

  • parseKey handles single label, multiple labels, no labels, empty braces
  • go build ./... compiles all packages
  • End-to-end validation against staging VM (pending o11y team credentials)

- make TTL semantics are consistent and the engine stays purely input-driven.
@davidbirdsong

davidbirdsong commented Jun 5, 2026

Copy link
Copy Markdown
Author

This change is part of the following stack:

Change managed by git-spice.

@davidbirdsong davidbirdsong force-pushed the davidbirdsong/net-146-switch-cloudflare-exporter-from-pull-based-scraping-to-push-2 branch from b44f81a to 0953909 Compare June 5, 2026 23:57
@davidbirdsong davidbirdsong marked this pull request as draft June 8, 2026 23:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant