Skip to content

feat(routing): emit BackendTrafficPolicy when routing.streaming is true#123

Closed
viniciusdc wants to merge 1 commit into
docs/streaming-timeouts-designfrom
feat/streaming-timeouts
Closed

feat(routing): emit BackendTrafficPolicy when routing.streaming is true#123
viniciusdc wants to merge 1 commit into
docs/streaming-timeouts-designfrom
feat/streaming-timeouts

Conversation

@viniciusdc

Copy link
Copy Markdown
Collaborator

Summary

Implements the design in #122 (stacked on top of that PR — base will switch to main once #122 merges). Independent of PR #120 / #121 (multi-port routes); the two feature pairs are orthogonal and can land in either order.

  • Adds RoutingConfig.Streaming bool (default false). When true, the operator emits a single Envoy Gateway BackendTrafficPolicy named <app>-streaming in the NebariApp's namespace, owner-referenced for GC.
  • Policy targets every operator-owned HTTPRoute (main always, public when publicRoutes is set), so SSE / long-poll / gRPC streams aren't cut by Envoy's 15s default requestTimeout. Fixed canned values: requestTimeout: 0s, connectionIdleTimeout: 300s.
  • New StreamingReconciler in internal/controller/reconcilers/routing/streaming.go, paralleling the existing routing / TLS / auth reconcilers. Wired into the main controller after the public-route reconcile (so the policy's targetRefs resolve).
  • StreamingReady condition surfaces state without blocking the rest of the reconcile:
    • True (PolicyReconciled) — policy is in sync.
    • False (ForeignPolicyExists) — a BackendTrafficPolicy already exists under our chosen name without our owner ref. We refuse to overwrite.
    • False (CRDMissing) — Envoy Gateway BackendTrafficPolicy CRD not installed in the cluster. Rest of reconcile continues; only the streaming guarantee is degraded.
    • False (ReconcileFailed) — apiserver error.
    • Condition removed entirely when streaming is unset/false (no noise on every NebariApp).

Files touched

  • api/v1/nebariapp_types.go — add RoutingConfig.Streaming, ConditionTypeStreamingReady, four EventReasonBackendTrafficPolicy* constants.
  • internal/controller/reconcilers/routing/streaming.go — new reconciler (253 lines).
  • internal/controller/reconcilers/routing/streaming_test.go — 5 test cases covering disabled / enabled / public-routes / transition-to-disabled / foreign-policy.
  • internal/controller/nebariapp_controller.go — wire StreamingReconciler, add reconcileStreaming helper to keep the main Reconcile under gocyclo's 30 ceiling, add kubebuilder RBAC marker for backendtrafficpolicies.
  • internal/controller/utils/conditions/conditions.go — new RemoveCondition helper (used when streaming flips back to disabled).
  • internal/controller/utils/constants/constants.goBackendTrafficPolicySuffix = \"streaming\".
  • internal/controller/utils/naming/naming.goBackendTrafficPolicyName(...), added to ValidateResourceNames.
  • cmd/operator/main.go — instantiate the new reconciler and wire it into NebariAppReconciler.
  • Regenerated: config/crd/bases/reconcilers.nebari.dev_nebariapps.yaml, config/rbac/role.yaml, docs/api-reference.md.

Why a boolean, not Envoy timeout knobs

Documented at length in #122. Short version: contract independence (CRD shouldn't pin to Envoy-typed fields), canned values match every streaming workload anyone has asked for, and widening to a streamingTimeouts: { requestTimeout, idleTimeout } struct later is additive.

Backwards compatibility

  • Existing NebariApp manifests are unaffected. routing.streaming is additive, optional, default false.
  • Apps that hand-rolled their own BackendTrafficPolicy (e.g. `openteams-ai/nebari.openteams.ai#12`) migrate by setting routing.streaming: true and deleting their hand-rolled YAML. If they hit the operator's chosen name (<app>-streaming) before deleting, the reconciler refuses to overwrite and reports ForeignPolicyExists.

Test plan

  • go build ./... passes
  • go vet ./... clean
  • make lint (golangci-lint) — 0 issues (after extracting reconcileStreaming helper to keep main Reconcile under the gocyclo ceiling)
  • make generate-dev regenerates CRD, RBAC, and deepcopy cleanly
  • make docs regenerates api-reference.md cleanly
  • Unit tests pass: TestStreamingReconciler_{DisabledCreatesNoPolicy,EnabledCreatesPolicy,EnabledWithPublicRoutesTargetsBoth,TransitionToDisabledDeletesPolicy,RefusesForeignPolicy} — all green
  • TestControllers (envtest-based) — pre-existing failure on main (should successfully reconcile the resource), reproduces with no code changes. Not introduced by this PR.
  • e2e smoke against a real cluster with Envoy Gateway installed — recommend before flipping out of draft
  • e2e behavior verification: confirm an SSE backend stays connected past 15s when streaming: true, and gets cut at 15s when streaming: false

Related

Adds an opt-in routing.streaming field on RoutingConfig and a new
StreamingReconciler that emits a single Envoy Gateway
BackendTrafficPolicy targeting every operator-owned HTTPRoute for the
NebariApp (main route always, public route when publicRoutes is set).
The policy disables Envoy's default 15s HTTP request timeout and sets
the connection idle timeout to 5 minutes — fixed canned values rather
than exposing Envoy-typed knobs on the CRD.

When routing.streaming is false or absent, the reconciler deletes any
policy it previously owned and clears the StreamingReady condition.

Foreign-policy detection: if a BackendTrafficPolicy already exists in
the namespace under the operator's chosen name without our owner
reference, the reconciler refuses to overwrite it and surfaces
StreamingReady=False (ForeignPolicyExists) instead.

Failure modes (CRD missing, foreign policy, apiserver errors) degrade
gracefully via StreamingReady=False without blocking the rest of the
reconcile.

Adds RBAC for gateway.envoyproxy.io/backendtrafficpolicies, wires the
new reconciler into the main controller, and adds unit tests covering
the create / update / delete / public-routes / foreign cases.

Design rationale: docs/design/streaming-timeouts.md.
@viniciusdc

Copy link
Copy Markdown
Collaborator Author

Closing — design needs more thought. Branch feat/streaming-timeouts left in place; can be reopened when the design firms up.

@viniciusdc viniciusdc closed this May 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants