Skip to content

Feat: make reconcile timeout configurable via --reconcile-timeout flag#228

Open
qiaoxincheng880624-sudo wants to merge 1 commit intokubevela:mainfrom
qiaoxincheng880624-sudo:feat/configurable-reconcile-timeout
Open

Feat: make reconcile timeout configurable via --reconcile-timeout flag#228
qiaoxincheng880624-sudo wants to merge 1 commit intokubevela:mainfrom
qiaoxincheng880624-sudo:feat/configurable-reconcile-timeout

Conversation

@qiaoxincheng880624-sudo
Copy link
Copy Markdown

@qiaoxincheng880624-sudo qiaoxincheng880624-sudo commented Apr 15, 2026

Problem

WorkflowRun reconcile timeout is hardcoded to 3 minutes in
controllers/workflowrun_controller.go. There is no way to configure
this value at runtime, which causes failures for large workflowruns
(1000+ steps):

failed to patch workflowrun status: client rate limiter Wait returned
an error: context deadline exceeded

The reconcile context expires at exactly 3 minutes, and the final
patchStatus call fails because the context is already dead when the
rate limiter's Wait(ctx) checks it.

Solution

  • Add ReconcileTimeout time.Duration to Args struct
  • Expose --reconcile-timeout flag (default: 3m to preserve backward compatibility)
  • Use r.Args.ReconcileTimeout in Reconcile function

Usage

--reconcile-timeout=30m

Backward Compatibility

Default value remains 3m. No breaking change.

---

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Make the reconcile timeout configurable via the `--reconcile-timeout` flag to avoid context deadline exceeded errors on large workflow runs. Default stays 3m; behavior is unchanged unless you set it.

- **New Features**
- Added `ReconcileTimeout` to `controllers.Args` and a `--reconcile-timeout` flag in `cmd/main.go` (default: `controllers.DefaultReconcileTimeout`, 3m).
- `WorkflowRunReconciler` and `BackupReconciler` now use the configured timeout, falling back to `controllers.DefaultReconcileTimeout` if unset or invalid.
- Example: `--reconcile-timeout=30m`.

<sup>Written for commit dacf53597d7091d8ef374d782499100aadb0333c. Summary will update on new commits.</sup>

<!-- End of auto-generated description by cubic. -->

  Previously ReconcileTimeout was a hardcoded package-level variable (3m)
  with no way to override it. This caused workflowruns with large numbers
  of steps (1000+) to consistently fail with context deadline exceeded
  when attempting to patch status at the end of each reconcile cycle.

  Changes:
  - Add ReconcileTimeout field to Args struct
  - Rename package var to DefaultReconcileTimeout constant
  - Use r.Args.ReconcileTimeout in both WorkflowRunReconciler and BackupReconciler,
    falling back to DefaultReconcileTimeout if not set
  - Register --reconcile-timeout flag in cmd/main.go (default: 3m)

  Usage: --reconcile-timeout=30m
Copy link
Copy Markdown

@cubic-dev-ai cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 3 files

@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 15, 2026

Codecov Report

❌ Patch coverage is 88.88889% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 67.62%. Comparing base (d7db9c4) to head (dacf535).
⚠️ Report is 12 commits behind head on main.

Files with missing lines Patch % Lines
cmd/main.go 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #228      +/-   ##
==========================================
+ Coverage   62.49%   67.62%   +5.13%     
==========================================
  Files          62       65       +3     
  Lines        4415     5415    +1000     
==========================================
+ Hits         2759     3662     +903     
- Misses       1324     1367      +43     
- Partials      332      386      +54     
Flag Coverage Δ
e2etests 21.34% <25.00%> (?)
unit-test 61.64% <88.88%> (-0.85%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@qiaoxincheng880624-sudo qiaoxincheng880624-sudo changed the title feat: make reconcile timeout configurable via --reconcile-timeout flag Feat: make reconcile timeout configurable via --reconcile-timeout flag Apr 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant