diff --git a/CHANGELOG.md b/CHANGELOG.md index 1eb58a405..4999a0c4c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Added - Added `rollback` ability to undo on matching status +- Added documentation for retries + +### Updated +- Added `retry_jitter` option to support symbol, proc, and callables ## [1.9.1] - 2025-10-22 diff --git a/docs/callbacks.md b/docs/callbacks.md index 3fcf8a64d..bff3a9b5b 100644 --- a/docs/callbacks.md +++ b/docs/callbacks.md @@ -73,7 +73,7 @@ end ### Class or Module -Implement reusable callback logic in dedicated classes: +Implement reusable callback logic in dedicated modules and classes: ```ruby class BookingConfirmationCallback diff --git a/docs/getting_started.md b/docs/getting_started.md index 0706956a3..51b2be0a3 100644 --- a/docs/getting_started.md +++ b/docs/getting_started.md @@ -281,7 +281,7 @@ end !!! warning "Important" - Retries reuse the same context. By default, all `StandardError` exceptions are retried unless you specify `retry_on`. + Retries reuse the same context. By default, all `StandardError` exceptions (including faults) are retried unless you specify `retry_on` option for specific matches. ### Registrations diff --git a/docs/retries.md b/docs/retries.md new file mode 100644 index 000000000..892229d47 --- /dev/null +++ b/docs/retries.md @@ -0,0 +1,122 @@ +# Retries + +CMDx provides automatic retry functionality for tasks that encounter transient failures. This is essential for handling temporary issues like network timeouts, rate limits, or database locks without manual intervention. + +## Basic Usage + +Configure retries upto n attempts without any delay. + +```ruby +class FetchExternalData < CMDx::Task + settings retries: 3 + + def work + response = HTTParty.get("https://api.example.com/data") + context.data = response.parsed_response + end +end +``` + +When an exception occurs during execution, CMDx automatically retries up to the configured limit. + +## Selective Retries + +By default, CMDx retries on `StandardError` and its subclasses. Narrow this to specific exception types: + +```ruby +class ProcessPayment < CMDx::Task + settings retries: 5, retry_on: [Stripe::RateLimitError, Net::ReadTimeout] + + def work + # Your logic here... + end +end +``` + +!!! warning "Important" + + Only exceptions matching the `retry_on` configuration will trigger retries. Uncaught exceptions immediately fail the task. + +## Retry Jitter + +Add delays between retry attempts to avoid overwhelming external services or to implement exponential backoff strategies. + +### Fixed Value + +Use a numeric value to calculate linear delay (`jitter * current_retry`): + +```ruby +class ImportRecords < CMDx::Task + # Fixed + settings retries: 3, retry_jitter: 0.5 + + def work + # Delays: 0s, 0.5s (retry 1), 1.0s (retry 2), 1.5s (retry 3) + context.records = ExternalAPI.fetch_records + end +end +``` + +### Symbol References + +Define an instance method for custom delay logic: + +```ruby +class SyncInventory < CMDx::Task + settings retries: 5, retry_jitter: :exponential_backoff + + def work + context.inventory = InventoryAPI.sync + end + + private + + def exponential_backoff(current_retry) + 2 ** current_retry # 2s, 4s, 8s, 16s, 32s + end +end +``` + +### Proc or Lambda + +Pass a proc for inline delay calculations: + +```ruby +class PollJobStatus < CMDx::Task + # Proc + settings retries: 10, retry_jitter: proc { |retry_count| [retry_count * 0.5, 5.0].min } + + # Lambda + settings retries: 10, retry_jitter: ->(retry_count) { [retry_count * 0.5, 5.0].min } + + def work + # Delays: 0.5s, 1.0s, 1.5s, 2.0s, 2.5s, 3.0s, 3.5s, 4.0s, 4.5s, 5.0s (capped) + context.status = JobAPI.check_status(context.job_id) + end +end +``` + +### Class or Module + +Implement reusable delay logic in dedicated modules and classes: + +```ruby +class ExponentialBackoff + def call(task, retry_count) + base_delay = task.context.base_delay || 1.0 + [base_delay * (2 ** retry_count), 60.0].min + end +end + +class FetchUserProfile < CMDx::Task + # Class or Module + settings retries: 4, retry_jitter: ExponentialBackoff + + # Instance + settings retries: 4, retry_jitter: ExponentialBackoff.new + + def work + # Your logic here... + end +end +``` diff --git a/lib/cmdx/executor.rb b/lib/cmdx/executor.rb index ebdea7b06..f538f489a 100644 --- a/lib/cmdx/executor.rb +++ b/lib/cmdx/executor.rb @@ -151,7 +151,18 @@ def retry_execution?(exception) task.to_h.merge!(reason:, remaining_retries:) end - jitter = task.class.settings[:retry_jitter].to_f * current_retries + jitter = task.class.settings[:retry_jitter] + jitter = + if jitter.is_a?(Symbol) + task.send(jitter, current_retries) + elsif jitter.is_a?(Proc) + task.instance_exec(current_retries, &jitter) + elsif jitter.respond_to?(:call) + jitter.call(task, current_retries) + else + jitter.to_f * current_retries + end + sleep(jitter) if jitter.positive? true diff --git a/mkdocs.yml b/mkdocs.yml index 005158b96..475201796 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -108,6 +108,7 @@ nav: - Middlewares: middlewares.md - Logging: logging.md - Internationalization: internationalization.md + - Retries: retries.md - Deprecation: deprecation.md - Workflows: workflows.md - Tips and Tricks: tips_and_tricks.md diff --git a/spec/cmdx/executor_spec.rb b/spec/cmdx/executor_spec.rb index fe3185f38..88766ff59 100644 --- a/spec/cmdx/executor_spec.rb +++ b/spec/cmdx/executor_spec.rb @@ -359,6 +359,252 @@ end end + describe "#retry_execution?" do + let(:exception) { StandardError.new("test error") } + let(:logger) { instance_double(Logger) } + + before do + allow(logger).to receive(:warn) + allow(task).to receive_messages(logger: logger, to_h: { id: "123" }) + end + + context "when retries is not configured" do + before do + allow(task.class).to receive(:settings).and_return({}) + end + + it "returns false" do + expect(worker.send(:retry_execution?, exception)).to be(false) + end + end + + context "when retries is 0" do + before do + allow(task.class).to receive(:settings).and_return({ retries: 0 }) + end + + it "returns false" do + expect(worker.send(:retry_execution?, exception)).to be(false) + end + end + + context "when retries are exhausted" do + before do + allow(task.class).to receive(:settings).and_return({ retries: 2 }) + allow(task.result).to receive(:metadata).and_return({ retries: 2 }) + end + + it "returns false" do + expect(worker.send(:retry_execution?, exception)).to be(false) + end + end + + context "when exception type does not match retry_on" do + before do + allow(task.class).to receive(:settings).and_return({ retries: 3, retry_on: [ArgumentError] }) + allow(task.result).to receive(:metadata).and_return({ retries: 0 }) + end + + it "returns false" do + expect(worker.send(:retry_execution?, exception)).to be(false) + end + end + + context "when retry should happen" do + before do + allow(task.class).to receive(:settings).and_return({ retries: 3 }) + allow(task.result).to receive(:metadata).and_return({ retries: 0 }) + end + + it "returns true" do + expect(worker.send(:retry_execution?, exception)).to be(true) + end + + it "increments retry count in metadata" do + metadata = { retries: 1 } + allow(task.result).to receive(:metadata).and_return(metadata) + + worker.send(:retry_execution?, exception) + + expect(metadata[:retries]).to eq(2) + end + + it "logs warning with reason and remaining retries" do + expect(logger).to receive(:warn) do |&block| + result = block.call + expect(result[:reason]).to eq("[StandardError] test error") + expect(result[:remaining_retries]).to eq(3) + end + + worker.send(:retry_execution?, exception) + end + end + + context "with retry_on configuration" do + context "when exception matches configured type" do + before do + allow(task.class).to receive(:settings).and_return({ retries: 2, retry_on: [StandardError] }) + allow(task.result).to receive(:metadata).and_return({ retries: 0 }) + end + + it "returns true" do + expect(worker.send(:retry_execution?, exception)).to be(true) + end + end + + context "when exception is subclass of configured type" do + let(:custom_error) { CMDx::TestError.new("test error") } + + before do + allow(task.class).to receive(:settings).and_return({ retries: 2, retry_on: [StandardError] }) + allow(task.result).to receive(:metadata).and_return({ retries: 0 }) + end + + it "returns true" do + expect(worker.send(:retry_execution?, custom_error)).to be(true) + end + end + + context "when multiple exception types are configured" do + before do + allow(task.class).to receive(:settings).and_return({ retries: 2, retry_on: [ArgumentError, StandardError] }) + allow(task.result).to receive(:metadata).and_return({ retries: 0 }) + end + + it "returns true if exception matches any type" do + expect(worker.send(:retry_execution?, exception)).to be(true) + end + end + end + + context "with retry_jitter as numeric value" do + before do + allow(task.class).to receive(:settings).and_return({ retries: 3, retry_jitter: 0.5 }) + allow(task.result).to receive(:metadata).and_return({ retries: 1 }) + end + + it "sleeps for jitter multiplied by current retries" do + expect(worker).to receive(:sleep).with(0.5) + + worker.send(:retry_execution?, exception) + end + + context "when first retry" do + before do + allow(task.result).to receive(:metadata).and_return({ retries: 0 }) + end + + it "does not sleep when jitter calculation is 0" do + expect(worker).not_to receive(:sleep) + + worker.send(:retry_execution?, exception) + end + end + + context "when second retry" do + before do + allow(task.result).to receive(:metadata).and_return({ retries: 1 }) + end + + it "sleeps for jitter * 1" do + expect(worker).to receive(:sleep).with(0.5) + + worker.send(:retry_execution?, exception) + end + end + + context "when third retry" do + before do + allow(task.result).to receive(:metadata).and_return({ retries: 2 }) + end + + it "sleeps for jitter * 2" do + expect(worker).to receive(:sleep).with(1.0) + + worker.send(:retry_execution?, exception) + end + end + end + + context "with retry_jitter as symbol" do + before do + allow(task.class).to receive(:settings).and_return({ retries: 3, retry_jitter: :custom_jitter }) + allow(task.result).to receive(:metadata).and_return({ retries: 1 }) + allow(task).to receive(:custom_jitter).with(1).and_return(2.5) + end + + it "calls method on task with current retries" do + expect(task).to receive(:custom_jitter).with(1).and_return(2.5) + expect(worker).to receive(:sleep).with(2.5) + + worker.send(:retry_execution?, exception) + end + end + + context "with retry_jitter as proc" do + let(:jitter_proc) { ->(retries) { retries * 0.75 } } + + before do + allow(task.class).to receive(:settings).and_return({ retries: 3, retry_jitter: jitter_proc }) + allow(task.result).to receive(:metadata).and_return({ retries: 2 }) + end + + it "instance_execs proc with current retries" do + expect(worker).to receive(:sleep).with(1.5) + + worker.send(:retry_execution?, exception) + end + end + + context "with retry_jitter as callable object" do + let(:jitter_callable) do + Class.new do + def call(_task, retries) + retries * 1.25 + end + end.new + end + + before do + allow(task.class).to receive(:settings).and_return({ retries: 3, retry_jitter: jitter_callable }) + allow(task.result).to receive(:metadata).and_return({ retries: 2 }) + end + + it "calls object with task and current retries" do + expect(jitter_callable).to receive(:call).with(task, 2).and_return(2.5) + expect(worker).to receive(:sleep).with(2.5) + + worker.send(:retry_execution?, exception) + end + end + + context "when jitter calculation returns negative value" do + before do + allow(task.class).to receive(:settings).and_return({ retries: 3, retry_jitter: -0.5 }) + allow(task.result).to receive(:metadata).and_return({ retries: 1 }) + end + + it "does not sleep" do + expect(worker).not_to receive(:sleep) + + worker.send(:retry_execution?, exception) + end + end + + context "when jitter calculation returns zero" do + before do + allow(task.class).to receive(:settings).and_return({ retries: 3, retry_jitter: 0 }) + allow(task.result).to receive(:metadata).and_return({ retries: 1 }) + end + + it "does not sleep" do + expect(worker).not_to receive(:sleep) + + worker.send(:retry_execution?, exception) + end + end + end + describe "#raise_exception" do let(:exception) { StandardError.new("test error") }