Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .github/workflows/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,12 @@ jobs:
- name: Test
run: bundle exec rake

# Lint once, on a modern Ruby only (rubocop drops old Rubies; running it on
# every cell would fail on 2.6/2.7).
- name: Lint
if: matrix.os == 'ubuntu' && matrix.ruby == '3.4'
run: bundle exec rubocop

# Upload coverage once (single representative cell) to avoid duplicate reports.
- name: Upload coverage to Codecov
if: matrix.os == 'ubuntu' && matrix.ruby == '3.4'
Expand Down
6 changes: 3 additions & 3 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,10 @@
### Added
- Initial release of `dedupe_requests`.
- Server-side fingerprint de-duplication of POST/PUT/PATCH requests (no client idempotency key required).
- Controller macro `dedupe_requests` with `only:` / `also:` / `skip:` set operations over an inherited action set, plus `skip_dedupe_requests`.
- Global configuration via `DedupeRequests.configure` (redis, mode, ttl, digest, namespace, caller_id, fingerprint override, max_body_bytes, conflict status/body, logger, metrics hooks).
- Controller macro `dedupe_requests` with `only:` (add) and `skip:` (remove) over an inherited per-action map, plus `skip_dedupe_requests`; per-action TTL by repeating the line.
- Global configuration via `DedupeRequests.configure` (redis, mode, ttl, digest, namespace, caller_id, fingerprint override, conflict status/body, logger, metrics hooks).
- Three operating modes: `:off`, `:observe` (shadow), `:enforce`.
- Atomic `SET NX EX` claim with a random token and token-safe Lua check-and-del release.
- Releases the fingerprint on any non-2xx response or raised exception (around pattern), so failed requests can be retried.
- Keeps the fingerprint on a 2xx or 3xx (incl. Post/Redirect/Get) response; releases it on a 4xx/5xx response or a raised exception (around pattern), so failed requests can be retried.
- Fail-open behavior when Redis is unreachable.
- `duplicate_detected` / `duplicate_rejected` observability hooks.
5 changes: 3 additions & 2 deletions Gemfile
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,10 @@ source "https://rubygems.org"
gemspec

group :test do
gem "actionpack", ">= 7.0"
gem "rack-test"
gem "actionpack", ">= 5.2"
gem "mock_redis"
gem "rack-test"
gem "redis"
gem "rubocop", require: false
gem "simplecov", require: false
end
62 changes: 43 additions & 19 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ This is different from the usual idempotency-key gems: the **server** computes t
3. It runs an atomic `SET key <token> NX EX <ttl>` in Redis.
- **Key already existed** → it's a duplicate. In `enforce` mode, respond `409`; in `observe` mode, just record it and let it through.
- **Key created** → first occurrence. Run the action normally.
4. After the action: a `2xx` keeps the fingerprint until the TTL expires (so a later duplicate is blocked); **any non-2xx or a raised exception releases the fingerprint**, so a genuine retry of a failed request is allowed.
4. After the action: a **2xx, or a 3xx redirect** (the Post/Redirect/Get pattern is a successful create), keeps the fingerprint until the TTL expires so a later duplicate is blocked; a **4xx/5xx or a raised exception releases** the fingerprint, so a genuine retry of a failed request is allowed.

GET and DELETE are never deduped. Time is not part of the fingerprint — the window is the Redis TTL.

Expand All @@ -42,7 +42,7 @@ DedupeRequests.configure do |c|
c.ttl = 90 # the dedup window, in seconds
c.digest = :sha256 # :sha256 | :sha512 | :sha1 | :md5 | ->(bytes) { ... }
c.namespace = "myapp" # Redis key prefix
c.caller_id = ->(req) { req.get_header("HTTP_AUTHORIZATION") } # per-caller scoping
c.caller_id = ->(controller) { controller.current_user&.id } # per-caller scoping
c.logger = Rails.logger # where Redis/fail-open errors are logged
end
```
Expand All @@ -56,36 +56,58 @@ Include the concern once (usually in `ApplicationController`), then declare whic
```ruby
class ApplicationController < ActionController::Base
include DedupeRequests::Controller
dedupe_requests only: %i[create update] # project-wide baseline
dedupe_requests on: %i[create update] # project-wide baseline
end
```

Subclasses adjust the inherited baseline with three set operations:
Each `dedupe_requests` line **adds** the actions it names to the guarded set — it does not replace anything (same as Rails' own `before_action only:`). A controller inherits its parent's guarded actions and can add more or drop some:

| Option | Meaning for this controller | Result vs. inherited set |
| ------- | ---------------------------------- | ------------------------ |
| `only:` | exact list — ignore the baseline | replace |
| `also:` | baseline **plus** these | inherited ∪ these |
| `skip:` | baseline **minus** these | inherited − these |
| Option | Effect on this controller |
| ------- | ---------------------------------------------- |
| `on:` | guard these actions (uses this line's `ttl:`) |
| `skip:` | stop guarding these actions — no dedupe at all |

```ruby
class ReportsController < ApplicationController
dedupe_requests only: %i[generate] # ignore baseline; just this action
class OrdersController < ApplicationController
dedupe_requests on: %i[approve cancel] # adds approve/cancel to the inherited create/update
end

class DraftsController < ApplicationController
dedupe_requests skip: %i[create] # baseline minus create
dedupe_requests skip: %i[create] # guards everything inherited except create
end
```

class OrdersController < ApplicationController
dedupe_requests also: %i[approve cancel] # baseline plus these two
#### Per-action TTL

A `ttl:` applies to exactly the actions named on its line. Give different actions different windows by repeating the line — a list shares one TTL:

```ruby
class PaymentsController < ApplicationController
dedupe_requests on: %i[create charge], ttl: 120 # create + charge → 120s
dedupe_requests on: [:refund], ttl: 600 # refund → 600s
end
```

A per-controller TTL override rides on the same line: `dedupe_requests only: %i[create], ttl: 120`.
An action with no `ttl:` falls back to the global `config.ttl`; re-declaring an action updates its TTL.

You never specify HTTP verbs per action — the route already determines the verb, and the gem only ever guards POST/PUT/PATCH.

### 3. Per-caller identity (`caller_id`)

Dedup is scoped per caller, so two different users sending the same payload don't collide. `caller_id` is a callable given the **controller**, so it can read whatever identifies the caller:

```ruby
DedupeRequests.configure do |c|
c.caller_id = ->(controller) { controller.current_user&.id } # current_user
# c.caller_id = ->(controller) { controller.request.get_header("HTTP_X_API_KEY") } # a header
# c.caller_id = ->(controller) { controller.some_method } # any controller method
end
```

If you don't set it, the default derives identity from the `Authorization` header, falling back to a Rails session cookie — so token- and cookie-auth apps work with no configuration.

> **Note:** make sure you configure `caller_id` correctly for your API. If it can't derive an identity (no `Authorization` header and no session cookie), it falls back to `nil` — and then *different* callers sending the same payload to the same endpoint are treated as one request, so the second gets a 409. That's probably not what you want, so set `caller_id` to whatever identifies a caller in your app.

## Modes and safe rollout

`mode` has three states:
Expand All @@ -102,13 +124,15 @@ Wire the hooks to your metrics/logging backend (Datadog, StatsD, logs — your c

```ruby
DedupeRequests.configure do |c|
c.on_duplicate_detected = ->(info) { StatsD.increment("dedupe.detected", tags: info) }
c.on_duplicate_rejected = ->(info) { StatsD.increment("dedupe.rejected", tags: info) }
c.on_duplicate_detected = ->(info) { StatsD.increment("dedupe.detected", tags: { controller: info[:controller], action: info[:action], verb: info[:verb] }) }
c.on_duplicate_rejected = ->(info) { StatsD.increment("dedupe.rejected", tags: { controller: info[:controller], action: info[:action], verb: info[:verb] }) }
end
```

Each hook receives `{ fingerprint:, controller:, action:, verb:, path: }`. `duplicate_detected` fires in both `observe` and `enforce`; `duplicate_rejected` only when a 409 is actually returned.

When tagging metrics, use only `controller`, `action`, and `verb` — these come from a small fixed set. Do **not** tag with `fingerprint` or `path`: the fingerprint is unique per request and the path usually contains record ids, so tagging with them creates a separate counter per request (a surprise bill on Datadog, or dropped series and broken dashboards). Log those instead if you need them.

## The 409 response

Default body (override via `config.conflict_body`, and status via `config.conflict_status`):
Expand All @@ -129,6 +153,7 @@ A `409` is deliberate: well-behaved retrying clients do **not** loop on a 409 (t

- **Fail open.** If Redis is unreachable, the request proceeds normally — a Redis outage never blocks traffic. Redis errors are rescued and logged (set `config.logger`). The logger is used **only** for these Redis/fail-open errors — not for normal duplicate handling (use the hooks above for that) — and it is wired automatically only when the store is built from `config.redis`. If you inject your own `config.store`, pass it a logger directly.
- **Token-safe release.** Each claim stores a random token; release deletes the key only if it still holds that token (via a Lua check-and-del), so a slow request whose TTL expired can't wipe a newer request's fresh claim.
- **Compile Ruby with OpenSSL — for speed.** The fingerprint hashes the request body on the hot path. It uses `OpenSSL::Digest`, which runs on the CPU's SHA instructions (SHA-NI / ARM crypto) at ~1.5–2 GB/s. If your Ruby is built **without** OpenSSL, the gem still works — it falls back to the stdlib `Digest` — but that's a portable software implementation (~300–500 MB/s, no SHA instructions), several times slower on large bodies. So build Ruby with OpenSSL in production.

## Configuration reference

Expand All @@ -140,9 +165,8 @@ A `409` is deliberate: well-behaved retrying clients do **not** loop on a 409 (t
| `ttl` | `90` | Dedup window, in seconds. |
| `digest` | `:sha256` | `:sha256` / `:sha512` / `:sha1` / `:md5`, or a callable. |
| `namespace` | `"dedupe_requests"` | Redis key prefix (`<namespace>:dedup:<hash>`). |
| `caller_id` | Authorization / session cookie | Callable returning a per-caller identity. |
| `caller_id` | Authorization / session cookie | Callable **given the controller**, returns a per-caller identity (e.g. `->(c){ c.current_user&.id }`, a header via `c.request`, or any controller method). Default derives it from the Authorization header / session cookie. |
| `fingerprint` | `nil` | Callable to fully override fingerprint computation. |
| `max_body_bytes` | `nil` | Cap how many body bytes are hashed (for very large payloads). |
| `conflict_status` | `409` | Status returned for a rejected duplicate. |
| `conflict_body` | structured errors | JSON body for a rejected duplicate. |
| `logger` | `nil` | Where Redis errors are logged. |
Expand Down
5 changes: 3 additions & 2 deletions dedupe_requests.gemspec
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# frozen_string_literal: true

require_relative "lib/dedupe_requests/version"

# rubocop:disable Layout/LineLength
Gem::Specification.new do |spec|
spec.name = "dedupe_requests"
spec.version = DedupeRequests::VERSION
Expand All @@ -21,8 +21,9 @@ Gem::Specification.new do |spec|
spec.files = Dir["lib/**/*.rb", "README.md", "CHANGELOG.md", "LICENSE.txt"]
spec.require_paths = ["lib"]

spec.add_dependency "activesupport", ">= 7.0"
spec.add_dependency "activesupport", ">= 5.2"

spec.add_development_dependency "rake", "~> 13.0"
spec.add_development_dependency "rspec", "~> 3.0"
end
# rubocop:enable Layout/LineLength
19 changes: 14 additions & 5 deletions lib/dedupe_requests/configuration.rb
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,18 @@ class Configuration
}]
}.freeze

# Default per-caller identity: the Authorization header, falling back to a
# Rails-style session cookie. Override via `config.caller_id`.
DEFAULT_CALLER_ID = lambda do |request|
# Per-caller identity. The callable is given the CONTROLLER, so it can read
# anything the controller exposes — `current_user`, a helper method, or a
# header via `controller.request`. Examples:
# c.caller_id = ->(controller) { controller.current_user&.id }
# c.caller_id = ->(controller) { controller.request.get_header("HTTP_X_API_KEY") }
#
# The default derives identity from the request's Authorization header,
# falling back to a Rails-style session cookie (so token- and cookie-auth
# apps work with no configuration). It accepts either a controller or a bare
# request.
DEFAULT_CALLER_ID = lambda do |context|
request = context.respond_to?(:request) ? context.request : context
if request.respond_to?(:get_header)
auth = request.get_header("HTTP_AUTHORIZATION")
return auth if auth && !auth.to_s.empty?
Expand All @@ -26,7 +35,7 @@ class Configuration
end

attr_accessor :redis, :ttl, :digest, :namespace, :caller_id, :fingerprint,
:max_body_bytes, :conflict_status, :logger,
:conflict_status, :logger,
:on_duplicate_detected, :on_duplicate_rejected
attr_writer :store, :conflict_body
attr_reader :mode
Expand All @@ -40,7 +49,6 @@ def initialize
@namespace = "dedupe_requests"
@caller_id = DEFAULT_CALLER_ID
@fingerprint = nil
@max_body_bytes = nil
@conflict_status = 409
@logger = nil
@on_duplicate_detected = nil
Expand All @@ -53,6 +61,7 @@ def mode=(value)
unless MODES.include?(sym)
raise ArgumentError, "unknown mode #{value.inspect} (expected one of #{MODES.join(', ')})"
end

@mode = sym
end

Expand Down
87 changes: 57 additions & 30 deletions lib/dedupe_requests/controller.rb
Original file line number Diff line number Diff line change
Expand Up @@ -8,40 +8,53 @@ module DedupeRequests
#
# class ApplicationController < ActionController::Base
# include DedupeRequests::Controller
# dedupe_requests only: %i[create update]
# dedupe_requests on: %i[create update]
# end
#
# Registers a SINGLE around_action that, for actions in the controller's
# de-dupe set, claims before the action runs and releases on any non-2xx
# response or raised exception. The set is an inherited class_attribute so
# subclasses can replace (only:), extend (also:), or trim (skip:) it.
# Registers a SINGLE around_action that, for each guarded action, claims before
# the action runs and releases on a 4xx/5xx response or a raised exception
# (2xx/3xx keep the claim). The guarded actions and their per-action TTLs live
# in an inherited class_attribute map, so subclasses extend or trim it.
module Controller
extend ActiveSupport::Concern

included do
class_attribute :dedupe_requests_actions, instance_accessor: false, default: nil
class_attribute :dedupe_requests_options, instance_accessor: false, default: {}
# Map of guarded action (Symbol) => TTL (Integer, or nil meaning "use the
# global config TTL"). Inherited and copy-on-write, so a subclass can add
# to or trim it without touching the parent.
class_attribute :dedupe_requests_action_ttls, instance_accessor: false, default: {}
around_action :dedupe_requests_around
end

class_methods do
def dedupe_requests(only: nil, also: nil, skip: nil, **options)
inherited = dedupe_requests_actions || []
new_set =
if only
Array(only).map(&:to_sym)
else
set = inherited.dup
set |= Array(also).map(&:to_sym) if also
set -= Array(skip).map(&:to_sym) if skip
set
end
self.dedupe_requests_actions = new_set.uniq
self.dedupe_requests_options = dedupe_requests_options.merge(options) unless options.empty?
# Guard the named actions. A `ttl:`, if given, applies to exactly the
# actions named in THIS call. Calls accumulate, so per-action TTLs are
# expressed by repeating the line:
#
# dedupe_requests on: %i[create update] # both, global TTL
# dedupe_requests on: [:create], ttl: 120 # create → 120
# dedupe_requests on: [:update], ttl: 180 # update → 180
#
# Re-naming an action overrides its TTL. Subclasses inherit the map and can
# add to it or remove from it (`skip:` / `skip_dedupe_requests`).
def dedupe_requests(on: nil, skip: nil, ttl: nil)
map = dedupe_requests_action_ttls.dup

Array(on).each { |action| map[action.to_sym] = ttl }
Array(skip).each { |action| map.delete(action.to_sym) }

self.dedupe_requests_action_ttls = map
end

def skip_dedupe_requests(only: nil)
self.dedupe_requests_actions = (dedupe_requests_actions || []) - Array(only).map(&:to_sym)
def skip_dedupe_requests(on: nil)
map = dedupe_requests_action_ttls.dup
Array(on).each { |action| map.delete(action.to_sym) }
self.dedupe_requests_action_ttls = map
end

# The set of guarded actions (the keys of the TTL map).
def dedupe_requests_actions
dedupe_requests_action_ttls.keys
end
end

Expand All @@ -53,8 +66,11 @@ def dedupe_requests_around
return
end

ttl = self.class.dedupe_requests_options[:ttl] || DedupeRequests.config.ttl
result = dedupe_requests_guard.claim(request, ttl: ttl)
result = dedupe_requests_guard.claim(
request,
ttl: dedupe_requests_ttl_for(action_name),
caller_id: dedupe_requests_caller_id
)

case result.outcome
when :duplicate
Expand Down Expand Up @@ -82,19 +98,30 @@ def dedupe_requests_around
def dedupe_requests_applies?
return false unless DedupeRequests.config.enabled?

actions = self.class.dedupe_requests_actions
!actions.nil? && actions.include?(action_name.to_sym)
self.class.dedupe_requests_action_ttls.key?(action_name.to_sym)
end

# Per-action TTL, falling back to the global config TTL.
def dedupe_requests_ttl_for(action)
self.class.dedupe_requests_action_ttls[action.to_sym] || DedupeRequests.config.ttl
end

# Resolve the caller identity by handing the whole controller to the
# configured `caller_id` callable (so it can use current_user, a header, etc.).
def dedupe_requests_caller_id
DedupeRequests.config.caller_id&.call(self)
end

def dedupe_requests_guard
@dedupe_requests_guard ||= DedupeRequests::Guard.new(DedupeRequests.config)
end

# A request didn't complete successfully → free the fingerprint so a genuine
# retry isn't blocked. Only a 2xx keeps the fingerprint for the full TTL.
# Keep the fingerprint when the request was handled — a 2xx, or a 3xx
# redirect (the Post/Redirect/Get pattern is a *successful* create) — so a
# later duplicate is still blocked for the full TTL. Only a 4xx/5xx (or a
# raised exception) releases it, so a genuinely failed request can be retried.
def dedupe_requests_release?(status)
code = status.to_i
code < 200 || code >= 300
status.to_i >= 400
end

def dedupe_requests_render_conflict
Expand Down
Loading
Loading