[CE-012] idempotency.rs stores keys in memory only — process restart loses all idempotency guarantees

[Jaydbrown]

CE-012 — idempotency.rs stores keys in memory only — process restart loses all idempotency guarantees

Affected Files

• src/idempotency.rs — IdempotencyStore using DashMap (primary bug site)
• src/payment/mod.rs — payment submission logic that calls idempotency.check_and_set()
• src/main.rs — where IdempotencyStore is constructed and shared
• src/config.rs — Redis connection config; idempotency TTL config
• docker-compose.yml — Redis service definition

Root Cause

The IdempotencyStore uses a DashMap (concurrent in-memory hash map) to store idempotency keys:

// src/idempotency.rs — current (BUGGY)
pub struct IdempotencyStore {
    entries: DashMap<String, Entry>,
    ttl: Duration,  // 24 hours
}

#[derive(Clone)]
struct Entry {
    response: CachedResponse,
    expires_at: Instant,
}

impl IdempotencyStore {
    pub fn new(ttl: Duration) -> Self {
        Self { entries: DashMap::new(), ttl }
    }

    pub fn check_and_set(&self, key: &str, response: CachedResponse) -> Option<CachedResponse> {
        if let Some(entry) = self.entries.get(key) {
            if entry.expires_at > Instant::now() {
                return Some(entry.response.clone());  // duplicate request
            }
        }
        self.entries.insert(key.to_string(), Entry {
            response,
            expires_at: Instant::now() + self.ttl,
        });
        None  // first request
    }
}

When the process restarts (deploy, crash, OOM), the DashMap is gone. A client that:

1. Sends a payment with Idempotency-Key: pay-xyz-123
2. Receives a network timeout (doesn't know if the payment went through)
3. Retries with the same Idempotency-Key: pay-xyz-123

...will receive a fresh payment execution if the process restarted between the original request and the retry, resulting in a duplicate payment.

The 24-hour TTL also means all keys expire together if there's a burst of traffic — there's no per-key TTL refresh and no eviction beyond TTL.

Impact

• Severity: High — Duplicate payments are possible whenever the engine restarts. In a Kubernetes rolling deploy, at least one pod restarts every deploy. Mobile clients that retry on timeout (standard behavior) are the most likely to trigger this.
• Financial impact: a user initiating a $100 XLM→USDC swap could have two swaps executed if a deploy happens between their request and retry
• The DashMap also has unbounded growth — with 24h TTL and any volume, it accumulates entries in memory indefinitely until the process restarts

Fix

Replace the DashMap with Redis-backed persistence using SET NX EX (set if not exists, with expiry):

// src/idempotency.rs — FIXED with Redis
use redis::AsyncCommands;

pub struct IdempotencyStore {
    redis: Arc<redis::Client>,
    ttl_seconds: u64,  // 24 * 3600 = 86400
    key_prefix: String,  // "idem:"
}

impl IdempotencyStore {
    pub async fn check_and_set(
        &self,
        idempotency_key: &str,
        response: &CachedResponse,
    ) -> Result<Option<CachedResponse>, IdempotencyError> {
        let redis_key = format!("{}:{}", self.key_prefix, idempotency_key);
        let mut conn = self.redis.get_async_connection().await?;

        // Try to SET the key only if it doesn't exist (NX), with TTL (EX)
        let serialized = serde_json::to_string(response)?;
        let set_result: Option<String> = conn
            .set_options(&redis_key, &serialized, SetOptions::default().nx().ex(self.ttl_seconds))
            .await?;

        if set_result.is_some() {
            // We set it — this is the first request
            return Ok(None);
        }

        // Key already exists — return cached response
        let existing: String = conn.get(&redis_key).await?;
        let cached: CachedResponse = serde_json::from_str(&existing)?;
        Ok(Some(cached))
    }
}

For resilience when Redis is unavailable, define a degradation policy:

// Option A: Fail closed — reject all payments when Redis is down (safest for financial ops)
Err(IdempotencyError::RedisUnavailable) => {
    return (StatusCode::SERVICE_UNAVAILABLE, Json(error_body(
        "IDEMPOTENCY_UNAVAILABLE",
        "payment processing temporarily unavailable; please retry in a moment"
    ))).into_response();
}

Additional Files to Update

• src/idempotency.rs — replace DashMap with Redis; add check_and_set async function
• src/main.rs — pass Redis client to IdempotencyStore::new()
• src/payment/mod.rs — await check_and_set call; handle IdempotencyError::RedisUnavailable
• src/config.rs — add idempotency_ttl_hours: u64 config field
• docker-compose.yml — ensure Redis maxmemory-policy is set (see BE-024 for Redis config)
• tests/idempotency_tests.rs — add: restart simulation (drop and recreate store) shows key is gone with memory store, shows key is preserved with Redis store

[oomokaro1]

Hi @Jaydbrown, I'd like to take this on.

I have hands-on experience with Rust async patterns, Redis-backed persistence, and concurrent data structures like DashMap. I'm confident I can deliver a solid fix.

My approach:

1. Define an IdempotencyStore trait with async get/set methods to abstract the storage backend
2. Implement RedisIdempotencyStore using redis::aio::ConnectionManager with SET EX for TTL-based key storage and automatic expiry
3. Keep the existing DashMap implementation as MemoryIdempotencyStore for local dev and test environments (when REDIS_URL is not set)
4. Add REDIS_URL to Config::from_env() in src/config.rs and wire the appropriate store into Engine::new() in src/main.rs
5. Update src/api/payments.rs to await the now-async idempotency calls, with fail-open on Redis get errors (allow payment if Redis is temporarily down)
6. Update tests/integration_test.rs to use the memory store, and add a test verifying cached responses are returned on duplicate idempotency keys

I'm ready to start immediately and can have a PR up within a few days. Happy to adjust the plan if you have preferences on any of the details.

[Jaydbrown]

after reviewing your application i would say that i have the utomst confidence in you and hence i will be assigning the issue to you

[grantfox-oss]

🦊 GrantFox — @oomokaro1 has been assigned to this issue as part of the Official Campaign campaign!

Next steps:

1. Open a Pull Request referencing this issue (e.g., Closes #52)
2. Your PR will be reviewed by the Nodus-protocol maintainers

Good luck! Track your progress on GrantFox.

[grantfox-oss]

🎉 This issue has been marked as completed on GrantFox as part of the Official Campaign campaign!

@oomokaro1's PR #100 was approved and merged by @Jaydbrown.

🏆 @oomokaro1: You earned 35 FoxPoints for this contribution! Your current tier: Explorer (118 total points). Track your full progress on GrantFox.

👏 Great work, @oomokaro1! Keep contributing to Nodus-protocol.