PROTOCOL: No Message Ordering Guarantees

[devfire]

Severity: HIGH

Issue

UDP multicast provides no ordering guarantees. Messages can arrive out of order, causing protocol violations and failures.

Problem Scenario

Expected order:

1. Alice creates group
2. Alice sends commit adding Bob
3. Alice sends Welcome to Bob
4. Bob processes Welcome and joins

Actual UDP multicast order:

1. Bob receives Welcome (no group context yet - FAILS)
2. Bob receives commit (cannot process - no group)
3. Alice's group creation arrives

Current State

No sequence tracking or reordering:

pub async fn receive_message(&self) -> Result<ProtoMlsMessageIn> {
    let (len, remote_addr) = self.socket.recv_from(&mut buffer).await?;
    let packet = ProtoMlsMessageIn::decode(&buffer[..len])?;
    Ok(packet) // Process immediately, no ordering
}

Impact

• Welcome messages arrive before group creation
• Commits processed out of order
• Epoch mismatches
• Group state desynchronization
• Random failures that are hard to reproduce

Solution Options

Option 1: Add sequence numbers

message MlsMessageOut {
    uint64 sequence_number = 6;
    uint64 timestamp_ms = 7;
    // existing fields...
}

struct MessageQueue {
    pending: BTreeMap<u64, Message>,
    next_expected: u64,
}

Option 2: Use TCP for control messages

• Keep UDP multicast for application messages
• Use TCP for group management (Welcome, commits)
• Guarantees ordering where it matters

Option 3: Causal ordering

• Track message dependencies
• Hold messages until dependencies satisfied
• Process in causal order

Option 4: Use MLS epoch numbers

• Buffer messages from future epochs
• Process when epoch advances
• Leverage MLS built-in ordering

Recommended Approach

Combine Option 1 and 4:

1. Add sequence numbers to protobuf
2. Use MLS epochs for coarse ordering
3. Buffer out-of-order messages
4. Process when ordering satisfied

Related Issues

• Replay protection issue SECURITY: No Replay Attack Protection #20
• Protocol robustness
• Race conditions in group operations

Labels

protocol, reliability, enhancement