Improved Inbox/Outbox/DLQ Storage Epic

[jeremydmiller]

Nobody get too excited here because a lot of this is driven by JasperFx client needs and this is related to some pending client engagements that have not been signed yet

Usability through Database Management

For RDBMS users -- which is most of our user base so far -- have an extensible storage that provides more usability through basic database management tools than is provided by the envelope serialization. So, for example, support:

• Storing the tenant_id in a separate column -- this one is easy, and we might even be able to add this w/ opt in flags sooner rather than later
• Tagging DLQ records by a group identifier like an event stream identity or saga id. That could use the Envelope.GroupId
• Store the message body as JSON in a separate column so it's easily visible and maybe queryable -- this is way more usable if the system is only using JSON serialization
• Possibly storing other header values and metadata in a JSON serialized dictionary similar to how we do it w/ Marten events

Command Line Tools

• A new command line tool that can attempt to force Wolverine to process all inbox messages inline as a kind of "catch up now!" mode. Focus on messages w/ owner_id.
• Same command line tool has an option to reassign/release all Incoming envelopes in the inbox
• New command line to tool to force all outbox messages to be sent
• Maybe an IHost helper extension to "push" all outbox messages in automated testing. Or something to "wait" for all messages to be sent. Tangentially related, it might be helpful to have a testing time override to force all sending endpoints to be Inline???? Maybe finally an outbox + inline option? A new DurableInline option??? Or really another flag on the durable option?

CritterWatch Integration

• Ability to query the inbox/outbox in addition to what's already been built for the DLQ
• Especially the ability to analyze the scheduled messages
• Ability to "force" stuck inbox messages to be played "now Mister!" (https://www.youtube.com/watch?v=BMR8AnL3RXw)
• Ability to "force" stuck outbox messages to be played "now Mister!"
• Ability to delete or reschedule scheduled messages
• Ability to play scheduled message right now

EF Core Backed Message Store

Just what it says, leverage EF Core for a message store and an equivalent DurabilityAgent. This is partially to get us more flexibility and write less code, but also to potentially extend Wolverine inbox/outbox support to other relational database engines. We'd need to see about how to utilize EF Core migrations a bit better within an IStatefulResource model.

[JarrodJ83]

Couple other things I could think of that might help with general monitoring & troubleshooting of the system:

• Possible other fields to add:
  • Add a field to the tables that is the date they record was created. Would help with troubleshooting. For example, I don't see any timestamp in wolverine_outgoing_envelopes. That might be the only one? Incoming has execution_time but I'm not clear what tht represents.
  • Latest failure field?
• Additional metrics: Currently I think outbox has one counter but I don't see much else around these unless they are undocumented.
  • inbox/outbox counts so we can tell when things are not processing as expected
  • something to tell the time spent in any of these tables so we can track how long messages are living in them? Aside from scheduled messages I would expect the time spent in here to be very short when things are working as expected so this may be another smell things may not be working even if the count is not high

[jeremydmiller]

@JarrodJ83,

• Wolverine can already export counters on the state of the inbox & outbox
• "something to tell the time spent in any of these tables so we can track how long messages are living in them?" -- that's the "effective time" metric. Monitor that. The table storage isn't really important per se, what's important is how long you go from publishing to finished.

See: https://wolverinefx.net/guide/logging.html#metrics

I'll have to re-familiarize myself w/ the inbox/outbox counts, but that is a thing.

[JarrodJ83]

@jeremydmiller thanks! I'm not sure I totally follow. Here are some follow up questions.

Wolverine can already export counters on the state of the inbox & outbox

I didn't see any metrics in the docs specific to the inbox or outbox and the only dead letter I see is wolverine-dead-letter-queue. Maybe I'm missing something or the docs could use some additional info? For example, how would I monitor the number of current items in the inbox and outbox using OTEL using the current metric set that's listed in the docs?

"something to tell the time spent in any of these tables so we can track how long messages are living in them?" -- that's the "effective time" metric. Monitor that. The table storage isn't really important per se, what's important is how long you go from publishing to finished.

• Does this work with all transports? The docs make it seem like an nservicebus thing?
• Does this work when both ends are not wolverine apps? Docs make it sound like only if both ends are wolverine?
• As I understand it from the docs this is measure time from publish to delivery to transport so it could include time spent in outbox if that is enabled but wouldn't if not? Personally I think there is value in being able to measure the time spent actually publishing the messages separately from the time a messages is delayed in the outbox. It helps narrow down what could be going wrong more quickly by having separate metrics, IMO. If the publish times are going up it's something between my process and the transport but if my messages are hanging out in outbox for extended periods of time it could be something else keeping the background processing from working smoothly or at all.
• What metric would I track to tell if messages are getting stuck in the inbox and not processing?

[jeremydmiller]

@JarrodJ83 To your point:

"Does this work with all transports? The docs make it seem like an nservicebus thing?" -- that's transport agnostic. The idea is copied from NSB, but nothing to do w/ NSB itself

"Does this work when both ends are not wolverine apps? Docs make it sound like only if both ends are wolverine?", yeah, sorry. Wolverine to Wolverine. Depends on a bit of metadata sent by Wolverine. If your upstream system is publishing the timestamp of when the message was sent from their end, you could map that to Wolverine

The effective time is the time at which the message is completely successful or moved to the DLQ or discarded compared to the time it was sent. So yes, that does include time in the inbox/outbox. But again, in normal operation, Wolverine does not poll against the inbox/outbox tables. "Hanging out in the outbox" would normally only be the time it takes Wolverine in between the publishing and possibly batching or sending inline. Should never be more than maybe a couple dozen milliseconds. The outbox in Wolverine is "store and forward", w/ Wolverine deleting rows when messages are successfully sent. Message are never read from the outbox except in the case of a node going offline w/o being drained or from Wolverine having to latch the sending because maybe Rabbit is down or the actual broker is down.

[jeremydmiller]

See #986. Pull that into the "alternative" inbox

[jeremydmiller]

I don't think this is ever going to get done, and CritterWatch being just about out eliminates the need for this IMO