Support native DLQ to be used automatically for failures

[andreasohlund]

This branch introduces an option to change the normal NServiceBus failure
path from MoveToError to native Azure Service Bus dead-lettering.

NOTE: All NServiceBus fault metadata is preserved

[andreasohlund]

@danielmarbach @DavidBoike I had a chat with @ramonsmits, and he brought up the idea of always using the DLQ for all failures, since that is the more efficient way to do things. Since we apply all faults metadata, this would be fully equivalent to the default way of "sending to the error queue", with the only difference that the DLQ source, reason, and description would also be set.

Thoughts?

[andreasohlund]

I looked into what MT and Wolverine do in this area:

MT

By default, MassTransit moves faulted messages to its _error queue, but on Azure Service Bus it can be
configured to move faulted messages to the native ASB DLQ instead, while still generating
fault events. It also has ConfigureDeadLetterQueueDeadLetterTransport() for skipped messages
to use the native DLQ instead of _skipped.

Wolverine

Wolverine uses the native ASB DLQ only for inline endpoints. For buffered and
durable endpoints, failed messages go to a separate dead-letter queue (default
wolverine-dead-letter-queue, configurable with ConfigureDeadLetterQueue(...)) rather than the
ASB subqueue.

[andreasohlund]

This does unlock use of the azure portal for rudimentatry error management

[andreasohlund]

@ramonsmits spoke to @danielmarbach, and we believe this is the middle ground for now, where dlq auto forwarding on created queues is a separate feature from "move all fault to the error queue"

[udidahan]

If users make use of this, does that mean that they can no longer manage/retry those failed messages through ServicePulse/Control?

[andreasohlund]

@udidahan ServiceControl can and should still be used ( see https://github.com/Particular/NServiceBus.Transport.AzureServiceBus/pull/1373/changes#diff-decced24b3ffe2c2e4693fa12eda01ec063149f6b3c5a763c99f0346765f8e9cR16 which shows this flow (errorspy would be equivalent to SC)

The scenario that would be recommended is the following:

1. Automatic DLQ forwarding is turned on for all endpoints: This means that all DLQ messages will end up in the central error queue
2. (Optionally) turn this feature on since it would be more performant and result in slightly lower costs for the user, since there wouldn't be a "copy and send" operation to get messages into the error queue, but rather a "native reject and DLQ" which is assumed to be more efficient
3. Manage errors centrally in ServicePulse

This means that no matter if the service bus service itself (e.g., due to max delivery count being met), NServiceBus (due to e.g., headers being corrupted), or the user themselves causes a message to be dead lettered, all failures would be manageable via the Platform (unlike today, where dlq's are up to the users to manage separately)

[danielmarbach]

I would even add it makes the platform more useful

[udidahan]

Is there some documentation that would be worth updating with this information?
Other customer communication?

[andreasohlund]

Is there some documentation that would be worth updating with this information?
Other customer communication?

Definitely, we are working on the docs, I will ping you FYI once the PR is ready for final review

We'll talk internally about some dedicated communcation as well