Skip to content

RabbitMQ: dropped connection is never re-established, all publishes fail until restart #998

Description

@LendritIbrahimi

Describe the bug
RabbitMQQueue dials the AMQP connection once on init in queue_rabbitmq.go. gocloud's rabbitpubsub re-opens channels but not connections, so once the connection dies, every publish from that process fails with pubsub (code=FailedPrecondition): Exception (504) Reason: "channel/connection is not open" until someone restarts the pod.
Connections happen to die in normal operation: managed RabbitMQ (CloudAMQP, Amazon MQ) restarts nodes for maintenance and upgrades routinely, plus missed heartbeats and network blips.

To Reproduce

  1. Run any Outpost service against RabbitMQ
  2. Drop the connection from the broker side: rabbitmqctl close_all_connections "test" (or just restart the broker)
  3. Publish anything

Every publish now fails with the 504 above until the process fully restarts.

Expected behavior
The connection is re-established after a drop, like other AMQP clients handle it, and publishing resumes without a process restart.

Additional context
During our testing:

  • staging: the API service's deliverymq publisher wedged over a weekend. Every publish failed with failed to enqueue delivery task, so nothing was delivered at all until a manual restart.
  • production: one delivery pod's logmq publisher wedged for six days. Deliveries kept working (HTTP was fine) but every attempt log from that pod was lost failed to publish attempt log. The delivery message is ACKed even when the log publish fails, so those attempt records are gone permanently: missing delivery history, and every failed delivery left an orphaned retry task spinning forever on the no prior attempt found in logstore path (the retrymq limit & DLQ #663 loop).

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions