processUnack not called after re-establishing connection to Redis DynoQueue

[manjurad]

I have a Netflix conductor server along with Dynomite DB that is running a workflow where the client doesn't respond and the timeout kicks in via sweeper functionality the task times out and is re-scheduled.

But if I take the Dynomite server down temporarily and bring it back, then timeout of tasks stop working. When Dynomite server is not reachable I see logs/traceback about error in running sweeper which is as expected, once connection is re-established those errors stops but sweeper doesn't pick up any workflows to sweep.

Based on debug logs and redis commands issued it looks like the unack queue has some entries but those unack entries are not processed.
I have tried dyno-queue with version release 2.0.13 and see this behavior.

Any ideas/comments on how to fix this?

[manjurad]

Any help here? Is it possible we are missing exception handling and the thread exits when we lose connection to the queue?

dyno-queues/dyno-queues-redis/src/main/java/com/netflix/dyno/queues/redis/v2/RedisPipelineQueue.java

Line 628 in 5089a7c

try {

I see we have added exception handling here

dyno-queues/dyno-queues-redis/src/main/java/com/netflix/dyno/queues/redis/RedisDynoQueue.java

Line 1378 in ad6c57f

} catch (Exception e) {