Skip to content

Cannot remove taskId from the queue shard conductor.test.QUEUE.HTTP.a #94

@manan164

Description

@manan164

Hi @apanicker-nflx @kishorebanala ,
We are facing a lot of these warnings in conductor.

cannot add 2b5a87f2-c97f-4294-b883-57869f5fd65d from the queue shard conductor.test.QUEUE.HTTP.a
cannot remove b3ce517b-eeef-474f-94e9-3468e6b0927b from the queue shard conductor.test.QUEUE._deciderQueue.a
cannot add b3ce517b-eeef-474f-94e9-3468e6b0927b from the queue shard conductor.test.QUEUE._deciderQueue.a

We are running Redis cluster setup. Also, enabled workflow execution locking a few days back but that did not help much.
We also see the deciderQueue getting piled up and we run Sweeper. But running Sweeper also does not help much because sweeper also does pop from redis and I see that in pop itself these exception comes.
Can you please help us with this?
Does it cause any harm in running workflows?
We also see PENDING_WORKFLOWS set also gets piled up and we terminate all old workflows based on the workflow creation time.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions