Parallel queue processing doesn't work

[unicoder88]

Hi @BlackbitDevs @Jan-Walther-Basilicom !

DataDirector version 3.9.36 - latest at this moment.

We're struggling with queue not picking up new dataports in queue while very-long-running dataport is still processing.

Steps to reproduce:

1. Put to queue first long running job

   $queue = new \Blackbit\DataDirectorBundle\model\Queue();
   $queue->create([
       'command' => 'dd:complete 8',
       'triggered_by' => 'Result callback function of command ...',
   ]);

2. Put to queue second job

   $queue = new \Blackbit\DataDirectorBundle\model\Queue();
   $queue->create([
       'command' => 'dd:complete 258',
       'triggered_by' => 'Result callback function of command ...'
   ]);

I'm running queue in debug right now, and I can clearly see that while first queue item process is running, queue array is not rebuilt at all:

XDEBUG_TRIGGER=PHPSTORM bin/console data-director:process-queue -v

Demo video with issue being reproduced -

data-director.queue.demo.mp4

Because our dataports can run as long as 24 hours, everything else stalls, and this is hurting business a lot.

Please review :)

[olenakitova]

Hi @unicoder88 thank you for your request, we are on it and will get back to you as soon as possible.

Sincerely,
Blackbit Team

[unicoder88]

Thanks @olenakitova ! For now we developed a patch that hopefully fixed the issue for us, you may find it useful as a starting point.

DataDirector-3.9.36-queue-processor-check-new-queue-items.patch

[Jan-Walther-Basilicom]

Hi @unicoder88,

thanks for your suggestion. We have already implemented this feature for version 3.10 as it is quite a big change in queue processing.