Root Cause of Task not completing bug

[immangat]

"Investigate the root cause of the bug preventing the harvester task from ending.

This will help fix the root issue instead of relying on a workaround."

[VimanyuAgg]

The celery worker stays alive after performing the scraping. The current workaround is to kill the workers after some time. We see error logs at the end of the celery run - which is the workaround to kill the workers.

[VimanyuAgg]

There is no functional impact.