Fix celery worker exhaustion by isolating Scrapy in a subprocess

[immangat]

Description

Issue Link: N/A

Fixed a bug where Celery workers would hang and exhaust the worker pool.

Type of Change

• Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

• Integration Test

Reproduction/Verification:

• Ran docker-compose up.
• Triggered 10+ consecutive harvesting tasks.
• Verified via Celery logs that all tasks completed successfully (previously hung after ~7 attempts).

Checklist:

• I have checked that my code adheres to the code style of this project.
• I have performed a self-review of my own code.
• I have commented my code, particularly in hard-to-understand areas.
• I have added tests that prove my fix is effective.
• New and existing unit tests pass locally with my changes.