🔄 Refactor: Additional Refactoring of Jobs

[MrAdder]

Summary

This PR introduces consistent queue configuration, rate limiting, overlap protection, and improved observability across MSHIP and Training jobs. It also updates Horizon to ensure the new dedicated queues are actively consumed by workers.

The goal is to:

• Prevent duplicate or overlapping processing per account/check.
• Protect external services with rate limiting.
• Improve reliability with retries/backoff.
• Standardise logging and failure handling.
• Explicitly route jobs to dedicated service queues.

---

MSHIP Sync Jobs

Updated the following jobs:

• SyncToCTS
• SyncToDiscord
• SyncToHelpdesk
• SyncToMoodle

Changes

• Strongly typed Account property.
• Added:
  • $tries = 3
  • $backoff = 30
  • Dedicated $queue per service (cts, discord, forums, helpdesk, moodle)
• Added structured start/complete logging with consistent context.
• Implemented failed(Throwable $exception) logging.
• Added middleware:
  • RateLimitedWithRedis('<service>-sync')
  • WithoutOverlapping($accountId) with:
    • releaseAfter(5)
    • expireAfter(120)

This ensures:

• Per-account sync work is serialised.
• External APIs are protected.
• Failures are clearly logged.
• Jobs are isolated onto dedicated queues.

---

Training Jobs

Updated:

• ActionWaitingListRetentionCheckRemoval
• SendWaitingListRetentionCheck
• UpdateAccountWaitingListEligibility

Changes

• Added:
  • $tries = 3
  • $backoff (30 for retention jobs, 15 for eligibility)
  • Dedicated queues:
    • training-retention
    • training-eligibility
• Added structured start/complete logging.
• Added failed(Throwable $exception) logging.
• Added middleware:
  • RateLimitedWithRedis(...)
  • WithoutOverlapping(...) with scoped keys:
    • retention-check:{id}
    • waiting-list-account:{id}
    • account:{id}

Behaviour Improvements

• Retention removal job:
  • Explicitly fails if notification cannot be sent.
  • Prevents accidental removal if email delivery fails.
• Retention send job:
  • Uses DB transaction.
  • Rolls back and fails cleanly on notification error.
• Eligibility job:
  • Logs number of waiting lists processed.
  • Serialises updates per account.

---

Discord Listener Adjustment

SetupDiscordUser:

• Removed explicit ->onQueue('default').
• Now allows SyncToDiscord to self-route to the dedicated discord queue.

This aligns job routing with the new queue architecture.

---

Horizon Configuration

Updated config/horizon.php:

Queues now include:

[
  'high',
  'default',
  'discord',
  'helpdesk',
  'moodle',
  'cts',
  'training-retention',
  'training-eligibility',
]

This ensures dedicated service queues are actually consumed by workers in both production and local.

---

Tests Added

MSHIP

• SyncJobConfigurationTest
  • Verifies:
    • tries
    • backoff
    • queue name
    • middleware presence (RateLimitedWithRedis, WithoutOverlapping)

Training

• TrainingJobConfigurationTest
  • Validates queue config and middleware.
• Updated notification failure test:
  • Asserts structured logging context is present.

---

Result

This PR:

• Standardises queue configuration across services.
• Improves resilience and observability.
• Prevents duplicate execution and race conditions.
• Introduces clear separation of external integrations via dedicated queues.
• Ensures Horizon is correctly configured to process new queues.
• Adds test coverage for configuration guarantees.
• Keeps legacy integrations (Forums) aligned with current architecture.

This lays groundwork for safer horizontal scaling and improved operational visibility.

[MrAdder]

Removed the Refactor for the Forums as its now been removed

[MrAdder]

@kristiankunc or @CLC0609 These work fine locally and will not break the site, due to restrictions on my side I can not test the full functionality to Helpdesk or moodle so thats all that will need testing