[I18N-1560] Fix Quartz from getting into a blocked state if ScheduledJobListener throws

[mattwilshire]

Summary

Quartz aborts all remaining listener notifications for a job the moment one listener throws.

ScheduledJobListener.jobWasExecuted persists the job's completion status (success/failure) back to the database - if it
throws (e.g. a transient DB error), that notification is dropped and the job's row is left stuck in IN_PROGRESS / BLOCKED state forever, even though it actually finished. It then never gets picked up again by the scheduler.

Solution

The solution here is to ensure nothing in the jobWasExecuted method throws which stops Quartz from sending the "job has finished" notification to the db. This PR ensures any exceptions are captured in the listener and does retries with exponential backoff.

Concern

Another concern was if the db is actually down for something like maintenance (e.g version upgrade) even if we don't throw in the jobWasExecuted, wouldn't Quartz not be able to send the notification to the db saying the "job has finished" also causing the equivalent issue with the job being forever in a BLOCKED state?

The answer to this is Quartz will retry every 15 seconds by default to report the job status back to the db infinitely, more info here under the org.quartz.scheduler.dbFailureRetryInterval property: https://www.quartz-scheduler.org/documentation/quartz-2.2.2/configuration/ConfigMain.html

Testing

All of this was tested locally by stopping the local MySQL instance during job execution to simulate the blocked state. The retries take place and Quartz successfully unblocks itself when the db comes back up.