diff --git a/CHANGELOG.md b/CHANGELOG.md index 2681968..591908a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,18 @@ All notable changes to this project are documented in this file. ## [Unreleased] ### Fixed +- **Startup race after a full-server reboot silently disabling a task for days** (GenRTP incident, 2026-07-10 → 2026-07-15). Root causes addressed: + - Per-action `TimeoutMinutes` (persisted since day one but never enforced) is now applied to every execution path: `Executable`/`Batch` child processes and `sqlcmd` run through `RunProcessWithTimeout`, which kills the entire process tree on expiry; the PowerShell in-process timeout now uses the action's configured value instead of a fixed 4 h. A scheduler-level backstop (`action timeout + 2 min`) additionally kills the tracked child processes and releases the task lock if the execution path itself hangs (e.g. output streams never closing). + - Stale-task recovery now actually recovers: `TaskRunner` tracks child processes by PID per task (`KillProcessesForTask`), and `CleanupStaleTasks` no longer resets the running state at 15 minutes (which caused endless re-dispatch into a held lock); instead it alerts, then at `sum of action timeouts + 10 min` it kills the owning process(es), force-releases the lock, and resets the state so the next schedule fires. + - Stale/blocked alerts now repeat every `Reliability/StaleAlertRepeatMinutes` (default 60) with an escalating `ALERT #N` counter while the condition persists; a task that keeps being skipped (lock unavailable) also raises repeating alerts instead of logging ~6,700 silent warnings. + - `ExecuteSqlTask` no longer reports success when `sqlcmd` fails or times out. +- Scheduler tick no longer blocks for up to 30 s per task on lock acquisition: dispatch is offloaded to the thread pool with a pending-dispatch guard against double launch. + +### Added +- SQL Server readiness gate (`SqlReadinessChecker`): before dispatching SQL-dependent tasks (any task with a SQL action, plus names matching `Reliability/SqlGatedTasks`, default `GenRTP,Extracteur`), the scheduler probes SQL Server with a 5 s connection test. While unavailable, the task is deferred to the next tick (schedule untouched) and probes retry with exponential backoff (15 s → 10 min cap), each attempt logged. Login/database errors count as "engine up" so a permissions issue cannot block tasks forever. Probe server: `Reliability/SqlReadinessServer`, falling back to `SqlConfiguration/DefaultServer`. +- Configurable post-reboot startup grace: the service waits `Reliability/StartupDelaySeconds` (default 90) before the first dispatch, and simultaneous due tasks are staggered by `Reliability/TaskLaunchStaggerSeconds` (default 5) instead of all launching in the same second. +- Non-interactive child execution: the service calls `SetErrorMode(SEM_FAILCRITICALERRORS Or SEM_NOGPFAULTERRORBOX Or SEM_NOOPENFILEERRORBOX)` at startup (inherited by children) so Windows hard-error dialogs cannot block a child invisibly, and the PsExec `-i` flag is only used in interactive sessions. +- New `Reliability` settings section (all keys optional, defaults apply): `StartupDelaySeconds`, `TaskLaunchStaggerSeconds`, `SqlReadinessEnabled`, `SqlReadinessServer`, `SqlGatedTasks`, `StaleAlertRepeatMinutes`. - Task deletion not persisting when `LiteTaskService` is running: the service process keeps its own in-memory `_tasks` dictionary and re-writes the full set to `settings.xml` at the end of every execution (via `SaveTasks`). Because the GUI mutated XML directly without telling the service, the service's stale copy resurrected just-deleted tasks (and missed adds/edits) until the next service restart. The GUI now sends `LiteTaskService.CMD_RELOAD_TASKS` (custom service control 128) via `ServiceController.ExecuteCommand` after every task mutation (delete, create, edit, toggle-enable, import), which routes through `LiteTaskService.OnCustomCommand` → `CustomScheduler.ReloadTasks`. `ReloadTasks` evicts entries no longer present in XML (skipping anything currently executing) and refreshes the rest. The notification is best-effort and logs a warning when the GUI user lacks `SERVICE_USER_DEFINED_CONTROL` on the service. - Task deletion not persisting: `XMLManager.DeleteTask` only searched the canonical `LiteTaskSettings/Tasks/Task` path while `GetAllTaskNames`/`LoadTask` accept three layouts, so non-canonical entries survived deletion. `CustomScheduler.SaveTasks` then re-merged the surviving XML entry back into `_tasks` via `GetAllTasks`, resurrecting the task. `DeleteTask` now removes every matching node across supported layouts (under the existing write lock, with atomic save) and `SaveTasks` treats the in-memory `_tasks` dictionary as the source of truth. diff --git a/Class/CustomScheduler.vb b/Class/CustomScheduler.vb index d5979c0..db43e45 100644 --- a/Class/CustomScheduler.vb +++ b/Class/CustomScheduler.vb @@ -27,6 +27,28 @@ Namespace LiteTask Private _lastStaleCleanupTime As DateTime = DateTime.MinValue Private Const STALE_CLEANUP_INTERVAL_SECONDS As Integer = 300 + ' Reliability settings (configurable via the Reliability section of settings.xml) + Private ReadOnly _sqlReadiness As SqlReadinessChecker + Private _sqlReadinessEnabled As Boolean = True + Private _sqlGatedTaskPatterns As String() = {"GenRTP", "Extracteur"} + Private _staggerSeconds As Integer = 5 + Private _staleAlertRepeatMinutes As Integer = 60 + + ' Extra recovery/alerting state: + ' - _pendingDispatch prevents double-dispatch of a task whose (possibly + ' staggered) launch has been scheduled but has not completed yet. + ' - _staleAlertCounts numbers repeated stale/blocked alerts so escalation + ' is visible in the subject line. + ' - _skippedRun* track consecutive scheduler skips (lock unavailable / + ' already running) so a persistently blocked task raises repeated alerts + ' instead of failing silently every 30 seconds. + Private ReadOnly _pendingDispatch As New ConcurrentDictionary(Of String, Boolean) + Private ReadOnly _staleAlertCounts As New ConcurrentDictionary(Of String, Integer) + Private ReadOnly _blockedTaskAlerts As New ConcurrentDictionary(Of String, DateTime) + Private ReadOnly _skippedRunFirstTime As New ConcurrentDictionary(Of String, DateTime) + Private ReadOnly _skippedRunCounts As New ConcurrentDictionary(Of String, Integer) + Private Const STALE_KILL_GRACE_MINUTES As Integer = 10 + ' Memory monitoring properties (configurable via Options > Monitoring) Private _memoryMonitorEnabled As Boolean = True Private _memoryCheckIntervalSeconds As Integer = 300 ' Default: 5 minutes @@ -83,10 +105,14 @@ Namespace LiteTask _errorNotifier = New ErrorNotifier(xmlManager, logger) _dependencyManager = New TaskDependencyManager(logger, xmlManager, Me) _powerShellPathManager = ApplicationContainer.GetService(Of PowerShellPathManager)() + _sqlReadiness = New SqlReadinessChecker(xmlManager, logger) ' Load memory monitoring settings LoadMemoryMonitorSettings() + ' Load startup/recovery reliability settings + LoadReliabilitySettings() + ' Initialize mutex cleanup timer (run every 5 minutes) _mutexCleanupTimer = New Threading.Timer(AddressOf CleanupAbandonedMutexes, Nothing, TimeSpan.FromMinutes(5), TimeSpan.FromMinutes(5)) @@ -264,6 +290,52 @@ Namespace LiteTask End Try End Sub + ''' + ''' Reloads startup/recovery reliability settings from XML. Called on + ''' construction and can be called again after configuration changes. + ''' + Public Sub LoadReliabilitySettings() + Try + Dim settings = _xmlManager.GetReliabilitySettings() + _sqlReadinessEnabled = Boolean.Parse(If(settings.ContainsKey("SqlReadinessEnabled"), settings("SqlReadinessEnabled"), "True")) + Dim gatedTasks = If(settings.ContainsKey("SqlGatedTasks"), settings("SqlGatedTasks"), "") + _sqlGatedTaskPatterns = gatedTasks.Split(","c). + Select(Function(s) s.Trim()). + Where(Function(s) s.Length > 0). + ToArray() + _staggerSeconds = Math.Max(0, Integer.Parse(If(settings.ContainsKey("TaskLaunchStaggerSeconds"), settings("TaskLaunchStaggerSeconds"), "5"))) + _staleAlertRepeatMinutes = Math.Max(5, Integer.Parse(If(settings.ContainsKey("StaleAlertRepeatMinutes"), settings("StaleAlertRepeatMinutes"), "60"))) + _logger.LogInfo($"Reliability settings loaded: SqlReadiness={_sqlReadinessEnabled}, GatedTasks=[{String.Join(", ", _sqlGatedTaskPatterns)}], StaggerSeconds={_staggerSeconds}, StaleAlertRepeatMinutes={_staleAlertRepeatMinutes}") + Catch ex As Exception + _logger.LogWarning($"Error loading reliability settings, using defaults: {ex.Message}") + _sqlReadinessEnabled = True + _sqlGatedTaskPatterns = {"GenRTP", "Extracteur"} + _staggerSeconds = 5 + _staleAlertRepeatMinutes = 60 + End Try + End Sub + + ''' + ''' A task must wait for SQL Server readiness when it contains a SQL action + ''' or when its name matches one of the configured SQL-gated patterns + ''' (covers executables like GenRTP that talk to SQL internally). + ''' + Private Function RequiresSqlReadiness(task As ScheduledTask) As Boolean + If Not _sqlReadinessEnabled Then Return False + + If task.Actions IsNot Nothing AndAlso task.Actions.Any(Function(a) a.Type = TaskType.SQL) Then + Return True + End If + + For Each pattern In _sqlGatedTaskPatterns + If task.Name.IndexOf(pattern, StringComparison.OrdinalIgnoreCase) >= 0 Then + Return True + End If + Next + + Return False + End Function + Public Sub AddTask(task As ScheduledTask) Try If task Is Nothing Then @@ -313,6 +385,11 @@ Namespace LiteTask _taskStates.Clear() _activeMutexes.Clear() _staleTaskAlerts.Clear() + _staleAlertCounts.Clear() + _blockedTaskAlerts.Clear() + _skippedRunFirstTime.Clear() + _skippedRunCounts.Clear() + _pendingDispatch.Clear() _logger.LogInfo("Task states, mutex tracking, and stale alert tracking cleared") End Sub @@ -332,18 +409,38 @@ Namespace LiteTask ' Daily scheduled restart check CheckDailyRestart(now) + Dim dueIndex As Integer = 0 For Each task In _tasks.Values Try ' Skip disabled tasks If Not task.Enabled Then Continue For + If task.NextRunTime > now Then Continue For + + Dim taskState = _taskStates.GetOrAdd(task.Name, New TaskState()) + If taskState.IsRunning Then Continue For + + ' Dependency readiness gate: don't launch SQL-dependent tasks + ' while SQL Server is unavailable (e.g. right after a reboot). + ' NextRunTime is left untouched so the task is retried on the + ' next scheduler tick instead of being marked as executed. + If RequiresSqlReadiness(task) AndAlso Not _sqlReadiness.IsReady() Then + _logger.LogWarning($"Task {task.Name} deferred: SQL Server is not ready yet") + Continue For + End If - If task.NextRunTime <= now Then - Dim taskState = _taskStates.GetOrAdd(task.Name, New TaskState()) + ' TryAdd guards against double-dispatch: the launch below may be + ' staggered past the next scheduler tick. + If _pendingDispatch.TryAdd(task.Name, True) Then + Dim taskToRun = task + Dim launchDelayMs = dueIndex * _staggerSeconds * 1000 + dueIndex += 1 - If Not taskState.IsRunning Then - _logger.LogInfo($"Task due for execution: {task.Name}") - ExecuteTaskAsync(task).ConfigureAwait(False) - End If + _logger.LogInfo($"Task due for execution: {taskToRun.Name}{If(launchDelayMs > 0, $" (staggered by {launchDelayMs \ 1000}s)", "")}") + + ' Offload to the thread pool: lock acquisition in + ' ExecuteTaskAsync can block for up to 30 seconds and must + ' not stall the scheduler tick. + Threading.Tasks.Task.Run(Function() DispatchTaskAsync(taskToRun, launchDelayMs)) End If Catch ex As Exception _logger.LogError($"Error checking task {task.Name}: {ex.Message}") @@ -354,6 +451,24 @@ Namespace LiteTask End Try End Sub + ''' + ''' Runs one scheduled dispatch, applying the optional stagger delay so + ''' simultaneous launches don't all hit SQL Server and network shares in + ''' the same second, then always clears the pending-dispatch flag. + ''' + Private Async Function DispatchTaskAsync(taskToRun As ScheduledTask, launchDelayMs As Integer) As Task + Try + If launchDelayMs > 0 Then + Await Task.Delay(launchDelayMs) + End If + Await ExecuteTaskAsync(taskToRun) + Catch ex As Exception + _logger.LogError($"Error executing task {taskToRun.Name}: {ex.Message}") + Finally + _pendingDispatch.TryRemove(taskToRun.Name, Nothing) + End Try + End Function + ''' ''' Logs memory usage periodically and sends alerts when thresholds are exceeded. ''' @@ -602,24 +717,56 @@ Namespace LiteTask End Try End Sub + ''' + ''' Total configured timeout for a task (sum of its actions' timeouts, since + ''' actions run sequentially). Used to decide when a running task is beyond + ''' recovery-by-timeout and must be forcibly killed. + ''' + Private Function GetEffectiveTimeout(taskName As String) As TimeSpan + Dim task As ScheduledTask = Nothing + If _tasks.TryGetValue(taskName, task) AndAlso task.Actions IsNot Nothing AndAlso task.Actions.Count > 0 Then + Return TimeSpan.FromMinutes(task.Actions.Sum(Function(a) If(a.TimeoutMinutes > 0, a.TimeoutMinutes, 60))) + End If + Return TimeSpan.FromMinutes(60) + End Function + Private Sub CleanupStaleTasks() Try Dim staleTimeout = TimeSpan.FromMinutes(STALE_TIMEOUT_MINUTES) Dim now = DateTime.Now For Each taskEntry In _taskStates - If taskEntry.Value.IsRunning AndAlso - (now - taskEntry.Value.LastStartTime) > staleTimeout Then - _logger.LogWarning($"Cleaning up stale task state: {taskEntry.Key}") - - ' Send email alert if not already sent for this stale instance - SendStaleTaskAlert(taskEntry.Key, taskEntry.Value, staleTimeout) + If Not taskEntry.Value.IsRunning Then Continue For + + Dim taskName = taskEntry.Key + Dim runningFor = now - taskEntry.Value.LastStartTime + If runningFor <= staleTimeout Then Continue For + + ' Long-running task detected. Alert (repeats on a cadence while + ' the condition persists), but do NOT reset the running state: + ' resetting it while the underlying process is still alive only + ' causes the scheduler to re-dispatch into a held lock forever. + _logger.LogWarning($"Stale task detected: {taskName} has been running for {runningFor.TotalMinutes:F1} minutes") + SendStaleTaskAlert(taskName, taskEntry.Value, staleTimeout) + + ' Last-resort recovery: the per-action timeouts should have fired + ' long ago. Kill the owning process(es), release the lock, and + ' reset the state so the next schedule can proceed. + Dim killThreshold = GetEffectiveTimeout(taskName).Add(TimeSpan.FromMinutes(STALE_KILL_GRACE_MINUTES)) + If runningFor > killThreshold Then + _logger.LogError($"Task {taskName} exceeded its recovery threshold ({killThreshold.TotalMinutes:F0} minutes). Forcing recovery.") + + Dim killedCount = _taskRunner.KillProcessesForTask(taskName) + _logger.LogWarning($"Forced recovery of {taskName}: killed {killedCount} process(es)") + + ' Clear the in-process running marker so the abandoned lock + ' can be force-released, then release it. + _taskRunning.TryRemove(taskName, False) + TryCleanupAbandonedLock(taskName, forceRelease:=True) taskEntry.Value.IsRunning = False - taskEntry.Value.LastError = "Task cleaned up due to timeout" - - ' Also remove from mutex tracking - _activeMutexes.TryRemove(taskEntry.Key, Nothing) + taskEntry.Value.LastError = $"Forced recovery after {runningFor.TotalMinutes:F0} minutes (killed {killedCount} process(es))" + _activeMutexes.TryRemove(taskName, Nothing) End If Next Catch ex As Exception @@ -627,17 +774,18 @@ Namespace LiteTask End Try End Sub - ''' Sends an email alert for a stale task, with duplicate prevention + ''' Sends an email alert for a stale task. The alert repeats on a + ''' configurable cadence while the condition persists, with an escalating + ''' alert number in the subject, so a multi-day blockage cannot hide + ''' behind a single day-one email. Private Sub SendStaleTaskAlert(taskName As String, taskState As TaskState, staleTimeout As TimeSpan) Try Dim lastAlertTime As DateTime = Nothing Dim now = DateTime.Now - ' Check if we already sent an alert for this task recently (within the last hour) + ' Rate-limit repeats to the configured cadence If _staleTaskAlerts.TryGetValue(taskName, lastAlertTime) Then - If (now - lastAlertTime).TotalHours < 1 Then - ' Already sent an alert recently, don't spam - _logger.LogInfo($"Skipping duplicate stale task alert for {taskName} (last alert: {lastAlertTime})") + If (now - lastAlertTime).TotalMinutes < _staleAlertRepeatMinutes Then Return End If End If @@ -649,10 +797,12 @@ Namespace LiteTask Return End If + Dim alertNumber = _staleAlertCounts.AddOrUpdate(taskName, 1, Function(k, v) v + 1) + ' Build alert message - Dim subject = $"ALERT: Stale Task Detected - {taskName}" + Dim subject = $"ALERT #{alertNumber}: Stale Task Detected - {taskName}" Dim body As New StringBuilder() - body.AppendLine($"A stale task has been detected and cleaned up:") + body.AppendLine($"A stale task has been detected (alert #{alertNumber} for this incident):") body.AppendLine() body.AppendLine($"Task Name: {taskName}") body.AppendLine($"Started At: {taskState.LastStartTime:yyyy-MM-dd HH:mm:ss}") @@ -665,10 +815,11 @@ Namespace LiteTask body.AppendLine($"Last Error: {taskState.LastError}") End If body.AppendLine() - body.AppendLine("Action Taken: Task state has been reset and mutex released.") + body.AppendLine($"This alert will repeat every {_staleAlertRepeatMinutes} minutes until the task recovers.") + body.AppendLine("If the task exceeds its configured timeout, LiteTask will kill the owning process and release the lock automatically.") body.AppendLine() body.AppendLine("This typically indicates:") - body.AppendLine(" - The task process may be hung or stuck") + body.AppendLine(" - The task process may be hung or stuck (e.g. waiting on SQL Server or a network share)") body.AppendLine(" - An executable may still be running in the background") body.AppendLine(" - The task may have crashed without cleaning up properly") body.AppendLine() @@ -684,13 +835,76 @@ Namespace LiteTask ' Update alert tracking _staleTaskAlerts(taskName) = now - _logger.LogInfo($"Stale task alert sent for {taskName}") + _logger.LogInfo($"Stale task alert #{alertNumber} sent for {taskName}") Catch ex As Exception _logger.LogError($"Error sending stale task alert for {taskName}: {ex.Message}") End Try End Sub + ''' + ''' Records a scheduler skip (lock unavailable or task already running). + ''' A task that keeps getting skipped is effectively blocked: after the + ''' stale threshold, repeated escalating alerts are sent so the condition + ''' cannot persist silently (as happened when ~6,700 skips over 5 days + ''' produced a single email). + ''' + Private Sub RecordSkippedRun(taskName As String) + Try + Dim now = DateTime.Now + Dim firstSkip = _skippedRunFirstTime.GetOrAdd(taskName, now) + Dim skipCount = _skippedRunCounts.AddOrUpdate(taskName, 1, Function(k, v) v + 1) + Dim blockedFor = now - firstSkip + + If blockedFor.TotalMinutes < STALE_TIMEOUT_MINUTES Then Return + + ' Rate-limit repeats to the configured cadence + Dim lastAlertTime As DateTime = Nothing + If _blockedTaskAlerts.TryGetValue(taskName, lastAlertTime) Then + If (now - lastAlertTime).TotalMinutes < _staleAlertRepeatMinutes Then + Return + End If + End If + + Dim notificationManager = ApplicationContainer.GetService(Of NotificationManager)() + If notificationManager Is Nothing Then Return + + Dim alertNumber = _staleAlertCounts.AddOrUpdate(taskName, 1, Function(k, v) v + 1) + + Dim body As New StringBuilder() + body.AppendLine($"Task '{taskName}' is due but keeps being skipped (alert #{alertNumber} for this incident):") + body.AppendLine() + body.AppendLine($"Blocked Since: {firstSkip:yyyy-MM-dd HH:mm:ss}") + body.AppendLine($"Blocked Duration: {blockedFor.TotalMinutes:F0} minutes") + body.AppendLine($"Skipped Runs: {skipCount}") + body.AppendLine() + body.AppendLine("The task lock could not be acquired, which usually means a previous run") + body.AppendLine("is hung and still holding it. LiteTask will attempt automatic recovery") + body.AppendLine("(kill the owning process and release the lock) when the run exceeds its timeout.") + body.AppendLine() + body.AppendLine($"This alert will repeat every {_staleAlertRepeatMinutes} minutes until the task runs again.") + + notificationManager.QueueNotification( + $"ALERT #{alertNumber}: Task Blocked - {taskName} ({skipCount} runs skipped)", + body.ToString(), + NotificationManager.NotificationPriority.High) + + _blockedTaskAlerts(taskName) = now + _logger.LogWarning($"Blocked task alert #{alertNumber} sent for {taskName} ({skipCount} skipped runs over {blockedFor.TotalMinutes:F0} minutes)") + + Catch ex As Exception + _logger.LogError($"Error recording skipped run for {taskName}: {ex.Message}") + End Try + End Sub + + ''' Clears blockage/alert tracking after a successful run + Private Sub ClearBlockedTracking(taskName As String) + _skippedRunFirstTime.TryRemove(taskName, Nothing) + _skippedRunCounts.TryRemove(taskName, Nothing) + _blockedTaskAlerts.TryRemove(taskName, Nothing) + _staleAlertCounts.TryRemove(taskName, Nothing) + End Sub + Protected Overridable Sub Dispose(disposing As Boolean) If Not _disposed Then If disposing Then @@ -838,6 +1052,7 @@ Namespace LiteTask If Not hasLock Then _logger.LogInfo($"Could not acquire lock for task {_task.Name} after {retryCount} retries. Will retry later.") _activeMutexes.TryRemove(_task.Name, Nothing) + RecordSkippedRun(_task.Name) Return False End If @@ -848,6 +1063,10 @@ Namespace LiteTask _activeMutexes(_task.Name) = DateTime.Now _logger.LogInfo($"Lock acquired for task: {_task.Name}") If _taskRunning.TryAdd(_task.Name, True) Then + ' Flow the task name into TaskRunner (AsyncLocal) so child + ' processes started for this run are tracked under this task + ' and can be killed on timeout or stale-lock recovery. + _taskRunner.CurrentTaskName = _task.Name Try ' Get credential if needed Dim credential As CredentialInfo = Nothing @@ -861,7 +1080,25 @@ Namespace LiteTask action.LastRunTime = DateTime.Now Try - Dim success = Await ExecuteTaskAction(action, credential) + ' Backstop timeout: TaskRunner enforces the action's + ' timeout on the child process itself; this outer guard + ' catches the case where the execution path hangs without + ' the child exiting (e.g. output streams never closing), + ' kills the owning process and releases the lock so the + ' next schedule can fire. + Dim actionTimeout = TaskRunner.GetActionTimeout(action) + Dim backstopTimeout = actionTimeout.Add(TimeSpan.FromMinutes(2)) + Dim executionTask = ExecuteTaskAction(action, credential) + Dim finished = Await Task.WhenAny(executionTask, Task.Delay(backstopTimeout)) + + If finished IsNot executionTask Then + action.Status = TaskActionStatus.TimedOut + Dim killedCount = _taskRunner.KillProcessesForTask(_task.Name) + _logger.LogError($"Action {action.Name} of task {_task.Name} exceeded its {actionTimeout.TotalMinutes:F0}-minute timeout. Killed {killedCount} process(es).") + Throw New System.TimeoutException($"Action {action.Name} exceeded its {actionTimeout.TotalMinutes:F0}-minute timeout (killed {killedCount} process(es))") + End If + + Dim success = Await executionTask If success Then action.Status = TaskActionStatus.Completed _logger.LogInfo($"Action {action.Name} completed successfully") @@ -925,17 +1162,20 @@ Namespace LiteTask End Try Finally + _taskRunner.CurrentTaskName = Nothing _taskRunning.TryRemove(_task.Name, False) taskState.LastEndTime = DateTime.Now taskState.IsRunning = False - ' Clear any stale task alert tracking when task completes normally + ' Clear stale/blocked alert tracking now that the task got to run _staleTaskAlerts.TryRemove(_task.Name, Nothing) + ClearBlockedTracking(_task.Name) End Try Return True Else ' Task is already tracked as running in-process (secondary guard) _logger.LogWarning($"Task {_task.Name} is already running in-process, skipping.") + RecordSkippedRun(_task.Name) Return False End If @@ -1144,6 +1384,8 @@ Namespace LiteTask _taskStates.TryRemove(existingName, Nothing) _staleTaskAlerts.TryRemove(existingName, Nothing) _activeMutexes.TryRemove(existingName, Nothing) + _pendingDispatch.TryRemove(existingName, Nothing) + ClearBlockedTracking(existingName) Dim removedLock As Object = Nothing If _taskLocks.TryRemove(existingName, removedLock) Then @@ -1206,6 +1448,8 @@ Namespace LiteTask _taskRunning.TryRemove(taskName, Nothing) _staleTaskAlerts.TryRemove(taskName, Nothing) _activeMutexes.TryRemove(taskName, Nothing) + _pendingDispatch.TryRemove(taskName, Nothing) + ClearBlockedTracking(taskName) Dim removedLock As Object = Nothing If _taskLocks.TryRemove(taskName, removedLock) Then diff --git a/Class/LiteTaskService.vb b/Class/LiteTaskService.vb index 9f4fda0..1eccd37 100644 --- a/Class/LiteTaskService.vb +++ b/Class/LiteTaskService.vb @@ -93,6 +93,18 @@ Namespace LiteTask Private Shared Function CloseHandle(hObject As IntPtr) As Boolean End Function + ' SetErrorMode flags: child processes inherit the error mode, so setting it + ' at service startup prevents Windows hard-error dialogs (missing DLL, disk + ' error, GP fault) from invisibly blocking a child executable that runs in + ' a non-interactive service session. + + Private Shared Function SetErrorMode(uMode As UInteger) As UInteger + End Function + + Private Const SEM_FAILCRITICALERRORS As UInteger = &H1UI + Private Const SEM_NOGPFAULTERRORBOX As UInteger = &H2UI + Private Const SEM_NOOPENFILEERRORBOX As UInteger = &H8000UI + Private Shared Function CloseServiceHandle(hSCObject As IntPtr) As Boolean End Function @@ -212,11 +224,40 @@ Namespace LiteTask Try _logger.LogInfo("LiteTaskService is starting.") + ' Suppress Windows hard-error dialogs for this process and all child + ' processes: under a service there is no desktop to answer them, so a + ' dialog would block the child invisibly and hold the task lock forever. + Try + SetErrorMode(SEM_FAILCRITICALERRORS Or SEM_NOGPFAULTERRORBOX Or SEM_NOOPENFILEERRORBOX) + _logger.LogInfo("Error mode set: hard-error dialogs suppressed for service and child processes") + Catch ex As Exception + _logger.LogWarning($"Could not set process error mode: {ex.Message}") + End Try + _customScheduler.ClearTaskStates() _customScheduler.LoadTasks() + ' Configurable grace period before the first dispatch, so that after a + ' full-server reboot SQL Server and network shares have time to come up + ' before any task is launched. + Dim startupDelaySeconds = 90 + Try + Dim reliability = _xmlManager.GetReliabilitySettings() + Dim configuredDelay As Integer + If Integer.TryParse(reliability("StartupDelaySeconds"), configuredDelay) AndAlso configuredDelay >= 0 Then + startupDelaySeconds = configuredDelay + End If + Catch ex As Exception + _logger.LogWarning($"Could not read startup delay setting, using default {startupDelaySeconds}s: {ex.Message}") + End Try + _serviceTask = Task.Run(Async Function() Try + If startupDelaySeconds > 0 Then + _logger.LogInfo($"Startup delay: waiting {startupDelaySeconds}s before dispatching tasks (post-reboot grace period)") + Await Task.Delay(TimeSpan.FromSeconds(startupDelaySeconds), _cancellationTokenSource.Token) + End If + While Not _cancellationTokenSource.Token.IsCancellationRequested Try _customScheduler.CheckAndExecuteTasks() diff --git a/Class/SqlReadinessChecker.vb b/Class/SqlReadinessChecker.vb new file mode 100644 index 0000000..9a73740 --- /dev/null +++ b/Class/SqlReadinessChecker.vb @@ -0,0 +1,115 @@ +Imports System.Data.SqlClient + +Namespace LiteTask + ''' + ''' Lightweight readiness gate for SQL Server, used by the scheduler to defer + ''' SQL-dependent tasks until the database engine accepts connections (e.g. + ''' right after a server reboot, when the service starts before SQL Server). + ''' Probes are throttled with exponential backoff so an extended outage does + ''' not flood the server or the logs, and a successful probe is cached briefly + ''' so the scheduler tick does not open a connection every 30 seconds. + ''' + Public Class SqlReadinessChecker + Private ReadOnly _xmlManager As XMLManager + Private ReadOnly _logger As Logger + Private ReadOnly _syncRoot As New Object() + + Private _lastResult As Boolean + Private _lastSuccessTime As DateTime = DateTime.MinValue + Private _nextProbeTime As DateTime = DateTime.MinValue + Private _consecutiveFailures As Integer + + Private Const PROBE_CONNECT_TIMEOUT_SECONDS As Integer = 5 + Private Const SUCCESS_CACHE_SECONDS As Integer = 300 + Private Const INITIAL_BACKOFF_SECONDS As Integer = 15 + Private Const MAX_BACKOFF_SECONDS As Integer = 600 + + Public Sub New(xmlManager As XMLManager, logger As Logger) + _xmlManager = xmlManager + _logger = logger + End Sub + + ''' + ''' Server to probe: Reliability/SqlReadinessServer if set, otherwise the + ''' default server from SqlConfiguration. Empty means nothing to probe. + ''' + Private Function GetProbeServer() As String + Try + Dim server = _xmlManager.ReadValue("Reliability", "SqlReadinessServer", "") + If String.IsNullOrWhiteSpace(server) Then + server = _xmlManager.GetSqlConfiguration()("DefaultServer") + End If + Return server + Catch ex As Exception + _logger?.LogError($"[SqlReadiness] Error reading probe server configuration: {ex.Message}") + Return "" + End Try + End Function + + ''' + ''' Returns True when SQL Server accepts connections. While unavailable, + ''' actual probes only happen when the current backoff window has elapsed; + ''' calls in between return the cached negative result immediately. + ''' + Public Function IsReady() As Boolean + SyncLock _syncRoot + Dim now = DateTime.Now + Dim server = GetProbeServer() + + ' No server configured: nothing to probe, never block tasks + If String.IsNullOrWhiteSpace(server) Then + Return True + End If + + If _lastResult AndAlso (now - _lastSuccessTime).TotalSeconds < SUCCESS_CACHE_SECONDS Then + Return True + End If + + If Not _lastResult AndAlso now < _nextProbeTime Then + Return False + End If + + Return Probe(server, now) + End SyncLock + End Function + + ' Connectivity-level SqlException numbers (server unreachable). Any other + ' SqlException (e.g. 18456 login failed, 4060 cannot open database) means + ' the engine answered, so the gate must NOT keep blocking tasks. + Private Shared ReadOnly NetworkErrorNumbers As Integer() = {-1, 2, 40, 53, 258, 1231, 10060, 10061, 11001} + + Private Function Probe(server As String, now As DateTime) As Boolean + Try + _logger?.LogInfo($"[SqlReadiness] Probing SQL Server '{server}' (attempt {_consecutiveFailures + 1})") + Using connection As New SqlConnection($"Server={server};Database=master;Integrated Security=True;TrustServerCertificate=True;Connect Timeout={PROBE_CONNECT_TIMEOUT_SECONDS}") + connection.Open() + End Using + Return MarkReady(server, now) + + Catch ex As SqlException When Not NetworkErrorNumbers.Contains(ex.Number) + ' The server responded but refused this probe (login/database error): + ' the engine is up, which is all the readiness gate needs to know. + _logger?.LogInfo($"[SqlReadiness] SQL Server '{server}' answered with error {ex.Number} ({ex.Message.Trim()}); treating engine as available") + Return MarkReady(server, now) + + Catch ex As Exception + _consecutiveFailures += 1 + Dim backoffSeconds = Math.Min(INITIAL_BACKOFF_SECONDS * Math.Pow(2, Math.Min(_consecutiveFailures - 1, 10)), MAX_BACKOFF_SECONDS) + _nextProbeTime = now.AddSeconds(backoffSeconds) + _lastResult = False + _logger?.LogWarning($"[SqlReadiness] SQL Server '{server}' not ready (probe {_consecutiveFailures} failed: {ex.Message}). Next probe in {backoffSeconds:F0}s") + Return False + End Try + End Function + + Private Function MarkReady(server As String, now As DateTime) As Boolean + If _consecutiveFailures > 0 Then + _logger?.LogInfo($"[SqlReadiness] SQL Server '{server}' is available again after {_consecutiveFailures} failed probe(s)") + End If + _lastResult = True + _lastSuccessTime = now + _consecutiveFailures = 0 + Return True + End Function + End Class +End Namespace diff --git a/Class/TaskRunner.vb b/Class/TaskRunner.vb index 1fe090b..217a700 100644 --- a/Class/TaskRunner.vb +++ b/Class/TaskRunner.vb @@ -1,3 +1,4 @@ +Imports System.Collections.Concurrent Imports System.Collections.ObjectModel Imports System.Data Imports System.Data.SqlClient @@ -26,6 +27,107 @@ Namespace LiteTask Public Event OutputReceived(sender As Object, data As String) Public Event ErrorReceived(sender As Object, data As String) + ' Live child processes per execution context (task name), so the scheduler + ' can locate and kill the owning process when a task hangs past its timeout + ' or its lock goes stale. AsyncLocal flows the task name from the scheduler + ' into the nested async execution path without changing method signatures. + Private ReadOnly _trackedProcesses As New ConcurrentDictionary(Of String, ConcurrentDictionary(Of Integer, Process)) + Private ReadOnly _currentTaskName As New Threading.AsyncLocal(Of String) + Private Const ADHOC_CONTEXT As String = "_adhoc" + Private Const DEFAULT_TASK_TIMEOUT_MINUTES As Integer = 60 + + ''' + ''' Execution context for process tracking. Set by the scheduler before + ''' running a task's actions; flows through awaits into RunProcessWithTimeout. + ''' + Public Property CurrentTaskName As String + Get + Return _currentTaskName.Value + End Get + Set(value As String) + _currentTaskName.Value = value + End Set + End Property + + ''' + ''' Effective hard timeout for a task action. TimeoutMinutes has always been + ''' persisted per action but was never enforced; values <= 0 fall back to + ''' the default so every run has a bounded duration. + ''' + Public Shared Function GetActionTimeout(taskAction As TaskAction) As TimeSpan + Dim minutes = If(taskAction IsNot Nothing AndAlso taskAction.TimeoutMinutes > 0, + taskAction.TimeoutMinutes, + DEFAULT_TASK_TIMEOUT_MINUTES) + Return TimeSpan.FromMinutes(minutes) + End Function + + Private Function TrackProcess(process As Process) As String + Try + Dim key = If(CurrentTaskName, ADHOC_CONTEXT) + Dim bucket = _trackedProcesses.GetOrAdd(key, Function(k) New ConcurrentDictionary(Of Integer, Process)()) + bucket(process.Id) = process + _logger.LogInfo($"Tracking child process {process.Id} (context: {key})") + Return key + Catch ex As Exception + ' Process may have exited before Id could be read; nothing to track + _logger.LogWarning($"Could not track child process: {ex.Message}") + Return Nothing + End Try + End Function + + Private Sub UntrackProcess(key As String, process As Process) + If key Is Nothing Then Return + Try + Dim bucket As ConcurrentDictionary(Of Integer, Process) = Nothing + If _trackedProcesses.TryGetValue(key, bucket) Then + bucket.TryRemove(process.Id, Nothing) + End If + Catch + ' Id unavailable (process never started) - nothing was tracked + End Try + End Sub + + ''' + ''' Kills every tracked child process (entire process tree) started for the + ''' given task. Used by the scheduler's timeout and stale-lock recovery so a + ''' hung executable cannot hold the task lock indefinitely. + ''' + Public Function KillProcessesForTask(taskName As String) As Integer + Dim killed = 0 + Dim bucket As ConcurrentDictionary(Of Integer, Process) = Nothing + If Not _trackedProcesses.TryGetValue(taskName, bucket) OrElse bucket.IsEmpty Then + _logger.LogWarning($"No tracked child process found for task '{taskName}'") + Return 0 + End If + + For Each kvp In bucket + Try + If Not kvp.Value.HasExited Then + _logger.LogWarning($"Killing hung process {kvp.Key} (and children) for task '{taskName}'") + kvp.Value.Kill(entireProcessTree:=True) + killed += 1 + End If + Catch ex As ObjectDisposedException + ' The Process wrapper was disposed but the OS process may live on: kill by PID + Try + Using p = Process.GetProcessById(kvp.Key) + _logger.LogWarning($"Killing hung process {kvp.Key} (by PID) for task '{taskName}'") + p.Kill(entireProcessTree:=True) + killed += 1 + End Using + Catch + ' Process already gone + End Try + Catch ex As Exception + _logger.LogError($"Failed to kill process {kvp.Key} for task '{taskName}': {ex.Message}") + Finally + bucket.TryRemove(kvp.Key, Nothing) + End Try + Next + + Return killed + End Function + Public Sub New(logger As Logger, credentialManager As CredentialManager, toolManager As ToolManager, logPath As String, XMLManager As XMLManager) If logger Is Nothing Then Throw New ArgumentNullException(NameOf(logger)) @@ -129,8 +231,11 @@ Namespace LiteTask EnvironmentVariableTarget.Process) End Using - ' Handle domain credentials - Dim baseArgs = "-accepteula -nobanner -i" + ' Handle domain credentials. The -i (interactive) flag is only + ' valid in a user session: under the service there is no visible + ' desktop, so an interactive child can block invisibly on a dialog. + Dim baseArgs = "-accepteula -nobanner" + If Environment.UserInteractive Then baseArgs &= " -i" If taskAction.RequiresElevation Then baseArgs &= " -h" If credential.Username.Contains("\") Then @@ -167,7 +272,7 @@ Namespace LiteTask .StandardErrorEncoding = Encoding.UTF8 } } - Return Await RunProcess(process) + Return Await RunProcessWithTimeout(process, GetActionTimeout(taskAction)) End Using Finally @@ -194,7 +299,7 @@ Namespace LiteTask .StandardErrorEncoding = Encoding.UTF8 } } - Return Await RunProcess(process) + Return Await RunProcessWithTimeout(process, GetActionTimeout(taskAction)) End Using End If @@ -213,8 +318,11 @@ Namespace LiteTask taskAction.Parameters, $"""{taskAction.Target}"" {taskAction.Parameters}") - ' Build base arguments including domain handling - Dim baseArgs = "-accepteula -nobanner -i" + ' Build base arguments including domain handling. The -i (interactive) + ' flag is only valid in a user session: under the service there is no + ' visible desktop, so an interactive child can block invisibly on a dialog. + Dim baseArgs = "-accepteula -nobanner" + If Environment.UserInteractive Then baseArgs &= " -i" If taskAction.RequiresElevation Then baseArgs &= " -h" If credential.Username.Contains("\") Then @@ -254,7 +362,7 @@ Namespace LiteTask .StandardErrorEncoding = Encoding.GetEncoding(863) } - Return Await RunProcess(process) + Return Await RunProcessWithTimeout(process, GetActionTimeout(taskAction)) End Using Finally @@ -278,7 +386,7 @@ Namespace LiteTask .StandardErrorEncoding = Encoding.GetEncoding(863) } } - Return Await RunProcess(process) + Return Await RunProcessWithTimeout(process, GetActionTimeout(taskAction)) End Using End If @@ -313,9 +421,11 @@ Namespace LiteTask End Try End Function - Private Async Function ExecuteProcessWithOutput(process As Process, Optional verboseMode As Boolean = False) As Task(Of String) + Private Async Function ExecuteProcessWithOutput(process As Process, Optional verboseMode As Boolean = False, Optional timeout As TimeSpan? = Nothing) As Task(Of String) Dim output As New StringBuilder() Dim err As New StringBuilder() + Dim trackKey As String = Nothing + Dim effectiveTimeout = If(timeout, TimeSpan.FromMinutes(DEFAULT_TASK_TIMEOUT_MINUTES)) process.StartInfo.StandardOutputEncoding = Encoding.UTF8 process.StartInfo.StandardErrorEncoding = Encoding.UTF8 @@ -351,7 +461,23 @@ Namespace LiteTask process.Start() process.BeginOutputReadLine() process.BeginErrorReadLine() - Await process.WaitForExitAsync() + trackKey = TrackProcess(process) + + ' Enforce a hard timeout so a hung child (e.g. sqlcmd waiting on an + ' unavailable SQL Server) cannot hold the task lock forever + Using cts As New CancellationTokenSource(effectiveTimeout) + Try + Await process.WaitForExitAsync(cts.Token) + Catch ex As OperationCanceledException + _logger.LogError($"Process exceeded {effectiveTimeout.TotalMinutes:F0}-minute timeout. Killing process tree (PID {process.Id}).") + Try + process.Kill(entireProcessTree:=True) + Catch killEx As Exception + _logger.LogWarning($"Error killing timed-out process: {killEx.Message}") + End Try + Return $"Error: Process exceeded {effectiveTimeout.TotalMinutes:F0}-minute timeout and was killed" + End Try + End Using If process.ExitCode <> 0 Then Dim errorMessage = err.ToString() @@ -373,6 +499,8 @@ Namespace LiteTask Catch ' Suppress errors during cleanup End Try + + UntrackProcess(trackKey, process) End Try End Function @@ -487,7 +615,7 @@ Namespace LiteTask End Try End Function - Public Async Function ExecuteSqlCommandWithSqlCmd(query As String, server As String, database As String, credential As CredentialInfo) As Task(Of String) + Public Async Function ExecuteSqlCommandWithSqlCmd(query As String, server As String, database As String, credential As CredentialInfo, Optional timeout As TimeSpan? = Nothing) As Task(Of String) Try _logger.LogInfo($"Executing SQL command on {server}.{database} using sqlcmd") @@ -523,7 +651,7 @@ Namespace LiteTask .StandardErrorEncoding = Encoding.UTF8 } } - Return Await ExecuteProcessWithOutput(process) + Return Await ExecuteProcessWithOutput(process, timeout:=timeout) End Using Finally @@ -571,8 +699,12 @@ Namespace LiteTask _logger.LogInfo($"Using Server: {server}, Database: {database}") _logger.LogInfo($"Using credentials: {If(credential IsNot Nothing, credential.Username, "None")}") - ' Execute using sqlcmd - Dim result = Await ExecuteSqlCommandWithSqlCmd(sqlContent, server, database, credential) + ' Execute using sqlcmd, bounded by the action's configured timeout + Dim result = Await ExecuteSqlCommandWithSqlCmd(sqlContent, server, database, credential, GetActionTimeout(taskAction)) + If result IsNot Nothing AndAlso result.StartsWith("Error:", StringComparison.OrdinalIgnoreCase) Then + _logger.LogError($"SQL task {taskAction.Name} failed: {result}") + Return False + End If Return True Catch ex As Exception _logger.LogError($"Error in ExecuteSqlTask: {ex.Message}") @@ -634,9 +766,6 @@ Namespace LiteTask End Try End Function - ' Max PS task runtime before forced stop (prevents hung scripts from blocking the scheduler) - Private Const PS_TASK_TIMEOUT_HOURS As Integer = 4 - Public Async Function ExecutePowerShellTask(taskAction As TaskAction, credential As CredentialInfo) As Task(Of Boolean) Try _logger.LogInfo($"Starting PowerShell task execution: {taskAction.Name}") @@ -723,11 +852,13 @@ Namespace LiteTask Dim hasErrors = False Dim result As PSDataCollection(Of PSObject) = Nothing Try + ' Hard cap from the action's configured TimeoutMinutes (default 60) + Dim psTimeout = GetActionTimeout(taskAction) Dim invokeTask = powerShell.InvokeAsync() - Dim timeoutTask = Task.Delay(TimeSpan.FromHours(PS_TASK_TIMEOUT_HOURS)) + Dim timeoutTask = Task.Delay(psTimeout) If Await Task.WhenAny(invokeTask, timeoutTask) Is timeoutTask Then - _logger.LogError($"PowerShell task '{taskAction.Name}' exceeded {PS_TASK_TIMEOUT_HOURS}h timeout. Stopping forcibly.") + _logger.LogError($"PowerShell task '{taskAction.Name}' exceeded {psTimeout.TotalMinutes:F0}-minute timeout. Stopping forcibly.") powerShell.Stop() Return False End If @@ -894,69 +1025,16 @@ Namespace LiteTask _logger.LogInfo($"PowerShell Output: {informationRecord.MessageData}") End Sub - Private Async Function RunProcess(process As Process) As Task(Of Boolean) - Dim output As New StringBuilder() - Dim errors As New StringBuilder() - - ' Named handlers so they can be removed after execution (prevents closure leaks) - Dim outputHandler As DataReceivedEventHandler = Sub(sender, e) - If e.Data IsNot Nothing Then - output.AppendLine(e.Data) - _logger.LogInfo($"Process output: {e.Data}") - RaiseEvent OutputReceived(Me, e.Data) - End If - End Sub - - Dim errorHandler As DataReceivedEventHandler = Sub(sender, e) - If e.Data IsNot Nothing Then - errors.AppendLine(e.Data) - _logger.LogWarning($"Process error: {e.Data}") - RaiseEvent ErrorReceived(Me, e.Data) - End If - End Sub - - Try - AddHandler process.OutputDataReceived, outputHandler - AddHandler process.ErrorDataReceived, errorHandler - - process.Start() - process.BeginOutputReadLine() - process.BeginErrorReadLine() - Await process.WaitForExitAsync() - - If process.ExitCode <> 0 Then - _logger.LogError($"Process failed with exit code: {process.ExitCode}") - _logger.LogError($"Error output: {errors.ToString()}") - Return False - End If - - Return True - - Catch ex As Exception - _logger.LogError($"Error running process: {ex.Message}") - Return False - Finally - RemoveHandler process.OutputDataReceived, outputHandler - RemoveHandler process.ErrorDataReceived, errorHandler - - ' Kill the process if it's still running to prevent orphaned processes - Try - If Not process.HasExited Then - process.Kill(entireProcessTree:=True) - End If - Catch - ' Suppress errors during cleanup - End Try - End Try - End Function - ''' ''' Runs a process with a hard timeout. If the process exceeds the timeout, ''' it is killed (including the entire process tree) and False is returned. + ''' The process is tracked by PID under the current task context so the + ''' scheduler's stale-lock recovery can kill it if this method itself hangs. ''' Private Async Function RunProcessWithTimeout(process As Process, timeout As TimeSpan) As Task(Of Boolean) Dim output As New StringBuilder() Dim errors As New StringBuilder() + Dim trackKey As String = Nothing Dim outputHandler As DataReceivedEventHandler = Sub(sender, e) If e.Data IsNot Nothing Then @@ -981,13 +1059,14 @@ Namespace LiteTask process.Start() process.BeginOutputReadLine() process.BeginErrorReadLine() + trackKey = TrackProcess(process) - ' Enforce a hard timeout so hung scripts cannot hold the task lock forever + ' Enforce a hard timeout so hung executables cannot hold the task lock forever Using cts As New CancellationTokenSource(timeout) Try Await process.WaitForExitAsync(cts.Token) Catch ex As OperationCanceledException - _logger.LogError($"Process exceeded {timeout.TotalHours:F1}h timeout. Killing process tree.") + _logger.LogError($"Process exceeded {timeout.TotalMinutes:F0}-minute timeout. Killing process tree (PID {process.Id}).") Try process.Kill(entireProcessTree:=True) Catch killEx As Exception @@ -1020,6 +1099,8 @@ Namespace LiteTask Catch ' Suppress errors during cleanup End Try + + UntrackProcess(trackKey, process) End Try End Function diff --git a/Class/XMLManager.vb b/Class/XMLManager.vb index 06132b3..ef0cd63 100644 --- a/Class/XMLManager.vb +++ b/Class/XMLManager.vb @@ -982,6 +982,34 @@ Namespace LiteTask End Try End Sub + ''' + ''' Startup/recovery reliability settings. Defaults are chosen for a + ''' shared-reboot scenario where SQL Server and network shares come up + ''' after the service does. + ''' + Public Function GetReliabilitySettings() As Dictionary(Of String, String) + Try + Dim settings As New Dictionary(Of String, String) + settings("StartupDelaySeconds") = ReadValue("Reliability", "StartupDelaySeconds", "90") + settings("TaskLaunchStaggerSeconds") = ReadValue("Reliability", "TaskLaunchStaggerSeconds", "5") + settings("SqlReadinessEnabled") = ReadValue("Reliability", "SqlReadinessEnabled", "True") + settings("SqlReadinessServer") = ReadValue("Reliability", "SqlReadinessServer", "") + settings("SqlGatedTasks") = ReadValue("Reliability", "SqlGatedTasks", "GenRTP,Extracteur") + settings("StaleAlertRepeatMinutes") = ReadValue("Reliability", "StaleAlertRepeatMinutes", "60") + Return settings + Catch ex As Exception + _logger?.LogError($"Error reading reliability settings: {ex.Message}") + Return New Dictionary(Of String, String) From { + {"StartupDelaySeconds", "90"}, + {"TaskLaunchStaggerSeconds", "5"}, + {"SqlReadinessEnabled", "True"}, + {"SqlReadinessServer", ""}, + {"SqlGatedTasks", "GenRTP,Extracteur"}, + {"StaleAlertRepeatMinutes", "60"} + } + End Try + End Function + Public Function GetBackupSettings() As Dictionary(Of String, String) Try Dim settings As New Dictionary(Of String, String)