Upright::ProbeResult.cleanup_stale has same queue and priority as probe jobs

[ttilberg]

After integrating the recent changes from 0.3, I got an alert that my http probes were no longer reporting. I found that it was because there was a massive queue of cleanup jobs that made their way in via the newer Upright::ProbeResult.cleanup_stale job.

uptime(prod):003>   SolidQueue::Job.where(queue_name: "default").count
=> 610739
uptime(prod):004>   SolidQueue::ReadyExecution.count
=> 384871
uptime(prod):005>   SolidQueue::Job.where(queue_name: "default").group(:class_name).count
=> {"ActiveStorage::PurgeJob" => 596297, "Upright::ProbeCheckJob" => 14452}
uptime(prod):006>

This is my state about an hour after the cleanup_stale task fired off, and none of the probe checks have run.

I think it might be useful to schedule these cleanup tasks with lower priority or low-priority queue.

[ttilberg]

Actually, as my log above shows, it's the ActiveStorage::PurgeJob that needs to be addressed, not the Upright one. The ActiveStorage job can be adjusted with an initializer:

# config/initializers/active_storage.rb 
Rails.application.config.active_storage.queues.purge = :low_priority

However, in order to process this queue separately and with lower priority, we also have to adjust the queue.yml file in our host app:

# config/queue.yml
  workers:
    - queues: [default, solid_queue_recurring, low_priority]
      threads: 3
      processes: <%= ENV.fetch("JOB_CONCURRENCY", 1) %>
      polling_interval: 0.1

I don't know if there's any particular changes to this project that make sense to account for this, both are configurations in the host app.

[lewispb]

Hey @ttilberg thanks for reporting this one.

Looks like we never ran into this outselves because we've been running a custom cleanup regularly before 0.3 shipped.

Now your initial cleanup backlog has cleared the hourly one is fast enough to not cause any issues on the default queue. I don't think we need to make any future changes, do we?

[ttilberg]

I ended up switching to multiple queues configured as noted above, so I can't measure the effects of blocking. Given that the job is scheduled to run every hour, it's true that it wouldn't have nearly as much to do and I suspect would be mostly fine. That this issue is now documented with a potential fix in the comments I think is probably enough for now. Other people will find this and chirp up, and you could decide the importance of a change once there's more info. It's notable that this affects v < 0.3, which won't be many people and there's a good understanding of "weird stuff" in such versions.

[lewispb]

I'm seeing some small delays, causing a few probes to fail when the cleanup job runs. I believe this is due to sqlite contention. Going to look into switching to delete_all.