Document job "design/architecture"

[BigLep]

Done Criteria

https://github.com/FilOzone/dealbot/blob/main/docs/runbooks/jobs.md has been updated to include:

1. Describe the cardinality of a job. My understanding is that there is one DB row per <sp,jobType>.
2. What process creates a job? - For example, walk through through example of a new SP is added to the registry. How does a job get created for that SP?
   • Another example, what happens if an SP's status changes (i.e., is no longer approved). How does that get picked up?
3. What sets "nextRunAt" to a new value?
4. How often is there polling for whether there is a next job to run?
5. What is the maximum number of jobs that run in parallel?
6. Is there any back pressure in the system? If we intend to run a given job every 15 minutes but the job takes 20 minutes, what happens?
7. Critical environment variables for adjusting the system. I assume we'll link to the various environment variables defined in the environment variable doc.
8. Is there any concept of a job run? For example, is there any record in the database that says job for <sp1,dataStorage> ran at YYYY-MM-DDTHH:MM:SS, here is how long it took, here is its status?
9. Link to source code defining a job (e.g., https://github.com/FilOzone/dealbot/blob/master/apps/backend/src/database/entities/job-schedule-state.entity.ts#L4)

Why Important

"Jobs" are a fundamental concept of the system and its an important primitive for maintainers / contributors to have the right mental model for.

Notes

1. Do we simplify things too and just remove "cron" mode? I assume that probalby needs its own issue, but it would be nice to not have to say "this only applies to pg-boss" mode.
2. If expanded with the info above, I think the file could live under https://github.com/FilOzone/dealbot/blob/main/docs rather than in a runbooks subdirectory.
3. Generally for anything in the above, it would be good to reference the file/place that is the source of truth.

[BigLep]

Closed by #259