[💡 FEATURE REQUEST]: Add producer-only mode

[Apcenuu]

Plugin

Jobs

I have an idea!

Add a “producer-only” mode for the Jobs plugin that declares pipelines without starting any consumer loops or worker pools. The goal is to publish tasks to existing queues via the Jobs RPC/client while consumption is handled by another service/cluster and avoid idle worker pools and unneeded resource usage in producer-only services.

jobs:
  consume: [] #or consume: false
  pipelines:
    outcome:
      driver: amqp
      config:
          ...

[rustatian]

Hey @Apcenuu 👋🏻
Thank you for the feature request 👍🏻 I'll think about how to properly implement this mode.

[rustatian]

I thought about this feature, and probably the best way to solve that is to create a separate plugin. The case is, I don't want to overcomplicate a JOBS plugin with a lot of if/else (like some RPC operations should be blocked with error, like resume/pause, all worker pool operations, and so on) for an additional mode. Let's say - jobs_producer, w/o pool options and better batching for a specific purpose - push only mode.

[rauanmayemir]

I don't follow, this is already possible.
Not adding a pipeline name in the jobs.consume array makes it producer-only.

Unless there is an issue if you only have 1 pipeline?

[rustatian]

Pipeline is a separate concept. It does not relate to a worker pool. If you don't have any pipelines, the pool will still be available and operations like resume/pause, add/remove workers from the workers pool, and metrics (and some other) are allowed even w/o any pipelines.
If we enable producer-only mode, which, as I mentioned, is unrelated to pipelines but rather to the worker pool, we need to apply restrictions on several operations in the jobs plugin, which I don't like as an architectural concept.

[denis-korolev]

Hi, can you make separate consumer/producer modes?
I have a situation, my rabbit user have only read permittions.
I cant write message, but i dont need it.
I want only read it.
JOBS module need all setting exchange, quee. But i have only quee.

[rustatian]

Hi @denis-korolev 👋🏻
There is no need for a separate consumer mode, as it is available by default. Just don't use RPC to push JOBS.

[rauanmayemir]

@rustatian amqpjobs sends declare commands whether exchange already exists or not. so if a user does not have permissions, pipeline will likely fail to start.

[rauanmayemir]

@denis-korolev what about ACK or requeue? I don't think read-only is possible. Not to mention, requeue-ing with delay means creating new queues with TTL/DLQ.

[rustatian]

amqpjobs sends declare commands whether exchange already exists or not. so if a user does not have permissions, pipeline will likely fail to start.

The declare command is idempotent in the case of AMQP, and it is necessary to verify that the specified parameters in the configuration are set correctly for the existing exchange.
The JOBS plugin is a coordinator, and the declare/ttl/deal-letter-queues stuff is unrelated to it.