Is there a way to send data once to a process after it has been started, but before it gets work from the queue? This way i could reduce the overhead i'll have to send to the worker every time i've got stuff to do.
My use case is to calculate distances from ~20 million coordinates against a set of ~1000. My current solution is the send batches (e.g. 5000) combined with the other 1k in a rpc-call. But this, of course, results in me managing a buffer myself, until i have those 5k coordinates ready, and 20% communcation/parsing overhead due to the repeating set of 1k coordinates.
Another solution would be for the workers to fetch the 1k coordinates on startup themselves from an independend resource, but that doesnt sounds correct either.
Is there a way to send data once to a process after it has been started, but before it gets work from the queue? This way i could reduce the overhead i'll have to send to the worker every time i've got stuff to do.
My use case is to calculate distances from ~20 million coordinates against a set of ~1000. My current solution is the send batches (e.g. 5000) combined with the other 1k in a rpc-call. But this, of course, results in me managing a buffer myself, until i have those 5k coordinates ready, and 20% communcation/parsing overhead due to the repeating set of 1k coordinates.
Another solution would be for the workers to fetch the 1k coordinates on startup themselves from an independend resource, but that doesnt sounds correct either.