Optimization for asyncio queue? only make a single AsyncQueueListener

[Preston-Landers]

Hey @ZhuYuJin - you recently contributed in this area of the asyncio logging queue. I was reviewing this code a little more and it seemed to me that it might be unnecessary to create separate instances of these objects for each logger object:

AsyncQueueListener ->
   QueueHandler -> 
       queue.Queue  

As far as I can tell there's no harm in having a single AsyncQueueListener handle all the loggers within that process. Or am I maybe missing something? I wonder if you could perhaps try out this branch and let me know what you think.

It wouldn't be a major speed boost, but may some minor memory savings if there are a lot of loggers configured.

[ZhuYuJin]

Hi, Preston. Could we use a boolean variable to switch between multiple listeners and a single listener. I have to submit logs to local agent with UDP socket, which may cause backlog of log in a single listener. Besides, a single listener may cause significant lock competition while we have too many logger objects.

[Preston-Landers]

Thanks @ZhuYuJin. I'm planning to study this area more because I don't understand it very well. However what you said makes sense suggests there would be minimal value to this change.

I am curious about your use case though, if you can provide any more info. Are your application processes only sending logging messages over UDP to an agent/server, and that agent is the one using concurrent-log-handler? Or are the app processes both writing with concurrent-log-handler and also sending message over UDP as a separate thing?

[ZhuYuJin]

Our application uses concurrent log handler which writes local log files, and a remote log handler which sends logging messages over UDP to an agent. The agent will gather logs and send archive to remote server.
Besides, the application invokes function setup_logging_queues to write log in separate threads for each handler.