Get rid of Tokio

[dead-claudia]

I'd like to reduce program size, and I'd like for it to be able to run in Wine. I'd also like to eliminate the need for task switching.

• Windows: https://learn.microsoft.com/en-us/windows/win32/winsock/getting-started-with-winsock + I/O completion ports
• Linux: epoll + non-blocking FDs, register each FD for one-shot notification
• macOS: similar to Linux, but using kqueue.

---

Here's what I'm thinking:

Threads:

• One for logging (no polling required, just printf and nanosleep)
• One for epoll/equivalent
• TBD I/O runners (need to benchmark)

Structure:

• Thread pool condvar + mutex, shared by all threads
• I/O queue

Operation:

• Connection creator just creates connections and inserts the created sockets into the I/O check queue. It's dormant most the time and doesn't have much state of its own. It also only exists on the client.
• The I/O queue simulates round robin by, well, being a FIFO queue. Each entry contains an FD and an operation to perform:
  • Read
  • Write
  • Close (pushes Connect back before closing if a client, possibly allowing it to happen concurrently)
  • Connect
  • Accept
• Poll thread invokes epoll (Linux) or equivalent with a large statically-allocated buffer. Results are then pushed into each worker.
• When an I/O thread finishes its task, it will then cycle to the task placed into its queue. If the task returns EWOULDBLOCK, it instead gets (re-)added to the epoll FD with one-shot registration. (At program start, every FD is pushed to the I/O queue.)
• Signals are dropped unconditionally. No signal is blocked.
• Logging will be done on its own thread.
• The FD MPMC queues use a linked deque of page-size ring buffers allocated directly via anonymous memory mapping, to avoid locking even with large numbers of file descriptors. All entries are pushed to the last page and pulled from the first, which inherently minimizes average page count and fragmentation. (This very strongly shows the power of MMUs.)
  • Only one "next" pointer needs reserved in each page.
  • After every batch of pushing, every runner has a notification flag set to true and is subsequently waken.
  • The last emptied page is cached, to avoid memory map thrashing when the size of the buffer exceeds a page but remains otherwise relatively constant. When a page is reused or freed, it's atomically swapped with that and the old pointer reused/freed accordingly.
  • Useful link: https://www.linuxjournal.com/content/lock-free-multi-producer-multi-consumer-queue-ring-buffer