[linux] saturate disk (3-10gb/s depending on hardware), become compute bound.

[withinboredom]

Summary

GitHub ate my pull request

Pull request creation failed. Validation failed: Body is too long (maximum is 65536 characters)

... so I lost the benchmarks, and I don't have time to run the full gauntlet again.

With this PR, I bring over our commercial fork's disk loader from Swytch, and provide disk saturation through io_uring, Linux only. Tok/s becomes nearly 100% independent of cache size, and will saturate your entire disk and CPU cores. Combined with the better eviction algorithm of #223, I see ~0.4-0.6 tok/s with 12 cores. I tested it on a 60-core machine (Digital Ocean droplet) and saw ~1.5-2.2 tok/s.

Here's a quick run with chat that I still had in my scrollback (60 core machine):

root@ubuntu-c-60-intel-ams3:~/colibri/c# DIRECT=1 URING=1 PIPE_WORKERS=64 ./coli chat --ram 64 --repin -1 --topp 8 --topk 8

     ▄▀▀▀▄  ▄        colibrì v1.0
  ▄▄▄▄▀▀▀▀▄▀▀        tiny engine, immense model
      ▀▀▀▀▀▀▀        GLM-5.2 · 744B MoE · int4 · streaming CPU
        ▀▀▀▀         chat · glm52 · ram 64GB · topp 8.0
          ▀
  ──────────────────────────────────────────────────────────
  type and press Enter · :more continues · :reset clears memory · :q exits

  ╭────────────────────────────────────────────────────────────────────────────────────────────────╮
  │ › Tell me a story about a fairy princess who falls down a well. Keep it short, sardonic, and   │
  │   hilarious.                                                                                   │
  ╰────────────────────────────────────────────────────────────────────────────────────────────────╯

  ◆ colibrì
  TIERS 0 0 19456 0.00 0.00



  The Princess in the Well

  Princess Elara Glitterwing-Moondancer (yes, that was her actual legal name, and yes, the other fairies made fun of her for the hyphen) was walking through the Enchanted Forest on a Tuesday afternoon in May, because of course it was Tuesday, because nothing interesting ever happened on Tuesdays, and she was about to change that.

  She didn't.

  What she did was trip on a perfectly ordinary root and fall into an abandoned well on the property of one Gareth Blackthorn, a man who had never seen a fairy in his life and had, quite frankly, been minding his own business for the last forty-seven years.

  "Oh, for the love of—" Elara started.

  "Shut up down there," said Gary, because of course his name was Gary. "I just got this well sealed up. You're ruining my Tuesday."

  "You can't just leave me down here!"

  "Why not?"

  "Because I'm a fairy princess!"

  "Right," Gary said. "And I'm a Gary."

  He did not, in fact, help her out.

  ---

  *Gary went home, made a sandwich, and lived sardonically ever after.*
  └─ 250 tok · 1.53 tok/s · hit 57% · RSS 53.4 GB · 163s

And the 12 core machine:

root@beta ~/code/colibri/c # DIRECT=1 URING=1 PIPE_WORKERS=12 ./coli chat --ram 32 --repin -1 --ngen 256 --topp 8 --topk 8

     ▄▀▀▀▄  ▄        colibrì v1.0
  ▄▄▄▄▀▀▀▀▄▀▀        tiny engine, immense model
      ▀▀▀▀▀▀▀        GLM-5.2 · 744B MoE · int4 · streaming CPU
        ▀▀▀▀         chat · glm52 · ram 32GB · topp 8.0
          ▀
  ──────────────────────────────────────────────────────────
  type and press Enter · :more continues · :reset clears memory · :q exits

  ╭────────────────────────────────────────────────────────────────────────────────────────────────╮
  │ › Tell me a story about a fairy princess who falls down a well. Keep it short, sardonic, and   │
  │   hilarious.                                                                                   │
  ╰────────────────────────────────────────────────────────────────────────────────────────────────╯

  ◆ colibrì
  TIERS 0 409 19047 0.00 7.74
  Princess ElaraSparkle of the Glittering Glen was not having a particularly magical Tuesday.

  It started, as most catastrophic Tuesdays do, with a wardrobe malfunction. Specifically, her wings—pure, spun-sugar filaments of gossaverous joy—had developed a sentient grudge against her left shoe.

  She was walking near the old wishing well, trying to remember if she’d fed the fairy-dust dragon that lived in the dungeon beneath the well, when suddenly her left shoe—spiteful little bastard—untied itself.

  "Shoe!" she cried, in a voice as clear and melodious as a silver bell. "Shoe, you rebellious knave! How dare you betray me?"

  The shoe, naturally, said nothing, because it was a shoe.

  It did, however, conspire with gravity and a slick patch of moss to send her tumbling headfirst down the dark, dank abyss.

  Down, down, down she fell, fetching a pail of well-water on her head, ruining her tiara and her day.

  "Well, well, well," she muttered, wringing her sparkling skirts. "This is a fine kettle of fish—"

  "—that I've never met before!"
  └─ 256 tok · 0.44 tok/s · hit 33% · RSS 25.3 GB · 578s

And it ran a blog post in 10 minutes... https://gist.github.com/withinboredom/a1b6bd2a7e54206a8ff8d7a393eaf74e

Validation

• make -C c check
• CUDA changes were tested with make -C c cuda-test (if applicable)
• Performance claims include hardware, commands, and repeatable measurements

Compatibility

• The default CPU build remains dependency-free
• No model files, generated binaries, or benchmark artifacts are included