feat: implement I/O-aware RAM fallback router and native string-indexed memory#235
feat: implement I/O-aware RAM fallback router and native string-indexed memory#235dannythevibe wants to merge 1 commit into
Conversation
|
Could you rewrite this against current It's 121 commits behind The RAM/budget path is exactly where this week's OOM work landed (cap_for_ram now refuses to start when the projected peak exceeds available memory). A rebased version will collide with that, so it's worth checking your router still says something the new budget code doesn't. What changed under you, which makes a rewrite cheaper than it sounds:
What would help it land fast: the smallest version that does one thing. A 300–700 line PR touching If you'd rather not, say so and I'll close it with thanks — no hard feelings either way. And if you think I've misjudged and it should go in as-is, push back: I've been wrong twice today already and both times a contributor caught it. |
This PR introduces two major performance features:
I/O-Aware RAM Router: Detects if the model is running on a slow HDD and actively tracks disk latency during the generation loop. If the disk starts bottlenecking, the router falls back to picking experts exclusively from the RAM-resident pool to maintain steady generation speeds.
Native Local Memory: Implemented a lightweight MemoryManager in the API server that logs conversation turns to a local text file and uses fast literal string matching to pull relevant historical context into the system prompt, entirely bypassing the need for heavy RAG databases.