Skip to content

feat: implement I/O-aware RAM fallback router and native string-indexed memory#235

Open
dannythevibe wants to merge 1 commit into
JustVugg:devfrom
dannythevibe:main
Open

feat: implement I/O-aware RAM fallback router and native string-indexed memory#235
dannythevibe wants to merge 1 commit into
JustVugg:devfrom
dannythevibe:main

Conversation

@dannythevibe

Copy link
Copy Markdown

This PR introduces two major performance features:

  1. I/O-Aware RAM Router: Detects if the model is running on a slow HDD and actively tracks disk latency during the generation loop. If the disk starts bottlenecking, the router falls back to picking experts exclusively from the RAM-resident pool to maintain steady generation speeds.

  2. Native Local Memory: Implemented a lightweight MemoryManager in the API server that logs conversation turns to a local text file and uses fast literal string matching to pull relevant historical context into the system prompt, entirely bypassing the need for heavy RAG databases.

@JustVugg
JustVugg changed the base branch from main to dev July 16, 2026 17:29
@JustVugg

Copy link
Copy Markdown
Owner

Could you rewrite this against current dev? I'm keeping it open rather than closing it, but it can't land in its present shape.

It's 121 commits behind dev. On glm.c, which changes several times a day, that isn't a rebase — it's a rewrite, and doing it for you would mean rewriting your design decisions without understanding why you made them. So I'd rather ask.

The RAM/budget path is exactly where this week's OOM work landed (cap_for_ram now refuses to start when the projected peak exceeds available memory). A rebased version will collide with that, so it's worth checking your router still says something the new budget code doesn't.

What changed under you, which makes a rewrite cheaper than it sounds:

What would help it land fast: the smallest version that does one thing. A 300–700 line PR touching glm.c needs a maintainer who can run it, and for most platforms here that maintainer doesn't exist. A focused change with a test the CI can execute gets reviewed in an hour.

If you'd rather not, say so and I'll close it with thanks — no hard feelings either way. And if you think I've misjudged and it should go in as-is, push back: I've been wrong twice today already and both times a contributor caught it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants