Turn large folders of text and code into clean, token-sized context for AI tools.
LMTokenCook reads a project or document folder, counts tokens, builds a file map, and writes manageable text chunks that are easier to feed into conversational AI tools in order.
Try the browser app: https://lmtokencook.dropshockdigital.com
Copying a large project into an AI chat is unreliable: uploads may be rejected, content can be split in the wrong places, and the model may answer before receiving the full context. LMTokenCook makes that handoff predictable without pretending to expand a model's actual context window.
- Reads nested folders of text and source files.
- Skips common build output, hidden folders, and binary image/executable files.
- Counts tokens with
cl100k_basetokenization. - Splits content on line boundaries at a configurable token limit.
- Generates an optional project structure and token map.
- Adds optional ordered context headers so a chat can wait for all parts.
- Writes directly to an output folder in supported browsers or downloads a ZIP fallback.
The primary React workflow processes selected files inside your browser. The chosen folder is not uploaded by that workflow. Your browser still needs explicit permission to read the input and, where supported, write the output.
Treat generated chunks as copies of the source material: review them before sharing, and do not send private code, credentials, client data, or regulated information to an AI service unless that use is authorized.
git clone https://github.com/DropShock-Digital/LMTokenCook.git
cd LMTokenCook/src/ui
npm ci
npm run devOpen the local URL printed by Vite. Chromium-family browsers provide the fullest folder read/write experience; other browsers use the ZIP download fallback.
cd src/ui
npm ci
npm run buildpython -m venv .venv
source .venv/bin/activate
pip install -r src/server/requirements.txt
PYTHONPATH=. python -m pytest tests/The Python server is retained for compatibility and experiments; the browser-native React workflow is the primary product path.
| Path | Purpose |
|---|---|
src/ui/ |
Browser-native React application |
src/ui/src/lib/chunker.ts |
Token counting and line-aware chunking |
src/ui/src/lib/fs-handler.ts |
Permissioned local folder access |
src/server/ |
Optional Python compatibility/API path |
tests/ |
Python regression tests |
assets/ |
Brand and architecture artwork |
- LMTokenCook does not increase an AI model's context window.
- Token presets for third-party products are estimates and may change.
- Binary formats such as images, executables, and arbitrary office documents are not parsed by the browser workflow.
- You are responsible for reviewing output order, completeness, confidentiality, and suitability for the target model.
See CONTRIBUTING.md. Please report security concerns through SECURITY.md, not in a public issue.
LMTokenCook is available under the MIT License.
Built by DropShock Digital and Steven Seagondollar.