Needle

Helps you find the needle in the haystack.

Find any file on your machine, instantly — a whole-machine file-search MCP for AI agents. Windows / NTFS.

needle reads the NTFS MFT (Master File Table) directly and keeps a warm in-memory index that is refreshed incrementally from the USN Journal. It exposes a single fast_glob MCP tool so a coding agent (Claude Code, etc.) can locate any file on the entire machine — every NTFS drive, millions of files — in well under a millisecond, instead of walking the filesystem.

Filenames and paths only — not file contents. Use Grep/ripgrep for content.

Why Needle exists

Most file-search tools optimize search inside one project: directory traversal, fuzzy matching, ranking — tuned for a human in an editor who already knows roughly where they are.

An agent has a different need: it often has to find things across the whole machine with no prior knowledge of where they live — a config under C:\Users, an SDK in Program Files, a sibling repo on another drive. Walking the tree for that is slow and burns context. needle indexes every NTFS volume at once by reading the MFT, so a whole-disk lookup is as cheap as an in-memory hash scan — sub-millisecond, even across millions of files.

Needle is the one you want when the agent's search space is "the computer," not "the repo."

Example agent workflows

Things an agent can do in milliseconds with fast_glob that are slow or impossible with a project-scoped, traversal-based Glob:

• "Find every MP3 scattered across my drives and move them into D:\Music." fast_glob("**/*.mp3") returns every match on every NTFS volume instantly; the agent then organizes them. A built-in Glob can't even see outside the project.

• "Open that tax PDF from last year — it has 2024 in the name." fast_glob("**/*2024*.pdf") across the whole machine finds it in <1 ms, no "which folder did I save it in?" hunting.

• "Security sweep: list every private key and .env on this box." fast_glob("**/*.pem"), fast_glob("**/*.key"), fast_glob("**/.env") — whole-machine, in milliseconds, instead of crawling every directory.

• "Inventory every project on this machine." fast_glob("**/Cargo.toml"), fast_glob("**/package.json"), fast_glob("**/*.sln") to map all repos/solutions across drives at once.

• "I have my résumé saved in five places — find them all." fast_glob("**/*resume*") / fast_glob("**/*résumé*") surfaces every copy machine-wide so the agent can dedupe.

• "Reclaim space: where are all the node_modules / target build dirs?" fast_glob("**/node_modules"), fast_glob("**/target") finds build junk across every repo on every drive in one shot.

Because Needle returns absolute paths from a warm whole-machine index, the agent spends its turn acting on the files rather than walking the tree to find them.

Architecture

Reading the MFT requires administrator rights, but an MCP client launches its servers non-elevated. So Needle splits into two roles across that boundary:

needle serve   (ADMIN, long-running daemon)
  ├─ builds the MFT index per drive on first query
  ├─ one watcher thread per drive BLOCKS on FSCTL_READ_USN_JOURNAL and applies
  │  each change the instant NTFS records it (Everything-style; 0 idle CPU)
  └─ answers queries over loopback TCP 127.0.0.1:48923

needle mcp     (non-elevated, launched by the agent / MCP client)
  └─ exposes the `fast_glob` MCP tool, forwards queries to the daemon

The daemon exists purely to cross the privilege boundary — it is the only part that needs admin. The MCP frontend is a thin, stateless forwarder. Installed as a Windows service (needle service install), the daemon runs as LocalSystem and auto-starts at boot, so the privilege boundary is crossed once at install time and never again — the agent's non-elevated MCP process just connects to it.

Install

Prebuilt binary — grab the latest needle-vX.Y.Z-x86_64-pc-windows-msvc.zip from Releases, unzip, done. Each release ships a .sha256 to verify the download.

Scoop

scoop bucket add dedsec https://github.com/Dedsec-Xu/scoop-bucket
scoop install needle

Cargo (builds from source; needs the MSVC toolchain)

cargo install --git https://github.com/Dedsec-Xu/needle

After installing, register the daemon as a service once — it prompts for elevation (UAC) automatically, so any shell works:

needle service install

Build (from source)

cargo build --release

Run

1. Install the daemon as a service (once). It runs as LocalSystem and auto-starts at every boot — no manual launch, no recurring UAC prompt:

   needle service install      # prompts for elevation (UAC) once

   Remove it later with needle service uninstall.

   Prefer not to install a service?

   Run the daemon manually instead (must stay open, re-run after each reboot) from an elevated shell:

   needle serve

2. Register the MCP server with your agent. Needle speaks plain MCP over stdio, so any MCP-capable agent can use it — the command is always needle.exe mcp. A project-scoped .mcp.json is included; pick your agent below.

   Claude Code

   claude mcp add needle -- "D:\\path\\to\\needle\\target\\release\\needle.exe" mcp

   Or drop it into .mcp.json (project) / ~/.claude.json (global):

   {
     "mcpServers": {
       "needle": {
         "command": "D:\\path\\to\\needle\\target\\release\\needle.exe",
         "args": ["mcp"]
       }
     }
   }

   Codex (OpenAI Codex CLI)

   Add to ~/.codex/config.toml:

   [mcp_servers.needle]
   command = "D:\\path\\to\\needle\\target\\release\\needle.exe"
   args = ["mcp"]

   Hermes (NousResearch Hermes Agent)

   Add to the mcp_servers section of Hermes' config.yaml:

   mcp_servers:
     needle:
       command: "D:\\path\\to\\needle\\target\\release\\needle.exe"
       args: ["mcp"]

   OpenClaw

   Add to ~/.openclaw/openclaw.json under mcp.servers (or use openclaw mcp add):

   {
     "mcp": {
       "servers": {
         "needle": {
           "command": "D:\\path\\to\\needle\\target\\release\\needle.exe",
           "args": ["mcp"]
         }
       }
     }
   }

   Any other MCP client

   Point your client at the stdio command below — that's the whole contract:

   command: D:\path\to\needle\target\release\needle.exe
   args:    ["mcp"]
   

3. The fast_glob tool becomes available. To make the agent prefer it over the built-in Glob, add to your CLAUDE.md:

   To find files by name or path anywhere on this machine, use the fast_glob MCP tool instead of the built-in Glob — it is far faster and sees every drive. Use Grep for file-content search.

CLI

Once the service (or daemon) is running, query it with needle find — it just forwards to the daemon over loopback, so it is fast and needs no admin:

# query the running daemon (no admin; the service answers from its warm index)
needle find "**/*.rs" --root "D:\path\to\project" --max-results 50

# query the whole machine (no root scope)
needle find "**/appsettings.json"

# directories only
needle find "**/node_modules" --kind dir

# the 20 most recently modified PDFs on the machine
needle find "**/*.pdf" --sort mtime --order desc --max-results 20

# the largest log files in a project
needle find "**/*.log" --root "D:\proj" --sort size --order desc

# case-sensitive match
needle find "**/README.md" --case-sensitive

Flags (--kind, --case-sensitive, --sort, --order) work on both needle find and needle query.

If the daemon isn't running, needle query does a one-shot in-process build instead — handy for ad-hoc use, but it must itself be elevated and pays the full index-build cost each time:

# one-shot query (builds the index in-process; requires admin)
needle query "**/*.rs" --root "D:\path\to\project" --max-results 50

# benchmark a full index build for a drive
needle index C

Benchmark

Run the built-in benchmark (elevated) to time the index build and a set of whole-volume queries on your own machine:

./bench.ps1            # benchmark drive C (self-elevates)
./bench.ps1 D          # benchmark drive D

It prints a ready-to-paste Markdown table: the one-time index-build time plus the per-query latency for several whole-volume globs. Queries run against the warm in-memory index, which a per-drive watcher thread keeps live by blocking on the USN Journal — any file created, deleted, renamed, or moved shows up in the very next query, with no rescan and no polling delay.

Measured results

On a drive with 2,265,224 indexed entries (full index built once in ~4.3 s):

query (whole volume)	matches	time
**/*.rs	601	0.85 ms
**/Cargo.toml	25	0.20 ms
**/*.exe	928	0.89 ms
**/*.dll	13,932	12.8 ms
**/package.json	10,104	26.6 ms

Extension-pinned globs hit the inverted index and are answered in well under a millisecond. Latency for the last two scales only because the benchmark returns every match (max_results = 1,000,000); the fast_glob MCP tool defaults to max_results = 200, so even very broad globs return sub-millisecond in practice.

For comparison, before the inverted index every query was a full ~2.3M-entry scan at ~360 ms — the extension index is a ~400x speedup on the common cases.

vs what your agent searches with today

Coding agents find files with their built-in Glob/Grep tools — ripgrep-style directory traversal. Two things make that a poor fit for an autonomous agent:

1. It walks the tree on every call. No persistent index; each search re-crawls the filesystem and the cost grows with the tree.
2. It can't see past the working directory. The agent is blind to anything outside the project root — other repos, SDKs in Program Files, configs under C:\Users. For those, traversal isn't slow, it simply returns nothing.

Needle replaces both: a warm MFT index over the whole machine, answered from memory. Same whole-drive search — every *.rs on a 2.27M-file drive, 601 matches, identical results for every tool — measured end-to-end with demo/compare.ps1:

tool	how it works	time	vs Needle
Needle	NTFS MFT index	8.4 ms	baseline
ripgrep (what Glob/Grep run on)	directory walk	8811 ms	1049x slower
PowerShell Get-ChildItem	directory walk	4733 ms	563x slower
cmd dir /s /b	directory walk	4894 ms	583x slower
fd (parallel)	directory walk	667 ms	79x slower
es.exe (Everything)	NTFS MFT index	25.9 ms	3x slower

The tool your agent already uses — ripgrep-based traversal — is ~1000x slower here, and that's only for files inside the project; whole-machine lookups it can't do at all. Needle even edges out Everything's own CLI (both read the MFT, but Needle answers from a warm in-process index). Times are end-to-end wall-clock (incl. ~5–7 ms process startup); Needle's pure in-index query is sub-millisecond. Reproduce with demo/compare.ps1.

fast_glob tool parameters

param	default	meaning
pattern	—	glob matched vs the path relative to root
root	""	scope to this directory; empty = whole machine (all NTFS volumes)
max_results	200	cap on returned paths
respect_gitignore	true	apply root/.gitignore, always skip .git
kind	"any"	file, dir, or any
case_sensitive	false	case-sensitive pattern matching
sort	"none"	none (fastest), name, mtime, or size
order	"asc"	asc or desc

Patterns are matched against forward-slash-normalized paths (case-insensitive unless case_sensitive), so **/*.swift, src/**/*.rs, and **/Cargo.toml all behave as expected.

Sorting (recent / largest)

sort=none is the default and keeps the sub-millisecond streaming path — use it whenever order doesn't matter. sort=mtime / sort=size answer "most recent" and "largest" intents: the index finds the candidates, then Needle lazily stats them to fill size/mtime, sorts locally, and returns the top max_results. Pair with order=desc:

// the 20 newest screenshots anywhere on the machine
fast_glob({ "pattern": "**/*.png", "sort": "mtime", "order": "desc", "max_results": 20 })

// the biggest log files in a project
fast_glob({ "pattern": "**/*.log", "root": "D:\\proj", "sort": "size", "order": "desc" })

Metadata sorting stats up to 5000 matched candidates; beyond that the result is flagged sort_approximate (top-k over an arbitrary subset) rather than scanning millions of files. By design, size/mtime are not stored in the index — Needle stays a search primitive, not a metadata database.

Limitations

• Windows + NTFS only. The MFT/USN approach has no equivalent on ext4/APFS/etc.; non-NTFS volumes are skipped.
• The daemon requires administrator privileges.
• Extension-pinned globs use the inverted index; patterns without a fixed extension (e.g. Makefile, foo*, **/x/**) fall back to a full scan with a cheap basename pre-filter. Parallelizing that fallback (rayon) is a future win.
• Query latency scales with the number of returned matches (each match reconstructs its full path), so keep max_results modest for broad globs.

License

MIT