fix(list_files): replace O(n^2) parent-directory dedup with a set

[sudhanshushekhar10]

Summary

_list_files de-duplicates synthesized parent-directory entries with a linear scan over the
results list, executed once per path component of every file. That makes recursive listings
O(n²).

On large trees the agent session freezes completely: one thread at 100% CPU, no output, no
error, and no timeout — the existing subprocess.run(..., timeout=30) bounds only ripgrep's
enumeration, not the Python loop that processes its output. Because the loop holds the GIL, the
asyncio loop, message bus, and renderer threads are all starved, so the TUI locks up rather than
showing a slow tool.

This replaces the scan with a set. Output is unchanged.

Reproducing

Any list_files call on a directory tree with tens of thousands of files after ignore
filtering. list_files defaults to recursive=True and is typically the first tool an agent
reaches for, so this usually lands on the first tool call of a session.

The severe case is launching code-puppy from a home directory that looks like a project.
_list_files guards against recursing $HOME:

if context is not None and is_likely_home_directory(directory) and recursive:
    if not is_project_directory(directory):
        recursive = False

but is_project_directory() returns true if any of package.json, pyproject.toml,
Cargo.toml, .git, Makefile, requirements.txt, setup.py, go.mod, Gemfile,
pom.xml, build.gradle, CMakeLists.txt, or composer.json is present. A single stray
~/package.json from an accidental npm install disables the guard. In the case that led to
this PR, a 53-byte package.json exposed a 560,987-file recursive walk.

The hang is not limited to the launch directory — the model supplies the directory argument,
so it can wedge a session by listing a large path even when code-puppy was started somewhere
small.

py-spy dump of a hung process (captured on 0.0.531, where this code sat at lines 278–279;
it is unchanged on current main at lines 347–348, with list_files calling in at 1145):

Thread 0x176E93000 (active+gil)
    <genexpr> (code_puppy/tools/file_operations.py:279)
    _list_files (code_puppy/tools/file_operations.py:278)
    list_files (code_puppy/tools/file_operations.py:762)
    run (anyio/_backends/_asyncio.py:1002)

Root cause

code_puppy/tools/file_operations.py, in _list_files():

for i in range(len(path_parts)):
    partial_path = os.sep.join(path_parts[: i + 1])
    # Check if we already added this directory
    if not any(
        f.path == partial_path and f.type == "directory"
        for f in results
    ):
        results.append(ListedFile(path=partial_path, type="directory", ...))

rg --files returns files only, so the intermediate directory entries are synthesized here. The
duplicate check scans all of results, which grows as the walk proceeds — O(k) for the k-th
entry, repeated per path component. Total work is O(n²), with every comparison a Python-level
attribute read on a Pydantic model.

Measured scaling

Synthetic trees, before vs after (measured on stock 0.0.676/0.0.728; the function is unchanged
on current main):

files	before	after	speedup
1,000	0.11s	0.03s	3x
5,000	1.76s	0.12s	14x
10,000	7.01s	0.20s	35x
20,000	28.64s	0.45s	63x

Runtime quadruples as the file count doubles. Extrapolating from 20k, the 560,987-file case
needs several hours of CPU — indistinguishable from a permanent hang.

The fix

Track already-added directory paths in a set and test membership against it: O(1) instead of
O(n), making the listing O(n) overall.

     results = []
+    # Synthesized parent directories already added to ``results``. Membership is
+    # checked once per path component of every file, so this has to be O(1);
+    # rescanning ``results`` made the loop O(n^2) and hung large listings.
+    seen_dir_paths = set()
     directory = resolve_path(directory)
@@
                                 # Check if we already added this directory
-                                if not any(
-                                    f.path == partial_path and f.type == "directory"
-                                    for f in results
-                                ):
+                                if partial_path not in seen_dir_paths:
+                                    seen_dir_paths.add(partial_path)
                                     results.append(

The set only needs the synthesized parents. In the recursive branch every entry originates from
rg --files, so entry_type is always "file" there — I verified this rather than assuming it,
and deliberately did not add a seen_dir_paths update at the main append site: it would be
unreachable in practice (only a race between rg's enumeration and the later stat could produce a
directory there, with a one-line cosmetic duplicate as the worst outcome), and no test could
exercise it.

The non-recursive branch is untouched; it does not use this de-duplication path.

Tests

Adds TestListFilesParentDirectorySynthesis to
tests/tools/test_file_operations_coverage.py — 5 tests using tmp_path, following the existing
conventions in that file (small hand-built trees, no timing assertions, milliseconds to run):

• shared parents are not duplicated
• all ancestors of a deep path are present, exactly once, in order
• sibling branches sharing a prefix each appear once
• files at mixed depths are all listed
• a parent referenced by files at multiple depths is synthesized exactly once

The de-duplication itself had no coverage before (an existing test covers that nested
directories appear in a recursive listing, but nothing exercised the duplicate check).

On what these tests do and don't do. They pin the behaviour of the parent-directory
synthesis, which is what this change puts at risk — the fix alters complexity, not output, so the
real hazard is silently changing a listing. They do not fail on the pre-fix code, because the
old and new implementations produce identical output. A guard against reintroducing the quadratic
scan would have to assert on complexity, which means either wall-clock timing (machine-dependent,
flaky in CI) or counting operations through white-box instrumentation. Both seemed worse than the
code comment for a repo whose test suite is deterministic throughout. Happy to add one if you'd
prefer.

Verification

• Output equivalence: old and new implementations loaded side by side and run against the
  same directories — ListFileOutput.content byte-identical on code_puppy/, tests/, the repo
  root, and synthetic trees at 1k/5k/10k/20k files.
• Full suite: 7,316 passed, 28 skipped, 1 xpassed. (Three integration modules fail to collect without
  pexpect / pytest-asyncio; identical on unmodified HEAD.)
• Lint: ruff check reports 27 pre-existing issues on these two files both before and after —
  none introduced. ruff format --check clean.

[thomwebb]

Reviewed this properly — checked out the branch into a clean worktree, built a venv, installed ripgrep, and verified the claims rather than just reading the diff.

The bug is real and nasty. _list_files synthesizes parent-directory entries (rg --files only returns files) and deduped them with a linear scan of the growing results list per path component — hidden O(n²). Since the loop holds the GIL, this isn't just slow, it's a silent TUI freeze with no timeout catching it (the existing subprocess.run(timeout=30) only bounds ripgrep, not the Python post-processing). The ~/package.json-defeats-the-home-dir-guard repro is a plausible real-world trigger, and list_files being the typical first tool call of a session makes this a rough first impression.

Verification I did myself:

• Tests: 75/75 pass in tests/tools/test_file_operations_coverage.py (including the 5 new ones), 607/607 across all of tests/tools/.
• Ruff: checked out HEAD~1 and diffed lint output myself — same 27 pre-existing findings before and after, nothing new introduced. ruff format --check clean.
• Performance, independently reproduced with my own synthetic tree generator (different from the one in the PR description): new code scales roughly linearly (1k→0.06s, 4k→0.18s, 8k→0.30s) vs. old code trending super-linear (1k→0.09s, 4k→0.45s, 8k→1.39s). Output (line counts) identical between old and new at every size, confirming this is a pure complexity fix with no behavior change.

One nuance, already correctly called out in the PR description: the old dedup scanned the whole results list, so it would've also caught a duplicate if a directory entry ever landed via the main append site (only reachable through a TOCTOU race where rg --files output resolves to a directory). The new seen_dir_paths set only tracks synthesized entries, so that theoretical race could produce a one-line cosmetic duplicate the old code would've suppressed. Agree with the author's assessment that this is unreachable in practice and not a real regression — checked the surrounding code myself to confirm.

Small, correct, well-tested, and unusually honest about its own edge cases (including why a timing-based regression test wasn't added. LGTM.