Skip to content

feat(search): add path: substring filter#1

Closed
dkattan wants to merge 18 commits into
masterfrom
feat/path-filter
Closed

feat(search): add path: substring filter#1
dkattan wants to merge 18 commits into
masterfrom
feat/path-filter

Conversation

@dkattan

@dkattan dkattan commented Jun 25, 2026

Copy link
Copy Markdown
Owner

Summary

Add a path: search filter that keeps items whose full absolute path contains the argument as a substring. Multiple path: filters combine with AND, each narrowing the result set further.

This enables queries like:

main.js path:Ayla path:repos

to find main.js files whose path contains both Ayla and repos.

Changes

  • cardinal-syntax: new FilterKind::Path variant, registered as "path", given scope-filter priority (0) in the optimizer so multiple path: filters narrow the search space first.
  • search-cache: evaluate_path_filter does case-aware substring matching against node_path, intersecting with the incoming base set.
  • Docs: new 4.4 Path substring filter subsection in doc/pub/search-syntax.md with examples; section numbering updated.
  • CHANGELOG: entry under Unreleased.

Validation

cargo check -p cardinal-syntax -p search-cache
cargo test -p cardinal-syntax            # 63 passed
cargo test -p search-cache --test path_filter   # 8 passed
cargo test -p search-cache               # 86 passed
cargo fmt --all -- --check               # clean
cargo clippy -p cardinal-syntax -p search-cache --all-targets   # clean

Notes

  • Draft PR targeting this fork (dkattan/cardinal) only — not upstream cardisoft/cardinal — until the feature is verified working end-to-end.
  • Matching respects the UI case-sensitivity toggle (case-insensitive lowercases both needle and path).
  • A leading / in the argument is trimmed so path:/Ayla behaves like path:Ayla.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

dkattan and others added 18 commits June 25, 2026 04:55
@dkattan
feat(search): add path: substring filter
e6f580e 
Add a path: filter that keeps items whose full absolute path contains
the argument as a substring. Multiple path: filters combine with AND,
each narrowing the result set further (e.g. main.js path:Ayla path:repos).

- cardinal-syntax: FilterKind::Path variant, registered as "path", given
  scope-filter priority (0) in the optimizer so multiple path: filters
  narrow the search space first.
- search-cache: evaluate_path_filter does case-aware substring matching
  against node_path, intersecting with the incoming base set.
- Tests: syntax coverage plus search-cache integration covering single
  and multiple path: filters, case sensitivity, and leading-slash trim.
- Docs: new 4.4 subsection in search-syntax.md with examples.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@dkattan
perf(search): make path: filter allocation-free
2434973 
Walk the ancestor chain in place and check each interned &'static str
name for the needle instead of materializing a PathBuf per node. The
old approach called node_path (which allocates a Vec of segments,
reverses, and joins) for every node in the index; the new hot loop is
pure pointer-chasing over cached names.

This also refines the semantics slightly: path: now matches the needle
against individual path components rather than the joined path string,
which is more precise and still supports the main.js path:Ayla
path:repos use case.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@dkattan
perf(search): use name pool index for path: filter
ae3988d 
Instead of scanning every node and walking its ancestor chain, search
the NamePool for names containing the needle, fetch matching nodes from
the NameIndex, and expand their descendants. This mirrors how *.ext
queries leverage the index.

Benchmarks (warm cache, /opt/homebrew corpus):
  path:repos          62.9ms ->  4.5ms  (14x faster)
  path:Ayla path:repos 60.6ms ->  8.0ms  ( 8x faster)
  main.js path:repos   68.6ms ->  9.4ms  ( 7x faster)

Also add path: queries to the criterion benchmark suite.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@dkattan
feat(search): add FlatIndex data structure
a12ddd7 
FlatIndex stores filesystem entries in a sorted Vec with full paths
stored directly (not reconstructed from parent chains). Includes:
- FlatEntry: path, name (last segment), metadata
- FlatNameIndex: BTreeMap<name, Vec<index>> for *.ext/word search
- prefix_range: O(log n) binary search for parent:/infolder: queries
- insert/remove/remove_prefix for FS event maintenance
- 10 unit tests covering prefix ranges, name lookups, insert/remove

This is the foundation for replacing the tree-based SlabNode/FileNodes
index with a flat structure where path: is as fast as *.ext.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@dkattan
feat(fswalk): add walk_flat for flat entry collection
40dff7b 
Add walk_flat() that emits (PathBuf, Option<NodeMetadata>) tuples
sorted by path, instead of building a tree of Node structs. Each entry
has its full absolute path available directly from the directory entry.

This is the data source for the FlatIndex — no parent-chain
reconstruction needed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@dkattan
feat(search): build FlatIndex alongside tree in SearchCache
b00737f 
Add FlatIndex to SearchCache, built from the slab after tree
construction so indices match. The flat index stores full paths sorted
for O(log n) prefix queries (for future parent:/infolder: optimization).

The path: filter still uses the name-pool approach (4.7ms) since
PATH_POOL.search_substr on full paths is slower than NAME_POOL on
filenames. The flat index is available for scope filter optimization.

Also: walk_flat in fswalk, PATH_POOL global, flat_index module with
FlatEntry/FlatIndex/FlatNameIndex, 10 unit tests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@dkattan
perf(search): use flat index prefix_range for path: descendants
31b35b8 
Replace all_subnodes tree walk with flat index prefix_range for
descendant expansion in the path: filter. Paths are read from the
flat entry directly (O(1)) instead of walking the parent chain.

Benchmark (warm cache, /opt/homebrew):
  path:repos  4.7ms -> 4.5ms  (marginal; name pool search dominates)
  *.rs        3.3ms           (baseline)

The ~1.2ms gap is inherent: path:repos matches ancestor names and
must expand descendants, while *.rs matches the file's own name with
no descendant expansion.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@dkattan
fix(search): maintain flat index on FS events
1558b03 
Update push_node and remove_node to keep the flat index in sync
when files are created, deleted, or renamed after initial indexing.
Without this, path: would return stale results for dynamically
changed filesystems.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@dkattan
fix(search): filter base set in-place for path: to avoid hangs
5ad3f01 
When a base set already exists (e.g. from a prior path: filter or word
match), filter it in-place by checking each node's path against the
needle instead of eagerly expanding all descendants of matching
directories. This prevents hangs on queries like
'main.js path:Ayla path:repos' where expanding all descendants of
every 'Ayla' directory could traverse the entire filesystem.

The no-base path still uses the name pool + prefix range approach
for standalone path: queries.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@dkattan
fix(search): don't build flat index from persisted cache
2967a8c 
Building the flat index from the slab at cache load time called
node_path() (parent-chain walk) for every node, causing the app to
hang on 'Updating' with large caches (403MB cache = millions of nodes).

Now the flat index is left empty when loading from disk. The path:
filter falls back to node_path() per-candidate when the flat index is
not populated. For queries like 'main.js path:repos', the base set
from 'main.js' is small, so the fallback is fast.

Removed build_flat_index_from_slab (no longer used).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@dkattan
feat(app): bump cache version, add granular status messages
d9bd807 
- Bump LSF_VERSION from 6 to 7 so the old 403MB cache is automatically
  ignored, forcing a fresh filesystem walk (no migration needed).
- Add statusMessage field to StatusBarUpdate for granular progress
  messages during the walk (e.g. 'Walking filesystem… 12345 items',
  'Indexing… 100 dirs, 500 files').
- Thread statusMessage through the frontend: IPC types, useFileSearch
  hook, useAppWindowListeners, StatusBar component.
- StatusBar now displays the granular message when present, falling
  back to the lifecycle label otherwise.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@dkattan
fix(search): simplify path: filter to linear scan with cancellation
00bae76 
Replace the name-pool + descendant-expansion approach with a simple
linear scan over all nodes (or the base set). Each node's path is
checked for the needle substring. This:

1. Fixes the hang on 'path:repos' — the old code called all_subnodes
   which recursively walked entire subtrees (e.g. /Users/darrenkattan/
   source/repos contains hundreds of thousands of files) and silently
   swallowed cancellation signals.
2. Ensures proper cancellation — filter_nodes checks is_cancelled_sparse
   and propagates cancellation, so typing a new query aborts the old one.
3. Simplifies the code — no more separate no-base/base paths, no
   evaluate_path_filter_no_base function.

The linear scan is O(N) with O(1) per-node cost (path lookup from flat
index or parent-chain walk). For 'main.js path:repos', the base set
from 'main.js' is small, so the scan is fast.

Add 5 cancellation/large-tree tests verifying correctness and that
cancelled searches return promptly.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@dkattan
test(e2e): add xa11y-based end-to-end tests for Cardinal
2ea15d0 
Add a new e2e-tests crate that drives the running Cardinal desktop
app via macOS accessibility APIs using xa11y. Tests verify:

- App launches and shows search input
- *.js search returns results
- path:repos search does not hang
- Changing search query dismisses the spinner (cancellation works)
- main.js path:Ayla path:repos completes without hanging

Also add 5 Rust integration tests (e2e_search_flow) that verify the
search/cancel flow at the search-cache level, and 5 path_filter_cancel
tests that verify cancellation on large trees.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@dkattan
fix(e2e): use correct xa11y selectors for Tauri accessibility tree
42c67e5 
The Tauri/WebKit accessibility tree exposes the search input as a bare
text_field (no accessible name) and status text as static_text with the
display string in the value attribute. Updated selectors accordingly:

- Search input: text_field (bare role, only one in the window)
- Results text: static_text[value*=result] for the status bar

All 5 e2e tests pass against the running Cardinal app:
- app_launches_and_shows_search_input
- search_star_js_returns_results
- search_path_filter_does_not_hang
- changing_search_dismisses_spinner
- main_js_path_ayla_path_repos_query

Also add a dump_tree example for debugging the accessibility tree.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@dkattan
fix(a11y): add aria-label to search input
5214b66 
The search input had no accessible name, making it invisible to
assistive technologies. Added aria-label with i18n key
'search.aria.searchInput' ('Search input') threaded through App →
SearchBar. Updated all 15 locale files.

E2e tests now use the proper selector text_field[name*="Search"]
instead of a bare text_field fallback.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@dkattan
perf(search): populate flat index during walk, optimize path: filter
8430573 
Build the flat index (26M entries with full paths) during the initial
filesystem walk instead of leaving it empty. This eliminates the
O(N×depth) node_path() fallback that caused 10s+ hangs on path: queries.

Key changes:
- construct_node_slab_name_index now accumulates FlatEntry records
  with full paths during the tree-to-slab conversion, so the flat
  index is built with zero extra passes over the filesystem
- FlatEntry gains a slab_index field; FlatIndex gains a slab_map
  for O(log n) slab→entry lookups
- evaluate_path_filter scans flat index entries directly (O(1) path
  access per entry) instead of calling search_empty + node_path
- path_match_ci: case-insensitive substring match without allocation
- Search draining in background.rs: only process the latest search job,
  send cancelled results for superseded jobs
- LSF_VERSION bumped 7→8 (FlatEntry struct changed)
- E2e tests rewritten as Rust integration tests that walk the real
  filesystem (27.5M files) and measure search latency

Performance on 27.5M files:
  *.js:                    224ms (4.7M results)
  path:repos:              1.15s  (13.7M results)
  main.js path:Ayla path:repos: 1.33s (1004 results)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@dkattan
style: fix clippy warnings in path filter
6e8e11c 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@dkattan
perf(search): stop maintaining flat index on FS events, simplify status
38428cf 
The flat index's sorted Vec required O(N) shifts on every insert/remove,
making each FS event take 10-60s with 26M entries. Stop maintaining the
flat index on FS events — it's built once during the initial walk and
stays static. The path: filter falls back to node_path() for nodes added
after the initial walk, which is fine since FS events are rare.

Also simplify the status bar:
- Replace cycling Walking/Indexing/Scanning messages with steady Indexing…
- Don't send file count during walk (removes shifting number display)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@dkattan

dkattan commented Jun 25, 2026

Copy link
Copy Markdown
Owner Author

Superseded by upstream PR: cardisoft#219

@dkattan dkattan closed this Jun 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@dkattan