From 7e530ee0c85f33766b32e600e87bc591c0a1a0a4 Mon Sep 17 00:00:00 2001 From: sophisid Date: Mon, 27 Jul 2026 16:37:48 +0200 Subject: [PATCH] md files for LLMs --- AGENTS.md | 180 ++++++++++++++++++++++++++++++++++++++++++++++++++ CONTINUITY.md | 178 +++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 358 insertions(+) create mode 100644 AGENTS.md create mode 100644 CONTINUITY.md diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..5f3726a --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,180 @@ +# AGENTS.md + +This file is for future contributors and LLM agents working inside the DaiSy repository. +Read this before making changes, especially when adding a new algorithm, binding, benchmark, or test. + +## Project Intent + +DaiSy is a C++ similarity-search library for data series and vectors, with optional Python bindings. +The repository mixes: + +- Core algorithm implementations in `lib/algos/` +- Shared utilities and loaders in `commons/` +- Python bindings in `pybinds/setup.cpp` +- C++ tests in `tests/` +- Benchmarks in `benchmark/` +- Demos in `demos/` +- Contributor docs in `docs/` + +The main integration goal is consistency across C++, Python, tests, demos, and build flags. +Do not treat a new algorithm as complete if it only compiles in one layer. + +## Ground Rules + +- Preserve the existing public naming style: algorithm names are user-facing and appear across C++, Python, demos, tests, and docs. +- Prefer matching an existing algorithm pattern instead of inventing a new structure. +- Keep optional or hardware-specific algorithms behind CMake flags. +- Do not assume every algorithm supports every feature: + - Range search is optional and depends on `SearchConfig` + - Streaming insert support is currently special-case behavior + - MPI, CUDA, and FFTW-backed code must remain optional +- Build failures caused by missing optional dependencies are regressions. +- If a change affects algorithm availability, update docs and CMake in the same patch. + +## Files To Read First + +Before changing algorithm integration, inspect these files: + +- `README.md` +- `docs/how-to-contribute.md` +- `lib/algos/SimilaritySearchAlgorithm.hpp` +- `lib/algos/CMakeLists.txt` +- `tests/CMakeLists.txt` +- `benchmark/CMakeLists.txt` +- `demos/CMakeLists.txt` +- `pybinds/setup.cpp` + +If the work changes architecture or responsibilities, also inspect: + +- `docs/daisy-architecture.puml` +- `docs/daisy-components.puml` + +## Adding A New Algorithm + +Minimum expected work: + +1. Add the implementation in `lib/algos/`. +2. Make it inherit from `SimilaritySearchAlgorithm`. +3. Add it to `lib/algos/CMakeLists.txt`. +4. Expose it through the public surface if appropriate: + - C++ umbrella header: `lib/daisy.hpp` + - Python bindings: `pybinds/setup.cpp` +5. Add tests in `tests/`. +6. Add benchmarks in `benchmark/` if the algorithm is performance-relevant. +7. Add demos in `demos/` for the supported modes. +8. Update `README.md` and any relevant docs. + +Use an existing algorithm as the nearest template: + +- Simple in-memory baseline: `Bruteforce`, `LbBruteforce` +- iSAX-oriented patterns: `Messi`, `Fresh`, `DumpyOS` +- Optional dependency patterns: `Sing`, `Odyssey`, `Sofa` +- Streaming pattern: `Coconut` + +## Interface Expectations + +The base interface lives in `lib/algos/SimilaritySearchAlgorithm.hpp`. +Important points: + +- `buildIndex(DataSource *data_source)` is the core abstract entry point. +- Convenience overloads already exist for in-memory arrays and filenames. +- `searchIndex(query, n_query, k, I, D)` is the required top-k API. +- `searchIndex(..., SearchConfig, vector>&, vector>& )` has a default top-k implementation and throws by default for unsupported range search. +- `insert()` and `insertBatch()` throw by default. Only override if the algorithm truly supports streaming updates. +- `setNormalized()` controls breakpoint mode assumptions for algorithms that use SAX/iSAX behavior. Do not silently normalize user data. + +If your algorithm only supports a subset of features, fail explicitly and clearly. + +## Build And Dependency Rules + +Current important CMake options at the repo root: + +- `BUILD_PYTHON` +- `BUILD_BENCHMARK` +- `BUILD_TESTS` +- `BUILD_DEMO` +- `BUILD_ODYSSEY` +- `BUILD_SING` +- `BUILD_SOFA` +- `BUILD_COCONUT` +- `DEBUG_MSG` + +Dependency-sensitive features: + +- `Odyssey` depends on MPI +- `Sing` depends on CUDA +- `Sofa` depends on FFTW3 +- `Coconut` is toggleable via `BUILD_COCONUT` + +When adding a dependency-heavy algorithm: + +- Introduce a dedicated CMake option if it can be absent on developer machines or CI. +- Keep the default build usable without that dependency unless maintainers explicitly want otherwise. +- Ensure the library still configures when the dependency is missing and the feature is disabled. + +## Testing Expectations + +At minimum, new algorithms should be validated against known-correct behavior. +Prefer using existing test patterns and ground-truth assets where possible. + +Check: + +- Correctness versus brute-force or stored ground truth +- L2 squared and DTW when both are supported +- Edge conditions like small datasets, `k == 1`, `k > 1`, and invalid inputs +- Range search behavior if implemented +- Streaming insert behavior if implemented + +Do not add an algorithm without at least one executable test path in `tests/`. + +## Python Binding Expectations + +Python bindings are currently centralized in `pybinds/setup.cpp`. +That file is large and repetitive by design; follow the existing style unless you are intentionally refactoring it. + +When exposing a new algorithm: + +- Keep constructor and method naming aligned with the C++ class +- Validate NumPy array rank and shape +- Return `(indices, distances)` consistently +- Expose configuration setters/getters only if they are meaningful and stable +- Add docstrings for non-obvious parameters and behavior + +If a feature is optional at build time, guard the binding accordingly. + +## Demos And Benchmarks + +Treat demos as executable usage documentation. +If users would not know how to instantiate or query the algorithm from the code alone, the integration is incomplete. + +Benchmarks should: + +- Reuse existing benchmark helpers where possible +- Avoid inventing one-off data loading patterns +- Be added only when the feature is actually buildable under current flags + +## Documentation Policy + +Whenever you add or materially change an algorithm, update: + +- `README.md` +- `docs/how-to-contribute.md` if the contributor workflow changed +- Architecture docs if the new code introduces a new subsystem or dependency pattern + +If behavior is temporary, experimental, or known to be incomplete, write that down instead of implying full support. + +## Safety Checklist Before Finishing + +Before closing work, verify: + +- CMake knows about the new files +- The algorithm is not orphaned from tests, demos, or bindings +- Optional dependencies are properly guarded +- User-facing names are consistent across C++, Python, docs, and executable targets +- Unsupported features fail explicitly rather than silently misbehaving +- Docs mention major limitations + +## Continuity + +Also read `CONTINUITY.md`. +That file captures current repo state, active conventions, and known rough edges that are easy to miss when editing blindly. diff --git a/CONTINUITY.md b/CONTINUITY.md new file mode 100644 index 0000000..52b3354 --- /dev/null +++ b/CONTINUITY.md @@ -0,0 +1,178 @@ +# CONTINUITY.md + +This file captures the current shape of the DaiSy repository so future contributors and LLM agents can recover context quickly. +It is intentionally operational, not aspirational. + +## Current Repo Snapshot + +DaiSy is an alpha-stage exact similarity-search library with: + +- C++ core implementations +- Optional Python bindings +- Tests, demos, and benchmarks as separate build layers +- Multiple algorithm families spanning in-memory, disk-based, distributed, GPU, and streaming cases + +The repo is broad, but integration is still somewhat manual. Many additions require touching several disconnected files. + +## High-Level Layout + +- `lib/algos/`: algorithm implementations and the main extension point +- `lib/distance_computers/`: distance logic shared by algorithms +- `lib/isax/`: SAX/iSAX support used by several index-based algorithms +- `lib/ds_tree/`: tree structures used by `Hercules` +- `commons/`: shared loaders, params, test and benchmark helpers +- `pybinds/setup.cpp`: Python binding entry point +- `tests/`: C++ tests +- `benchmark/`: benchmark executables +- `demos/`: C++ and Python usage examples +- `docs/`: contributor and architecture docs +- `scripts/groundtruth/`: utilities to generate or validate ground-truth data + +## Public And Internal Integration Points + +When adding or changing an algorithm, the important wiring points are: + +- Base interface: `lib/algos/SimilaritySearchAlgorithm.hpp` +- Source registration: `lib/algos/CMakeLists.txt` +- Public umbrella include: `lib/daisy.hpp` +- Python exposure: `pybinds/setup.cpp` +- Tests: `tests/CMakeLists.txt` plus new `tests/test_*.cpp` +- Benchmarks: `benchmark/CMakeLists.txt` plus new `benchmark/bm_*.cpp` +- Demos: `demos/CMakeLists.txt` plus new `demos/demo_*` + +The project does not currently centralize algorithm metadata in one place. +Assume cross-file manual registration is required. + +## Algorithm Surface Today + +From the current source tree and public includes, the main algorithm set includes: + +- `Bruteforce` +- `LbBruteforce` +- `Coconut` +- `Messi` +- `ParIS` +- `Sing` +- `Odyssey` +- `Sofa` +- `Hercules` +- `DumpyOS` +- `Fresh` + +These algorithms do not all share the same dependency profile or feature set. + +## Important Behavioral Conventions + +- Top-k search is the baseline capability. +- Range search is modeled through `SearchConfig` and is not universally implemented. +- Streaming insert is not a generic capability; default behavior throws. +- `setNormalized(bool)` is a declaration about the input data, not a preprocessing step. +- Data can come from in-memory arrays or file-backed sources via `DataSource`. + +If you extend behavior, keep these semantics stable unless you are intentionally redesigning the interface. + +## Build System Reality + +The root `CMakeLists.txt` drives feature toggles and dependency detection. + +Important options: + +- `BUILD_PYTHON` +- `BUILD_BENCHMARK` +- `BUILD_TESTS` +- `BUILD_DEMO` +- `BUILD_ODYSSEY` +- `BUILD_SING` +- `BUILD_SOFA` +- `BUILD_COCONUT` + +Current optional dependency model: + +- MPI gates `Odyssey` +- CUDA gates `Sing` +- FFTW3 gates `Sofa` +- Coconut can be fully toggled off + +Backward-compatibility option aliases exist for some old names such as MPI and CUDA toggles. + +## Known Rough Edges + +These are worth checking before assuming the tree is fully uniform: + +- `pybinds/setup.cpp` is monolithic and repetitive; new bindings usually require copy-adapt work rather than a clean abstraction. +- Several test entries for DTW are present but commented out in `tests/CMakeLists.txt`. +- Test and benchmark registration is manual and verbose. +- Optional algorithms are not uniformly guarded everywhere at the conceptual level, so verify includes and bindings carefully when adding new conditional code. +- The repo mixes “library architecture” docs with practical contributor docs; not every workflow detail is encoded in one canonical place. + +## What To Check Before Adding A New Algorithm + +Read these first: + +- `docs/how-to-contribute.md` +- `README.md` +- `lib/algos/SimilaritySearchAlgorithm.hpp` +- A similar existing algorithm in `lib/algos/` +- `lib/algos/CMakeLists.txt` +- `pybinds/setup.cpp` +- `tests/CMakeLists.txt` +- `benchmark/CMakeLists.txt` +- `demos/CMakeLists.txt` + +Then decide: + +- Is the algorithm always buildable, or optional? +- Does it support top-k only, or also range? +- Does it support L2 squared, DTW, or both? +- Does it need new loaders, new shared utilities, or only local code? +- Should it be exposed to Python now, or intentionally remain C++-only? + +## Expected Contribution Pattern + +For a normal new algorithm contribution, expect to touch at least: + +- `lib/algos/.hpp` +- `lib/algos/.cpp` +- `lib/algos/CMakeLists.txt` +- `lib/daisy.hpp` if it should be part of the public header surface +- `tests/test__*.cpp` +- `tests/CMakeLists.txt` +- `demos/demo__*.cpp` +- Python demo and binding files if supported from Python +- `README.md` + +Benchmarks are also expected for performance-oriented algorithms. + +## Documentation Expectations + +The current repo already has useful support docs: + +- `docs/how-to-contribute.md` +- `docs/benchmark-framework-guide.md` +- `docs/test-framework-guide.md` +- `docs/pybinds-guide.md` +- `docs/demos-guide.md` + +Use them, but do not assume they fully replace source inspection. +The repository still relies heavily on pattern matching against existing implementations. + +## Practical Advice For LLMs + +- Do not edit only one layer and assume the job is done. +- Mirror existing algorithm naming exactly across files. +- Prefer the nearest existing algorithm as a template. +- Be explicit about unsupported features. +- Keep optional dependencies optional. +- If you add a build flag, document it immediately. +- If you notice a current inconsistency, record it in docs or fix it in the same change if safe. + +## When This File Should Be Updated + +Update `CONTINUITY.md` when: + +- A new algorithm is added or removed +- The expected contributor workflow changes +- A major build flag or optional dependency changes +- Python binding structure changes materially +- Tests, demos, or benchmarks gain a new required pattern +- A major known rough edge is fixed or a new one appears