md files for LLMs

AGENTS.md

@@ -0,0 +1,180 @@
+# AGENTS.md
+
+This file is for future contributors and LLM agents working inside the DaiSy repository.
+Read this before making changes, especially when adding a new algorithm, binding, benchmark, or test.
+
+## Project Intent
+
+DaiSy is a C++ similarity-search library for data series and vectors, with optional Python bindings.
+The repository mixes:
+
+- Core algorithm implementations in `lib/algos/`
+- Shared utilities and loaders in `commons/`
+- Python bindings in `pybinds/setup.cpp`
+- C++ tests in `tests/`
+- Benchmarks in `benchmark/`
+- Demos in `demos/`
+- Contributor docs in `docs/`
+
+The main integration goal is consistency across C++, Python, tests, demos, and build flags.
+Do not treat a new algorithm as complete if it only compiles in one layer.
+
+## Ground Rules
+
+- Preserve the existing public naming style: algorithm names are user-facing and appear across C++, Python, demos, tests, and docs.
+- Prefer matching an existing algorithm pattern instead of inventing a new structure.
+- Keep optional or hardware-specific algorithms behind CMake flags.
+- Do not assume every algorithm supports every feature:
+  - Range search is optional and depends on `SearchConfig`
+  - Streaming insert support is currently special-case behavior
+  - MPI, CUDA, and FFTW-backed code must remain optional
+- Build failures caused by missing optional dependencies are regressions.
+- If a change affects algorithm availability, update docs and CMake in the same patch.
+
+## Files To Read First
+
+Before changing algorithm integration, inspect these files:
+
+- `README.md`
+- `docs/how-to-contribute.md`
+- `lib/algos/SimilaritySearchAlgorithm.hpp`
+- `lib/algos/CMakeLists.txt`
+- `tests/CMakeLists.txt`
+- `benchmark/CMakeLists.txt`
+- `demos/CMakeLists.txt`
+- `pybinds/setup.cpp`
+
+If the work changes architecture or responsibilities, also inspect:
+
+- `docs/daisy-architecture.puml`
+- `docs/daisy-components.puml`
+
+## Adding A New Algorithm
+
+Minimum expected work:
+
+1. Add the implementation in `lib/algos/`.
+2. Make it inherit from `SimilaritySearchAlgorithm`.
+3. Add it to `lib/algos/CMakeLists.txt`.
+4. Expose it through the public surface if appropriate:
+   - C++ umbrella header: `lib/daisy.hpp`
+   - Python bindings: `pybinds/setup.cpp`
+5. Add tests in `tests/`.
+6. Add benchmarks in `benchmark/` if the algorithm is performance-relevant.
+7. Add demos in `demos/` for the supported modes.
+8. Update `README.md` and any relevant docs.
+
+Use an existing algorithm as the nearest template:
+
+- Simple in-memory baseline: `Bruteforce`, `LbBruteforce`
+- iSAX-oriented patterns: `Messi`, `Fresh`, `DumpyOS`
+- Optional dependency patterns: `Sing`, `Odyssey`, `Sofa`
+- Streaming pattern: `Coconut`
+
+## Interface Expectations
+
+The base interface lives in `lib/algos/SimilaritySearchAlgorithm.hpp`.
+Important points:
+
+- `buildIndex(DataSource *data_source)` is the core abstract entry point.
+- Convenience overloads already exist for in-memory arrays and filenames.
+- `searchIndex(query, n_query, k, I, D)` is the required top-k API.
+- `searchIndex(..., SearchConfig, vector<vector<idx_t>>&, vector<vector<float>>& )` has a default top-k implementation and throws by default for unsupported range search.
+- `insert()` and `insertBatch()` throw by default. Only override if the algorithm truly supports streaming updates.
+- `setNormalized()` controls breakpoint mode assumptions for algorithms that use SAX/iSAX behavior. Do not silently normalize user data.
+
+If your algorithm only supports a subset of features, fail explicitly and clearly.
+
+## Build And Dependency Rules
+
+Current important CMake options at the repo root:
+
+- `BUILD_PYTHON`
+- `BUILD_BENCHMARK`
+- `BUILD_TESTS`
+- `BUILD_DEMO`
+- `BUILD_ODYSSEY`
+- `BUILD_SING`
+- `BUILD_SOFA`
+- `BUILD_COCONUT`
+- `DEBUG_MSG`
+
+Dependency-sensitive features:
+
+- `Odyssey` depends on MPI
+- `Sing` depends on CUDA
+- `Sofa` depends on FFTW3
+- `Coconut` is toggleable via `BUILD_COCONUT`
+
+When adding a dependency-heavy algorithm:
+
+- Introduce a dedicated CMake option if it can be absent on developer machines or CI.
+- Keep the default build usable without that dependency unless maintainers explicitly want otherwise.
+- Ensure the library still configures when the dependency is missing and the feature is disabled.
+
+## Testing Expectations
+
+At minimum, new algorithms should be validated against known-correct behavior.
+Prefer using existing test patterns and ground-truth assets where possible.
+
+Check:
+
+- Correctness versus brute-force or stored ground truth
+- L2 squared and DTW when both are supported
+- Edge conditions like small datasets, `k == 1`, `k > 1`, and invalid inputs
+- Range search behavior if implemented
+- Streaming insert behavior if implemented
+
+Do not add an algorithm without at least one executable test path in `tests/`.
+
+## Python Binding Expectations
+
+Python bindings are currently centralized in `pybinds/setup.cpp`.
+That file is large and repetitive by design; follow the existing style unless you are intentionally refactoring it.
+
+When exposing a new algorithm:
+
+- Keep constructor and method naming aligned with the C++ class
+- Validate NumPy array rank and shape
+- Return `(indices, distances)` consistently
+- Expose configuration setters/getters only if they are meaningful and stable
+- Add docstrings for non-obvious parameters and behavior
+
+If a feature is optional at build time, guard the binding accordingly.
+
+## Demos And Benchmarks
+
+Treat demos as executable usage documentation.
+If users would not know how to instantiate or query the algorithm from the code alone, the integration is incomplete.
+
+Benchmarks should:
+
+- Reuse existing benchmark helpers where possible
+- Avoid inventing one-off data loading patterns
+- Be added only when the feature is actually buildable under current flags
+
+## Documentation Policy
+
+Whenever you add or materially change an algorithm, update:
+
+- `README.md`
+- `docs/how-to-contribute.md` if the contributor workflow changed
+- Architecture docs if the new code introduces a new subsystem or dependency pattern
+
+If behavior is temporary, experimental, or known to be incomplete, write that down instead of implying full support.
+
+## Safety Checklist Before Finishing
+
+Before closing work, verify:
+
+- CMake knows about the new files
+- The algorithm is not orphaned from tests, demos, or bindings
+- Optional dependencies are properly guarded
+- User-facing names are consistent across C++, Python, docs, and executable targets
+- Unsupported features fail explicitly rather than silently misbehaving
+- Docs mention major limitations
+
+## Continuity
+
+Also read `CONTINUITY.md`.
+That file captures current repo state, active conventions, and known rough edges that are easy to miss when editing blindly.

CONTINUITY.md

@@ -0,0 +1,178 @@
+# CONTINUITY.md
+
+This file captures the current shape of the DaiSy repository so future contributors and LLM agents can recover context quickly.
+It is intentionally operational, not aspirational.
+
+## Current Repo Snapshot
+
+DaiSy is an alpha-stage exact similarity-search library with:
+
+- C++ core implementations
+- Optional Python bindings
+- Tests, demos, and benchmarks as separate build layers
+- Multiple algorithm families spanning in-memory, disk-based, distributed, GPU, and streaming cases
+
+The repo is broad, but integration is still somewhat manual. Many additions require touching several disconnected files.
+
+## High-Level Layout
+
+- `lib/algos/`: algorithm implementations and the main extension point
+- `lib/distance_computers/`: distance logic shared by algorithms
+- `lib/isax/`: SAX/iSAX support used by several index-based algorithms
+- `lib/ds_tree/`: tree structures used by `Hercules`
+- `commons/`: shared loaders, params, test and benchmark helpers
+- `pybinds/setup.cpp`: Python binding entry point
+- `tests/`: C++ tests
+- `benchmark/`: benchmark executables
+- `demos/`: C++ and Python usage examples
+- `docs/`: contributor and architecture docs
+- `scripts/groundtruth/`: utilities to generate or validate ground-truth data
+
+## Public And Internal Integration Points
+
+When adding or changing an algorithm, the important wiring points are:
+
+- Base interface: `lib/algos/SimilaritySearchAlgorithm.hpp`
+- Source registration: `lib/algos/CMakeLists.txt`
+- Public umbrella include: `lib/daisy.hpp`
+- Python exposure: `pybinds/setup.cpp`
+- Tests: `tests/CMakeLists.txt` plus new `tests/test_*.cpp`
+- Benchmarks: `benchmark/CMakeLists.txt` plus new `benchmark/bm_*.cpp`
+- Demos: `demos/CMakeLists.txt` plus new `demos/demo_*`
+
+The project does not currently centralize algorithm metadata in one place.
+Assume cross-file manual registration is required.
+
+## Algorithm Surface Today
+
+From the current source tree and public includes, the main algorithm set includes:
+
+- `Bruteforce`
+- `LbBruteforce`
+- `Coconut`
+- `Messi`
+- `ParIS`
+- `Sing`
+- `Odyssey`
+- `Sofa`
+- `Hercules`
+- `DumpyOS`
+- `Fresh`
+
+These algorithms do not all share the same dependency profile or feature set.
+
+## Important Behavioral Conventions
+
+- Top-k search is the baseline capability.
+- Range search is modeled through `SearchConfig` and is not universally implemented.
+- Streaming insert is not a generic capability; default behavior throws.
+- `setNormalized(bool)` is a declaration about the input data, not a preprocessing step.
+- Data can come from in-memory arrays or file-backed sources via `DataSource`.
+
+If you extend behavior, keep these semantics stable unless you are intentionally redesigning the interface.
+
+## Build System Reality
+
+The root `CMakeLists.txt` drives feature toggles and dependency detection.
+
+Important options:
+
+- `BUILD_PYTHON`
+- `BUILD_BENCHMARK`
+- `BUILD_TESTS`
+- `BUILD_DEMO`
+- `BUILD_ODYSSEY`
+- `BUILD_SING`
+- `BUILD_SOFA`
+- `BUILD_COCONUT`
+
+Current optional dependency model:
+
+- MPI gates `Odyssey`
+- CUDA gates `Sing`
+- FFTW3 gates `Sofa`
+- Coconut can be fully toggled off
+
+Backward-compatibility option aliases exist for some old names such as MPI and CUDA toggles.
+
+## Known Rough Edges
+
+These are worth checking before assuming the tree is fully uniform:
+
+- `pybinds/setup.cpp` is monolithic and repetitive; new bindings usually require copy-adapt work rather than a clean abstraction.
+- Several test entries for DTW are present but commented out in `tests/CMakeLists.txt`.
+- Test and benchmark registration is manual and verbose.
+- Optional algorithms are not uniformly guarded everywhere at the conceptual level, so verify includes and bindings carefully when adding new conditional code.
+- The repo mixes “library architecture” docs with practical contributor docs; not every workflow detail is encoded in one canonical place.
+
+## What To Check Before Adding A New Algorithm
+
+Read these first:
+
+- `docs/how-to-contribute.md`
+- `README.md`
+- `lib/algos/SimilaritySearchAlgorithm.hpp`
+- A similar existing algorithm in `lib/algos/`
+- `lib/algos/CMakeLists.txt`
+- `pybinds/setup.cpp`
+- `tests/CMakeLists.txt`
+- `benchmark/CMakeLists.txt`
+- `demos/CMakeLists.txt`
+
+Then decide:
+
+- Is the algorithm always buildable, or optional?
+- Does it support top-k only, or also range?
+- Does it support L2 squared, DTW, or both?
+- Does it need new loaders, new shared utilities, or only local code?
+- Should it be exposed to Python now, or intentionally remain C++-only?
+
+## Expected Contribution Pattern
+
+For a normal new algorithm contribution, expect to touch at least:
+
+- `lib/algos/<Algo>.hpp`
+- `lib/algos/<Algo>.cpp`
+- `lib/algos/CMakeLists.txt`
+- `lib/daisy.hpp` if it should be part of the public header surface
+- `tests/test_<Algo>_*.cpp`
+- `tests/CMakeLists.txt`
+- `demos/demo_<Algo>_*.cpp`
+- Python demo and binding files if supported from Python
+- `README.md`
+
+Benchmarks are also expected for performance-oriented algorithms.
+
+## Documentation Expectations
+
+The current repo already has useful support docs:
+
+- `docs/how-to-contribute.md`
+- `docs/benchmark-framework-guide.md`
+- `docs/test-framework-guide.md`
+- `docs/pybinds-guide.md`
+- `docs/demos-guide.md`
+
+Use them, but do not assume they fully replace source inspection.
+The repository still relies heavily on pattern matching against existing implementations.
+
+## Practical Advice For LLMs
+
+- Do not edit only one layer and assume the job is done.
+- Mirror existing algorithm naming exactly across files.
+- Prefer the nearest existing algorithm as a template.
+- Be explicit about unsupported features.
+- Keep optional dependencies optional.
+- If you add a build flag, document it immediately.
+- If you notice a current inconsistency, record it in docs or fix it in the same change if safe.
+
+## When This File Should Be Updated
+
+Update `CONTINUITY.md` when:
+
+- A new algorithm is added or removed
+- The expected contributor workflow changes
+- A major build flag or optional dependency changes
+- Python binding structure changes materially
+- Tests, demos, or benchmarks gain a new required pattern
+- A major known rough edge is fixed or a new one appears