added coconut implementation (static & streaming)

CMakeLists.txt

@@ -40,6 +40,7 @@ option(BUILD_ODYSSEY "Build Odyssey components with MPI support" OFF)
 option(ODYSSEY_DEBUG_RESULTS "Print Odyssey search results (query_result) for debug before copy to I,D" OFF)
 option(BUILD_SING "Enable CUDA support" OFF)
 option(BUILD_SOFA "Build SOFA with FFTW3 support" OFF)
+option(BUILD_COCONUT "Build the COCONUT sortable-SAX algorithm (static + streaming)" ON)
 
 # Backward compatibility: ENABLE_MPI and ODYSSEY_MPI map to BUILD_ODYSSEY
 if(DEFINED ENABLE_MPI)
@@ -74,6 +75,7 @@ message(STATUS "  BUILD_TESTS: ${BUILD_TESTS}")
 message(STATUS "  BUILD_ODYSSEY: ${BUILD_ODYSSEY}")
 message(STATUS "  ODYSSEY_DEBUG_RESULTS: ${ODYSSEY_DEBUG_RESULTS}")
 message(STATUS "  BUILD_SING: ${BUILD_SING}")
+message(STATUS "  BUILD_COCONUT: ${BUILD_COCONUT}")
 
 # Define PROJECT_ROOT_DIR for all targets (paramSetup.cpp is compiled by commons_lib and by benchmark executables)
 get_filename_component(PROJECT_ROOT_DIR "${CMAKE_SOURCE_DIR}" ABSOLUTE)

README.md

@@ -48,6 +48,7 @@ The following table summarizes the key features of each algorithm:
 | **[Hercules](https://helios2.mi.parisdescartes.fr/~themisp/publications/pvldb22-hercules.pdf)** | In-memory hierarchical similarity search using EAPCA and SAX-based pruning |
 | **[DumpyOS](https://helios2.mi.parisdescartes.fr/~themisp/publications/vldbj24-dumpyos.pdf)** | In-memory scalable data series similarity search using an adaptive multi-ary iSAX index |
 | **[FreSH](http://publications.ics.forth.gr/tech-reports/2023/2023.TR489_FreSh_A_LockFree_Data_Series_Index.pdf)** | In-memory lock-free parallel similarity search using an iSAX index (SRDS 2023) |
+| **[COCONUT](http://www.vldb.org/pvldb/vol11/p677-kondylakis.pdf)** | Sortable-SAX index built bottom-up; supports both static datasets and **streaming** (incremental) inserts (PVLDB 2018) |
 
 
 
@@ -83,6 +84,7 @@ Based on the available hardware, you can specify the below arguments to enable/d
 | `BUILD_DEMO` | Build demonstration applications | `ON` | Core library |
 | `BUILD_ODYSSEY` | Enable MPI for distributed computing | `OFF` | OpenMPI/MPICH |
 | `BUILD_SING` | Enable CUDA for GPU acceleration | `OFF` | CUDA Toolkit |
+| `BUILD_COCONUT` | Enable the COCONUT sortable-SAX algorithm | `ON` | Core library |
 | `DEBUG_MSG` | Enable debug output | `OFF` | None |
 | `BUILD_SOFA` | Enable SOFA with SFA-Based indexing | `OFF` | FFTW3 | 
 
@@ -197,4 +199,3 @@ The logo of DaiSy was designed by [Eva Chamilaki](https://evachamilaki.github.io
 
 
 
-

benchmark/CMakeLists.txt

@@ -140,6 +140,30 @@ if(DEBUG_MSG)
     message(STATUS "Include directories added for bm_LbBruteforce_L2Square.")
 endif()
 
+# ////// COCONUT //////
+if(BUILD_COCONUT)
+    add_executable(bm_Coconut_L2Square
+        bm_Coconut_L2Square.cpp
+        bm_utils.cpp
+        ../commons/paramSetup.cpp
+        ../commons/test_bm_utils.cpp
+        ../commons/dataloaders.cpp
+    )
+    target_link_libraries(bm_Coconut_L2Square
+        PRIVATE
+        benchmark::benchmark
+        benchmark::benchmark_main
+        dino_lib
+    )
+    target_include_directories(bm_Coconut_L2Square
+        PRIVATE
+        ${CMAKE_CURRENT_SOURCE_DIR}/../lib
+        ${CMAKE_CURRENT_SOURCE_DIR}/../commons
+    )
+elseif(DEBUG_MSG)
+    message(STATUS "BUILD_COCONUT is OFF. Skipping Coconut benchmarks.")
+endif()
+
 # ////// MESSI //////
 if(DEBUG_MSG)
     message(STATUS "---")
@@ -590,4 +614,4 @@ endif()
 
 if(DEBUG_MSG)
     message(STATUS "--- Benchmark Configuration Complete ---")
-endif()
+endif()

benchmark/bm_Coconut_L2Square.cpp

@@ -0,0 +1,18 @@
+#include <benchmark/benchmark.h>
+#include "bm_utils.hpp"
+#include "../lib/algos/Coconut.hpp"
+
+static void BM_Coconut(benchmark::State& state) {
+    int config_idx = static_cast<int>(state.range(0));
+    const SSTestConfig& config = test_configs[config_idx];
+
+    daisy::Coconut search(daisy::DistanceType::L2_SQUARED);
+
+    for (auto _ : state) {
+        runSSTBenchmark(&search, config.dataset_path, config.query_path, config.thread_count, config.k_value);
+    }
+}
+
+BENCHMARK(BM_Coconut)->Arg(0)->MinTime(2.0)->Unit(benchmark::kMillisecond);
+
+BENCHMARK_MAIN();

demos/CMakeLists.txt

@@ -37,6 +37,27 @@ if(BUILD_DEMO)
         message(STATUS "Include directories added for demo_bruteforce_L2Square.")
     endif()
 
+    # ////// COCONUT (static + streaming) //////
+    if(BUILD_COCONUT)
+        if(DEBUG_MSG)
+            message(STATUS "## Demo: Coconut")
+        endif()
+        add_executable(demo_Coconut_L2Square demo_Coconut_L2Square.cpp)
+        target_link_libraries(demo_Coconut_L2Square PRIVATE dino_lib commons_lib)
+        target_include_directories(demo_Coconut_L2Square PRIVATE
+            ${CMAKE_CURRENT_SOURCE_DIR}/../lib
+            ${CMAKE_CURRENT_SOURCE_DIR}/../commons
+        )
+        add_executable(demo_Coconut_Streaming demo_Coconut_Streaming.cpp)
+        target_link_libraries(demo_Coconut_Streaming PRIVATE dino_lib commons_lib)
+        target_include_directories(demo_Coconut_Streaming PRIVATE
+            ${CMAKE_CURRENT_SOURCE_DIR}/../lib
+            ${CMAKE_CURRENT_SOURCE_DIR}/../commons
+        )
+    elseif(DEBUG_MSG)
+        message(STATUS "BUILD_COCONUT is OFF. Skipping Coconut demos.")
+    endif()
+
     # ////// BRUTEFORCE DTW //////
     if(DEBUG_MSG)
         message(STATUS "---")
@@ -562,4 +583,3 @@ else()
         message(STATUS "BUILD_DEMO is FALSE. Demo executables will NOT be built.")
     endif()
 endif()
-

demos/demo_Coconut_L2Square.cpp

@@ -0,0 +1,43 @@
+// COCONUT — static (batch) usage: build the sortable-SAX index bottom-up from a whole
+// dataset, then run exact kNN with L2 (squared) distance.
+
+#include "../commons/dataloaders.hpp"
+#include "../lib/daisy.hpp"
+
+int main()
+{
+    daisy::idx_t n_database = 200000;
+    unsigned long long dim = 96;
+    unsigned long long n_query = 10;
+    daisy::idx_t k = 5;
+
+    float *database = loadRandomData(n_database, dim, 100, true);
+    float *query = loadRandomData(n_query, dim, 50, true);
+
+    printf("Loaded %llu database points and %llu query points with dimension %llu\n",
+           n_database, n_query, dim);
+
+    daisy::Coconut coconut(daisy::DistanceType::L2_SQUARED);
+
+    // Static bottom-up build over the full dataset.
+    coconut.buildIndex(database, n_database, dim);
+
+    daisy::idx_t *I = new daisy::idx_t[n_query * k];
+    float *D = new float[n_query * k];
+    coconut.searchIndex(query, n_query, k, I, D);
+
+    for (daisy::idx_t i = 0; i < n_query; i++)
+    {
+        printf("Query %llu: ", i);
+        for (daisy::idx_t j = 0; j < k; j++)
+            printf("%llu ", I[i * k + j]);
+        printf("\n");
+    }
+
+    delete[] database;
+    delete[] query;
+    delete[] I;
+    delete[] D;
+
+    return 0;
+}

demos/demo_Coconut_L2Square.py

@@ -0,0 +1,34 @@
+import sys
+import os
+import numpy as np
+
+sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
+
+from daisy import DistanceType, Coconut
+
+def main():
+    n_database = 200000
+    dim = 96
+    n_query = 10
+    k = 5
+
+    np.random.seed(100)
+    db = np.random.randn(n_database, dim).astype(np.float32)
+
+    np.random.seed(50)
+    query = np.random.randn(n_query, dim).astype(np.float32)
+
+    index = Coconut(DistanceType.L2_SQUARED)
+    index.setNumThreads(1)
+    index.buildIndex(db)   # static bottom-up build
+
+    I, D = index.searchIndex(query, k)
+
+    for query_num in range(n_query):
+        print(f"Query {query_num}:")
+        print("Distances:", D[query_num])
+        print("Indices:", I[query_num])
+        print()
+
+if __name__ == "__main__":
+    main()

demos/demo_Coconut_Streaming.cpp

@@ -0,0 +1,56 @@
+// COCONUT — streaming: build on an initial batch, then insert new series into the live
+// index as they arrive and query at any point (no rebuild). COCONUT's distinguishing feature.
+
+#include "../commons/dataloaders.hpp"
+#include "../lib/daisy.hpp"
+
+int main()
+{
+    unsigned long long dim = 96;
+    daisy::idx_t initial = 50000;   // series available at build time
+    daisy::idx_t batch = 25000;     // series that arrive later, per streaming step
+    int steps = 3;
+    unsigned long long n_query = 5;
+    daisy::idx_t k = 5;
+
+    // Whole stream in one buffer; we feed it in pieces to simulate arrival over time.
+    daisy::idx_t total = initial + (daisy::idx_t)steps * batch;
+    float *stream = loadRandomData(total, dim, 100, true);
+    float *query = loadRandomData(n_query, dim, 50, true);
+
+    daisy::Coconut coconut(daisy::DistanceType::L2_SQUARED);
+
+    // 1. Build on the initial batch.
+    coconut.buildIndex(stream, initial, dim);
+    printf("Built on %llu series.\n", initial);
+
+    daisy::idx_t *I = new daisy::idx_t[n_query * k];
+    float *D = new float[n_query * k];
+
+    auto run_query = [&](const char *when)
+    {
+        coconut.searchIndex(query, n_query, k, I, D);
+        printf("[%s] query 0 kNN: ", when);
+        for (daisy::idx_t j = 0; j < k; j++)
+            printf("%llu(%.1f) ", I[j], D[j]);
+        printf("\n");
+    };
+    run_query("after build");
+
+    // 2. Stream the remaining series in batches, querying after each.
+    daisy::idx_t seen = initial;
+    for (int s = 0; s < steps; s++)
+    {
+        coconut.insertBatch(stream + (size_t)seen * dim, batch);
+        seen += batch;
+        printf("Streamed a batch; index now holds %llu series.\n", seen);
+        run_query("after insert");
+    }
+
+    delete[] stream;
+    delete[] query;
+    delete[] I;
+    delete[] D;
+
+    return 0;
+}

demos/demo_Coconut_Streaming.py

@@ -0,0 +1,43 @@
+import sys
+import os
+import numpy as np
+
+sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
+
+from daisy import DistanceType, Coconut
+
+# COCONUT streaming: build on an initial batch, then insert new series into the live index
+# as they arrive and query after each step (no rebuild).
+def main():
+    dim = 96
+    initial = 50000
+    batch = 25000
+    steps = 3
+    n_query = 5
+    k = 5
+
+    total = initial + steps * batch
+    np.random.seed(100)
+    stream = np.random.randn(total, dim).astype(np.float32)
+    np.random.seed(50)
+    query = np.random.randn(n_query, dim).astype(np.float32)
+
+    index = Coconut(DistanceType.L2_SQUARED)
+    index.setNumThreads(1)
+    index.buildIndex(stream[:initial])         # build on the initial batch
+
+    def show(when):
+        I, D = index.searchIndex(query, k)
+        print(f"[{when}] query 0 -> indices {I[0]}, distances {D[0]}")
+
+    show("after build")
+
+    seen = initial
+    for s in range(steps):
+        index.insertBatch(stream[seen:seen + batch])   # stream the next batch
+        seen += batch
+        print(f"streamed a batch; index now holds {seen} series")
+        show("after insert")
+
+if __name__ == "__main__":
+    main()

docs/demos-guide.md

@@ -3,6 +3,11 @@
 The demos module provides practical examples of how to use the DaiSy library's algorithms for data series similarity search. 
 Each demo illustrates a specific algorithm or specific distance metric. This module includes both C++ and Python implementations for various algorithms and use cases.
 
+Most demos follow the same batch pattern: `buildIndex(...)` once, then `searchIndex(...)`.
+The **Coconut** algorithm additionally supports **streaming**: `demo_Coconut_L2Square` shows
+the static (batch) build, while `demo_Coconut_Streaming` builds on an initial batch and then
+`insert`/`insertBatch`es new series into the live index, querying after each step.
+
 ## Demo Program Structure
 
 All demos are located in the [`demos/`](../demos/) directory.

docs/how-to-contribute.md

@@ -11,7 +11,8 @@ You may reach us by contacting the main authors of the project, [Francesca Del G
 Here is a (non-exhaustive) mockup of our future and ongoing goals:
 
 - Extension of DaiSy for subsequence similarity search
-- Extension of DaiSy for more algorithms (e.g., SFA, Hercules, Coconut, Dumpy, etc.)
+- Extension of DaiSy for more algorithms (e.g., SFA, Hercules, Dumpy, etc.)
+- Streaming / updatable indexing for more algorithms (currently supported by Coconut)
 - Implementation of a DaiSy autotuner to automatically optimize indexing and search parameters
 - Extension of DaiSy to support learned optimization approaches, e.g., LeaFi and ProS
 

lib/algos/CMakeLists.txt

@@ -27,6 +27,22 @@ if(DEBUG_MSG)
     message(STATUS "Bruteforce.cpp added.")
 endif()
 
+# ////// COCONUT //////
+if(BUILD_COCONUT)
+    if(DEBUG_MSG)
+        message(STATUS "Adding Coconut.cpp to dino_lib sources.")
+    endif()
+    target_sources(dino_lib
+        PRIVATE
+        ${CMAKE_CURRENT_SOURCE_DIR}/Coconut.cpp
+    )
+    target_compile_definitions(dino_lib PUBLIC COCONUT_ENABLED)
+else()
+    if(DEBUG_MSG)
+        message(STATUS "BUILD_COCONUT is OFF. Skipping Coconut sources.")
+    endif()
+endif()
+
 # ////// LBBRUTEFORCE //////
 if(DEBUG_MSG)
     message(STATUS "Adding LbBruteforce.cpp to dino_lib sources.")