Implement SQL-based document search matching MATLAB's database behavior

[stevevanhooser]

Summary

This PR implements comprehensive SQL-based document search functionality for SQLite, matching MATLAB's did.implementations.database.m behavior. It includes proper document flattening via doc2sql, field indexing, and SQL query generation for various search operations.

Key Changes

SQLiteDB (src/did/implementations/sqlitedb.py)

• Added _sqlite_regexp() function to support regex operations in SQLite
• Added _sql_escape() helper for safe SQL string literal escaping
• Implemented _get_field_idx() to manage field indexing with caching, converting triple-underscores to dots per MATLAB convention
• Implemented _populate_doc_data() to flatten documents via doc2sql and insert into fields and doc_data tables
• Updated _do_add_doc() to properly populate field data instead of placeholder logic
• Implemented search() method as the main entry point for SQL-based searches
• Implemented _search_doc_ids() for recursive search structure traversal:
  • Handles struct arrays (AND operations)
  • Handles OR operations with union semantics
  • Delegates leaf queries to SQL generation
  • Supports negation via ~ prefix
• Implemented _query_struct_to_sql_str() to convert query structures to SQL WHERE clauses, supporting:
  • String operations: exact_string, exact_string_anycase, contains_string, regexp
  • Numeric operations: exact_number, lessthan, lessthaneq, greaterthan, greaterthaneq
  • Field operations: hasfield, isa, depends_on
  • Falls back to None for unsupported operations
• Implemented _brute_force_search() fallback using field_search for unsupported SQL operations

doc2sql (src/did/implementations/doc2sql.py)

• Added _get_class_name() to extract class names from both DID-python and NDI/MATLAB document formats
• Added _get_superclass_str() to extract and format superclass strings matching MATLAB's comma-space separated, sorted format
• Added _serialize_depends_on() to serialize dependency information in MATLAB's format (name,value;name,value;)
• Added _flatten_dict() to flatten nested dictionaries using triple-underscore separators
• Added _get_meta_table_from() to create meta-tables from field groups
• Enhanced doc_to_sql() to properly build meta-tables with:
  • Standard meta table with doc_id, class, superclass, datestamp, creation, deletion, depends_on
  • Per-group tables for all top-level dict fields (excluding skip list)
  • Proper handling of both DID-python and MATLAB document schemas
• Improved get_field() to handle empty strings and IndexError exceptions

Query (src/did/query.py)

• Implemented to_search_structure() to resolve high-level operations
• Added _resolve_search_structure() for recursive resolution of search structures
• Added _resolve_single() to handle operation-specific resolution:
  • isa: kept unresolved to work with both field_search and SQL paths
  • depends_on: converted to hasanysubfield_exact_string with proper parameter handling
  • or: recursively resolves sub-structures
  • Supports negation prefix (~) throughout

Notable Implementation Details

• Field names in the database use {group}.{field} format matching MATLAB convention, with triple-underscores in column names converted to dots
• Search supports both AND (struct arrays) and OR operations with proper set semantics
• Negation is handled at each level of the search tree
• SQL generation includes fallback to brute-force search for unsupported operations
• Document flattening properly handles both DID-python and MATLAB/NDI document formats
• Superclass and dependency information is serialized in MATLAB-compatible formats

https://claude.ai/code/session_01DvD2oeqUFXWGPUfoenh2aj