Conversation
Implement LIKE/ILIKE pattern matching, BETWEEN operator, IN (list), CASE WHEN expressions, CAST type conversion, and COALESCE/NULLIF functions. All features compile to bytecode and execute in the VM. New opcodes: Like, Glob, Cast, IsNull New test modules: like, between, in_list, case, cast, coalesce Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Implement UNION/UNION ALL, INTERSECT, and EXCEPT set operations. UNION ALL combines all rows, UNION deduplicates using Distinct opcode. INTERSECT keeps only common rows, EXCEPT keeps rows in left but not right. New opcodes: Distinct, Intersect, Except New test modules: union, intersect, except (17 new tests) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Implement all outer join types using nested loop join pattern: - LEFT OUTER JOIN: outputs all left rows with NULL for unmatched right - RIGHT OUTER JOIN: swaps outer/inner roles, NULL for unmatched left - FULL OUTER JOIN: two-pass algorithm for complete outer join semantics New opcodes: NullRow, RewindInner, MarkMatch Tests: 16 new tests across left_join, right_join, full_join modules Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Phase 4A: Add index-based result types to table.rs - RowRef, RowSet, JoinedRowSet for tracking row indices - IndexedResult enum for unified result handling - LazyRow for deferred materialization - IndexedResultBuilder for constructing results Phase 4B: Modify Cursor to borrow &Table instead of cloning - Significant memory savings for large table scans - Add current_row_idx() method for index tracking Phase 4C: Add parallel result_indices tracking - Track row indices alongside materialized results - Track cursor source order for multi-table queries Phase 4D: Add debug validation for index tracking - validate_index_tracking() verifies indices match results - Foundation for future full switch to index-only mode All 159 tests pass. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Phase 4E: JOIN optimization with JoinedRowSet - Enhanced validation for JOIN index tracking - Added materialize_join_row_from_indices() for lazy materialization - Added cursor_count() and is_join_query() helper methods Phase 4F: Subquery support foundation - Added scalar subquery compilation (compile_scalar_subquery) - Added detection for IN (SELECT...) and EXISTS (SELECT...) - Stubs for IN and EXISTS subqueries (to be implemented) Scalar subqueries work for simple cases like: SELECT * FROM t1 WHERE col = (SELECT col FROM t2 LIMIT 1) All 159 tests pass. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add scalar subqueries with aggregate functions (MAX, MIN, COUNT, SUM, AVG) - Add IN (SELECT ...) and NOT IN (SELECT ...) subqueries - Add EXISTS (SELECT ...) and NOT EXISTS (SELECT ...) subqueries - Add AND/OR logical operator support in WHERE clause compilation - Implement compile-time subquery execution (evaluates subqueries during compilation and emits result literals, avoiding runtime complexity) - Add helper functions for subquery evaluation: - evaluate_scalar_subquery(): computes aggregate values - evaluate_exists_subquery(): checks for row existence - evaluate_in_subquery_values(): collects subquery result values - evaluate_where_at_compile_time(): filters rows during compilation - Create test suites: - tests/subquery/mod.rs: 7 tests for scalar subqueries - tests/in_subquery/mod.rs: 6 tests for IN subqueries - tests/exists/mod.rs: 7 tests for EXISTS subqueries All 179 tests pass, clippy clean. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Move inline test data out of source code into reusable static fixtures: New fixtures: - employees.csv: comprehensive 8-row fixture with salary/department/role - people.csv: simple 3-row fixture for basic tests - strings.csv: string function testing data - set_a.csv, set_b.csv: UNION/EXCEPT/INTERSECT test data Removed unused fixtures: - animals.csv, categories.csv, products_with_prices.csv Refactored test modules to use static fixtures: - aggregate, alias, group_by, string_functions, delimiter, basic, update Updated helpers/mod.rs with path helper functions for all fixtures. Kept create_custom_csv() for edge case tests requiring specific data. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Consolidate duplicate get_*_file() functions into tests/helpers/mod.rs. Updated join, join_on, distinct, limit_offset, and comparison modules to import from helpers instead of defining local duplicates. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Make VM the sole execution engine, removing legacy direct execution path - Remove unused string_functions.rs module entirely - Clean up dead code from table.rs, file_handler.rs, error.rs, bytecode.rs - Fix test assertions to work with VM verbose output - Fix all clippy warnings (collapsible if-let, type complexity, unused vars) - Add MultiTableProjection type alias for complex return type - Consolidate test data file getters across test modules All 179 tests pass with zero clippy warnings. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add runtime execution for correlated subqueries that reference outer query columns. Key changes: - Add correlation detection (detect_outer_references) to identify outer column references in subquery WHERE clauses - Implement correlated EXISTS/NOT EXISTS with nested loop execution - Implement correlated scalar subqueries (COUNT, SUM, AVG, MIN, MAX) using SQLite-aligned approach with Null accumulator initialization - Add current_outer_alias field to SqlCompiler for self-referential query support (same table in outer and inner with alias) - Update AggStep to reset accumulator when register is Null - Update AggFinal to return 0 for COUNT when no rows match Tests: 6 new correlated subquery tests, all 185 tests pass. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Phase 5 found that basic DML operations (INSERT, UPDATE, DELETE) were already fully implemented and working. The main addition was: - Add INSERT...SELECT support: compile SELECT source to read from source table and insert matching rows into target table - Support WHERE clause filtering in INSERT...SELECT - Support SELECT * and explicit column lists Tests: 186 tests pass (1 new INSERT...SELECT test) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…ions and arithmetic Add new VM opcodes and implementations: - MathFunc: ABS, ROUND, CEIL, FLOOR - StringFunc extensions: CONCAT, LEFT, RIGHT - DateFunc: DATE, TIME, NOW, CURRENT_DATE, CURRENT_TIME, CURRENT_TIMESTAMP - Arithmetic: Add, Subtract, Multiply, Divide, Remainder opcodes Includes 17 new integration tests covering all primitives. 203 tests pass, clippy clean. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…aggregates Add comprehensive window function support to the VM execution engine: - ROW_NUMBER() for sequential row numbering within partitions - RANK() and DENSE_RANK() for ranking with/without gaps for ties - LAG() and LEAD() for accessing previous/next row values with custom offsets - SUM/AVG/COUNT/MIN/MAX OVER() for running aggregate calculations All window functions support PARTITION BY for grouping and ORDER BY for sorting within windows. Partition boundaries are properly detected to reset state and return NULL for LAG/LEAD at edges. 219 tests pass, clippy clean. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…UNCATE, CREATE TABLE AS SELECT - Add DROP TABLE with IF EXISTS support - Add ALTER TABLE ADD COLUMN for adding new columns with NULL defaults - Add TRUNCATE TABLE to clear all rows from a table - Add CREATE TABLE AS SELECT with WHERE clause filtering - Add new opcodes: DropTable, AlterTableAdd, Truncate - Add TableModification variants for DDL operations - Add integration tests for all DDL operations Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Split the 12,067-line compiler.rs into 7 focused modules: - compiler.rs (6,461 lines): Core + SELECT + expressions + WHERE - compiler_join.rs (2,582 lines): JOIN compilation - compiler_aggregate.rs (902 lines): Aggregates, GROUP BY, HAVING - compiler_window.rs (697 lines): Window functions - compiler_dml.rs (605 lines): INSERT, UPDATE, DELETE - compiler_ddl.rs (511 lines): CREATE, DROP, ALTER, TRUNCATE - compiler_tests.rs (369 lines): Unit tests This improves maintainability by organizing code by functionality. All 231 tests pass, clippy clean. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Split the 494-line monolithic compile_function() into a thin 35-line dispatcher and 10 focused helper functions: - get_function_name(): Extract function name from AST - compile_func_coalesce(): COALESCE function - compile_func_nullif(): NULLIF function - compile_func_string_simple(): UPPER, LOWER, TRIM, LTRIM, RTRIM, LENGTH - compile_func_substr(): SUBSTR/SUBSTRING - compile_func_replace(): REPLACE - compile_func_concat(): CONCAT - compile_func_left_right(): LEFT, RIGHT - compile_func_math(): ABS, ROUND, CEIL, CEILING, FLOOR - compile_func_datetime_noarg(): NOW, CURRENT_TIMESTAMP, CURRENT_DATE, CURRENT_TIME - compile_func_datetime(): DATE, TIME Each helper is now a focused, self-contained function under 120 lines. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add new helper functions to SqlCompiler: - allocate_registers(count): allocate multiple consecutive registers - patch_jump(addr, target): patch jump instruction addresses - binary_op_to_comparison_opcode(op): convert BinaryOperator to OpCode - sql_type_to_internal(data_type): convert SQL type strings Apply helpers across all compiler modules: - Replace inline ObjectName conversion with get_table_name() - Replace manual register allocation loops with allocate_registers() - Replace repeated BinaryOperator match blocks with helper - Replace inline type mapping with sql_type_to_internal() Improve error handling: - Convert .unwrap() calls to ? operator in compiler_join.rs Net reduction of ~140 lines while improving consistency and maintainability. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Convert 306 verbose add_instruction(Instruction::new(...)) calls to emit() - Add emit() helper method for clean instruction emission - Add emit_column_loads() helper for batch column loading - Unify 6 comparison opcodes in engine.rs into single handler - Remove ~860 lines of redundant code Files converted: - compiler.rs: 205 instances - compiler_aggregate.rs: 42 instances - compiler_join.rs: 34 instances - compiler_window.rs: 25 instances Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add AGG_COUNT, AGG_SUM, AGG_AVG, AGG_MIN, AGG_MAX constants to bytecode.rs - Add agg_func_type() helper to centralize aggregate name-to-type conversion - Update engine.rs AggStep and AggFinal to use named constants - Remove duplicate get_agg_func_type() function - Improves code readability and maintainability Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add emit_and() helper for AND logic - Add emit_or() helper for OR logic - Add emit_comparison() helper for comparison operations - Refactor compile_multi_table_condition to use helpers (-140 lines) - Refactor compile_multi_join_condition to use emit_comparison - Refactor compile_join_condition to use emit_comparison - Refactor compile_having_condition to use emit_comparison Net reduction: 121 lines Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add find_column_in_tables() helper for table.column lookups - Add build_wildcard_schema() helper for SELECT * handling - Refactor resolve_multi_table_projection to use helpers (-56 lines) - Refactor resolve_join_projection to use helpers (-53 lines) - Net reduction: 66 lines of duplicated code Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The VM execution loop previously cloned every Instruction on each iteration due to Rust's borrow checker constraints. This change makes clone O(1) instead of O(n) by using Rc<str> for the string fields. Changes: - bytecode.rs: Change p4 and comment from Option<String> to Option<Rc<str>> - engine.rs: Update 12 p4 usages to use as_deref() instead of clone() - compiler_tests.rs: Update 3 p4 comparisons to use as_deref() Performance impact: - Clone cost: O(n) string copy -> O(1) refcount increment - Memory per p4: 24 bytes (String) -> 16 bytes (Rc<str>) - Hot execution loop: No more heap allocations per instruction Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Replace verbose tutorial-style documentation with concise reference format. Covers all VM compiler features: SELECT, JOINs, set operations, DML, DDL, functions, and operators. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add tsq binary for generating test SQL queries and data - Add table alias support in implicit joins (FROM a, b AS x) - Add LIMIT/OFFSET support for 3+ table implicit joins - Add aggregate functions in multi-table JOIN projection - Add GROUP BY support for multi-table queries Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add reusable row buffer to avoid per-row Vec allocations - Replace add_row_recovery with add_row_from_slice for buffer reuse - Add profiling build profile to Cargo.toml - Add analyze_profile.py tool for samply profile analysis Profiling shows Vec::from_iter total time reduced from 39.7% to 28.1% (29% improvement) when loading large CSV files. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Implement memory-mapped file storage for zero-copy CSV access: - Add src/storage/ module with MmapStorage backend - Use madvise MADV_SEQUENTIAL + MADV_WILLNEED for kernel read-ahead - Zero-copy string references directly into mmap'd region - Add libc dependency for madvise syscall Performance: 2.6x speedup over baseline on 14.5M row dataset. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Reduce allocation overhead with named capacity constants: - Add src/capacity.rs with hints for Vec, HashMap, HashSet - Use with_capacity() in VM engine, compiler, bytecode, mmap storage - Add estimate_row_count() for file-size-based row pre-allocation - Use .clamp() for bounds checking per clippy ~7% improvement on top of mmap changes (2.8x total vs baseline). Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- README: Condense to essential examples, update features list - database.md: Rewrite join section to reflect full JOIN support, remove outdated limitation - user_guide.md: Add --tabledef docs, consolidate examples section, fix numbering Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add tests for CREATE TABLE with LOCATION clause (tests/ddl/mod.rs) - Add tests for --tabledef CLI option (tests/tabledef/mod.rs) - Add tests for headerless file auto-detection (tests/headerless/mod.rs) - Fix --tabledef to preserve predefined columns during file loading - Fix mmap storage to support predefined columns via open_with_columns - Fix REPL to properly buffer multiline SQL statements until semicolon - Consolidate headerless detection utilities in src/capacity.rs - Add comment line filtering to mmap storage Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Implement atomic write pattern (temp file + fsync + rename) for file writeback to preserve original data on write failures - Fix output delimiter bug: stdout now uses source table's delimiter instead of hardcoded comma (principle of least surprise) - Add set_delimiter() method to Table for result table configuration - Add 10 new integration tests for stdout and file output behavior - Update delimiter tests to expect correct output format - Document I/O patterns in doc/database.md Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Simplify version specs to major-only for stable crates (1.x+) - Replace deprecated Command::cargo_bin() with cargo_bin_cmd!() macro in all test files (compatible with custom cargo build directories) - Remove unused assert_cmd::Command imports after migration Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Replace thiserror derive macro with manual impl Display, Error, and From traits for SqawkError - Replace rand crate with libc srand/rand for random number generation in tsq.rs, using time^pid as the default seed - Remove both dependencies from Cargo.toml Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Update version to 0.8.0 - Update categories: replace "database" with "parser-implementations" - Update rust-version from 1.65.0 to 1.70.0 - Add exclude list for smaller package (tests/, doc/, .github/, CLAUDE.md) - Remove documentation field (will add docs.rs URL post-publish) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
jgarzik
added
bug
There was a problem hiding this comment.
Pull request overview
This PR implements a major infrastructure refactoring to transition the codebase to a VM-based execution model. The changes add comprehensive test coverage for advanced SQL features including window functions, subqueries, set operations, outer joins, and DDL operations. The test infrastructure has been modernized to use cargo_bin_cmd! macro and static fixture files.
Changes:
- Added 20+ new test modules covering advanced SQL features (window functions, subqueries, joins, set operations, DDL)
- Refactored test infrastructure to use static CSV fixtures and helper functions
- Updated VM execution to track table modifications and return structured results
- Added new compiler test module and memory storage backend
- Modernized all tests to use
cargo_bin_cmd!macro for better compatibility with custom build directories
Reviewed changes
Copilot reviewed 84 out of 89 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| tests/mod.rs | Added 20+ new test module declarations for VM features |
| tests/helpers/mod.rs | Refactored helpers to use static fixtures, added path helper functions, removed Command import |
| tests/basic/mod.rs | Updated Command usage, added INSERT...SELECT test, fixed helper function calls |
| tests/window/mod.rs | New comprehensive window function tests (ROW_NUMBER, RANK, LAG, LEAD, aggregates) |
| tests/union/mod.rs | New UNION/UNION ALL set operation tests |
| tests/subquery/mod.rs | New scalar subquery tests with aggregates |
| tests/tabledef/mod.rs | New tests for --tabledef CLI option |
| tests/output/mod.rs | New tests for stdout and file output with atomic writes |
| tests/{left,right,full}_join/mod.rs | New outer join tests |
| tests/{exists,in_subquery}/mod.rs | New subquery tests |
| tests/{between,case,cast,coalesce}/mod.rs | New expression tests |
| tests/{arithmetic,math_functions,date_functions}/mod.rs | New function tests |
| tests/headerless/mod.rs | New auto-detection tests |
| tests/ddl/mod.rs | New DDL operation tests |
| tests/correlated/mod.rs | New correlated subquery tests |
| src/vm/mod.rs | Added VmExecutionResult, modification tracking, and verbose output |
| src/vm/compiler_tests.rs | New compiler test suite |
| src/vm/tests.rs | Formatting fixes and Value::String conversion |
| src/storage/memory.rs | New in-memory storage backend |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.