Sph 2026 02 21.03

[sphildreth]

This pull request introduces several improvements to the .NET bindings for DecentDB, focusing on enhanced SQL handling, better Entity Framework Core integration, and improved async support. The most important changes include robust multi-statement SQL execution, fixes for parameter handling, and new EF Core plugin registration. Additionally, benchmark data and documentation have been updated.

SQL Handling and Command Execution

• Added SqlStatementSplitter to support splitting and executing multi-statement SQL in DecentDBCommand.ExecuteNonQuery, ensuring correct row counts and execution for each statement. [1] [2]
• Introduced ClampOffsetParameters and StripUpdateDeleteAlias in SqlParameterRewriter to handle negative OFFSET values and unsupported table aliases, improving compatibility with EF Core and preventing errors. [1] [2]
• Improved parameter parsing in SqlParameterRewriter.AllocateIndex to support digit characters in parameter names.

Async Support

• Implemented async methods (OpenAsync, ExecuteNonQueryAsync, ExecuteScalarAsync, ExecuteDbDataReaderAsync) with proper cancellation token handling in DecentDBConnection and DecentDBCommand, aligning with ADO.NET async standards. [1] [2] [3] [4]

Entity Framework Core Integration

• Registered DecentDBNodaTimeMemberTranslatorPlugin for EF Core in DecentDBNodaTimeOptionsExtension, enabling custom translation of NodaTime members and improving LINQ query compatibility. [1] [2]
• Updated EF Core extension hash code logic to use plugin type hash, improving service provider uniqueness.

Benchmark and Documentation Updates

• Updated benchmark results in bench_summary.json for DecentDB, DuckDB, and SQLite, reflecting new performance metrics and run ID. [1] [2]
• Extended README to mention JSON scalar function support in query features.
• Added commit hygiene guidance in AGENTS.md, emphasizing user review before committing.

[Copilot]

Pull request overview

This PR updates DecentDB’s engine and bindings to improve SQL compatibility (identifier casing, type affinity, JSON functions), EF Core integration (primitive collections, NodaTime translation), async ADO.NET APIs, and embedding safety for the shared library, alongside updated docs/benchmarks and new regression tests.

Changes:

• Enhance SQL engine correctness/compatibility: case-insensitive identifiers, SQLite-like type affinity comparisons, planner/binder fixes (aggregates, composite PKs, UNION subqueries), partial index handling, JSON scalar functions.
• Improve .NET provider: multi-statement execution support, parameter rewriting fixes, async APIs, EF Core primitive collection translation, NodaTime member translation plugin + tick precision changes, design-time tooling hardening.
• Operational/maintenance updates: pager cache eviction on close, shared-library build flags, version bump + changelog/docs/bench updates.

Reviewed changes

Copilot reviewed 49 out of 56 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
tests/test_many_tables_reopen.nim	Adds a manual repro script for reopen/join scenarios (currently not part of test runner).
tests/test_join_reopen.nim	Adds a manual repro script for reopen/join scenarios (currently not part of test runner).
tests/test_capi_reopen.nim	Adds a C-API reopen/join repro script (currently not part of test runner).
tests/nim/test_sql_exec.nim	Adjusts planner test expectations for partial index planning behavior.
tests/nim/test_case_insensitive_identifiers.nim	Adds unit tests for case-insensitive identifier resolution semantics.
src/storage/storage.nim	Makes identifier comparisons case-insensitive in storage/index code; exports partial-index predicate helper; adds reusable BTree pager eviction.
src/sql/sql.nim	Improves canonical SQL generation for subqueries/joins; makes CREATE constraint matching case-insensitive.
src/sql/binder.nim	Normalizes identifier resolution; adds subquery metadata handling for set ops; fixes ORDER BY aggregate-alias binding behavior.
src/planner/planner.nim	Fixes rowid alias detection for composite PKs; rewrites ORDER BY aggregate expressions above aggregates.
src/exec/exec.nim	Adds pager eval-cache eviction; implements JSON scalar functions; extends type-affinity comparisons; improves column resolution and LEFT JOIN null padding behavior.
src/engine.nim	Updates unique/constraint handling for partial indexes and case-insensitive names; evicts pager refs from threadvar caches on close; case-insensitive column lookups in multiple paths.
src/catalog/catalog.nim	Normalizes catalog map keys and index/table/view lookup/drop behavior to case-insensitive semantics; excludes partial indexes from planner selection helpers.
src/c_api.nim	Fixes parameter scanning inside IN-subquery SQL.
src/btree/btree.nim	Adds append-cache eviction by pager to avoid leaked/dangling pager refs across open/close cycles.
docs/user-guide/sql-reference.md	Documents JSON scalar functions and adds examples; documents string aggregation alias.
docs/user-guide/comparison.md	Updates feature comparison list to include JSON functions and PRINTF.
docs/about/changelog.md	Adds 1.4.0 entry describing new features/fixes.
design/adr/0104-sql-planner-aggregate-and-composite-pk-fixes.md	ADR for aggregate ORDER BY, composite PK rowid seek, UNION derived-table fixes.
design/adr/0103-efcore-primitive-collection-translation.md	ADR for EF Core primitive collection translation strategy (json_each rewrites).
design/adr/0102-json-scalar-functions.md	ADR for JSON scalar functions in the engine.
design/adr/0101-nodatime-member-translation-via-plugin.md	ADR for NodaTime member translation via EF Core plugin.
design/adr/0100-partial-index-query-planner-exclusion.md	ADR for excluding partial indexes from general planner index selection.
design/adr/0099-sqlite-type-affinity-comparisons.md	ADR for SQLite-compatible type affinity coercion in comparisons.
design/adr/0098-left-join-subquery-column-resolution.md	ADR for LEFT JOIN subquery column resolution fixes.
design/adr/0097-shared-library-embedding-safety.md	ADR for shared-library embedding flags and pager eviction strategy.
design/adr/0096-case-insensitive-identifiers.md	ADR for case-insensitive identifier resolution design.
design/INMEMORY_SUPPORT_PLAN.md	Adds an in-memory DB support plan document.
decentdb.nimble	Bumps version to 1.4.0; updates shared library build flags; documents test runner tasks.
bindings/dotnet/tests/DecentDB.Tests/SqliteCompatibilityTests.cs	Adds extensive compatibility tests for new SQL engine behaviors (affinity, partial index, UNION subqueries, composite PK, aggregates, JSON).
bindings/dotnet/tests/DecentDB.Tests/SqlParameterRewriterTests.cs	Adds tests for digit-prefixed named parameter rewriting.
bindings/dotnet/tests/DecentDB.EntityFrameworkCore.Tests/PrimitiveCollectionTests.cs	Adds EF Core primitive collection translation tests.
bindings/dotnet/tests/DecentDB.EntityFrameworkCore.Tests/NodaTimeIntegrationTests.cs	Expands NodaTime integration tests; changes schema to tick-based columns for Instant/LocalDateTime.
bindings/dotnet/tests/DecentDB.EntityFrameworkCore.Tests/DesignTimeToolingTests.cs	Updates EF design package version; hardens dotnet CLI invocation and native library staging.
bindings/dotnet/src/DecentDB.Native/DecentDB.cs	Formatting fixes; improves decimal/guid binding code style.
bindings/dotnet/src/DecentDB.EntityFrameworkCore/Storage/Internal/DecentDBDatabaseCreator.cs	Improves HasTables detection using provider-specific table listing JSON; adds async wrapper.
bindings/dotnet/src/DecentDB.EntityFrameworkCore/Query/Internal/SqlExpressions/JsonEachExpression.cs	Adds JsonEachExpression marker for primitive collection translation.
bindings/dotnet/src/DecentDB.EntityFrameworkCore/Query/Internal/DecentDBQueryableMethodTranslatingExpressionVisitorFactory.cs	Registers custom query translator factory for primitive collections.
bindings/dotnet/src/DecentDB.EntityFrameworkCore/Query/Internal/DecentDBQueryableMethodTranslatingExpressionVisitor.cs	Implements json_each interception and scalar-function rewrites for Any/Count/Contains/ElementAt.
bindings/dotnet/src/DecentDB.EntityFrameworkCore/Extensions/DecentDBServiceCollectionExtensions.cs	Registers the custom query translator factory.
bindings/dotnet/src/DecentDB.EntityFrameworkCore.NodaTime/Storage/Internal/DecentDBNodaTimeTypeMappingSource.cs	Switches Instant/LocalDateTime to tick precision and updates mappings; adds DateTime/DateTimeOffset mapping changes.
bindings/dotnet/src/DecentDB.EntityFrameworkCore.NodaTime/Query/Internal/DecentDBNodaTimeMemberTranslatorPlugin.cs	Adds EF Core member translator plugin wrapper.
bindings/dotnet/src/DecentDB.EntityFrameworkCore.NodaTime/Query/Internal/DecentDBNodaTimeMemberTranslator.cs	Adds NodaTime LocalDate date-part translation using integer arithmetic.
bindings/dotnet/src/DecentDB.EntityFrameworkCore.NodaTime/DecentDBNodaTimeOptionsExtension.cs	Registers NodaTime mappings and member translator plugin; updates service provider hash.
bindings/dotnet/src/DecentDB.AdoNet/SqlStatementSplitter.cs	Adds multi-statement splitting support for ADO.NET ExecuteNonQuery batching.
bindings/dotnet/src/DecentDB.AdoNet/SqlParameterRewriter.cs	Improves param name parsing; adds OFFSET clamping and UPDATE/DELETE alias stripping.
bindings/dotnet/src/DecentDB.AdoNet/DecentDBDataReader.cs	Small formatting tweak in decimal handling.
bindings/dotnet/src/DecentDB.AdoNet/DecentDBConnection.cs	Adds OpenAsync with cancellation token check.
bindings/dotnet/src/DecentDB.AdoNet/DecentDBCommand.cs	Adds multi-statement ExecuteNonQuery behavior; adds async methods w/ cancellation checks; applies new SQL rewrites (OFFSET clamp, alias stripping).
benchmarks/embedded_compare/data/bench_summary.json	Updates embedded benchmark summary metrics and run id.
benchmarks/embedded_compare/assets/decentdb-benchmarks.svg	Updates benchmark chart asset timestamp/content.
README.md	Updates feature list to include JSON scalar functions.
AGENTS.md	Adds commit hygiene guidance (show full diff before commit).

[Copilot]

This repro test hardcodes a /tmp path and calls removeFile(dbPath) unconditionally (throws if the file doesn’t exist and isn’t portable to Windows). Also, files under tests/ aren’t executed by nimble test_nim (which runs tests/nim/*.nim). Consider converting this into a proper unittest in tests/nim/ using getTempDir() and fileExists checks, or moving it under examples/ if it’s only a manual repro.

[Copilot]

This repro test hardcodes a /tmp path and calls removeFile(dbPath) unconditionally (throws if missing; not portable to Windows). Also, tests under tests/ aren’t picked up by nimble test_nim (tests/nim/*.nim only). Consider migrating to a unittest under tests/nim/ using getTempDir() + fileExists checks, or moving it to examples/ if it’s meant for manual runs.

[Copilot]

ClampOffsetParameters uses IndexOf("OFFSET", OrdinalIgnoreCase) without token/word-boundary checks or skipping quoted strings/comments, so it can match inside identifiers/text (e.g., "MyOffset", '...OFFSET...') and clamp unrelated parameters. Consider scanning tokens similarly to SqlStatementSplitter (skip strings/comments) and verify OFFSET is a standalone keyword before rewriting.

Suggested change

	/// <summary>
	/// Clamps OFFSET parameter values to 0 when negative.
	/// EF Core may generate negative OFFSET values from untrusted page numbers.
	/// </summary>
	public static void ClampOffsetParameters(string sql, Dictionary<int, DbParameter> paramMap)
	{
	var searchFrom = 0;
	while (true)
	{
	var idx = sql.IndexOf("OFFSET", searchFrom, StringComparison.OrdinalIgnoreCase);
	if (idx < 0) break;
	var afterOffset = idx + 6;
	while (afterOffset < sql.Length && sql[afterOffset] == ' ')
	afterOffset++;
	if (afterOffset < sql.Length && sql[afterOffset] == '$')
	{
	var numStart = afterOffset + 1;
	var numEnd = numStart;
	while (numEnd < sql.Length && char.IsDigit(sql[numEnd]))
	numEnd++;
	if (numEnd > numStart &&
	int.TryParse(sql.Substring(numStart, numEnd - numStart), out var paramIndex) &&
	paramMap.TryGetValue(paramIndex, out var param) &&
	param.Value is IConvertible conv)
	{
	try
	{
	var val = conv.ToInt64(null);
	if (val < 0)
	param.Value = 0L;
	}
	catch { /* not numeric — leave as-is */ }
	}
	}
	searchFrom = afterOffset;
	private static bool IsSqlIdentifierChar(char c)
	{
	return char.IsLetterOrDigit(c) || c == '_';
	}
	private static bool IsStandaloneOffsetKeyword(string sql, int index)
	{
	const int keywordLength = 6; // "OFFSET"
	if (index < 0 || index + keywordLength > sql.Length)
	return false;
	// Compare characters case-insensitively without allocating substrings.
	if (!((sql[index + 0] == 'O' || sql[index + 0] == 'o') &&
	(sql[index + 1] == 'F' || sql[index + 1] == 'f') &&
	(sql[index + 2] == 'F' || sql[index + 2] == 'f') &&
	(sql[index + 3] == 'S' || sql[index + 3] == 's') &&
	(sql[index + 4] == 'E' || sql[index + 4] == 'e') &&
	(sql[index + 5] == 'T' || sql[index + 5] == 't')))
	{
	return false;
	}
	// Ensure preceding character is not part of an identifier.
	if (index > 0 && IsSqlIdentifierChar(sql[index - 1]))
	return false;
	// Ensure following character is not part of an identifier.
	var after = index + keywordLength;
	if (after < sql.Length && IsSqlIdentifierChar(sql[after]))
	return false;
	return true;
	}
	/// <summary>
	/// Clamps OFFSET parameter values to 0 when negative.
	/// EF Core may generate negative OFFSET values from untrusted page numbers.
	/// </summary>
	public static void ClampOffsetParameters(string sql, Dictionary<int, DbParameter> paramMap)
	{
	if (string.IsNullOrEmpty(sql) || paramMap == null || paramMap.Count == 0)
	return;
	var length = sql.Length;
	var i = 0;
	while (i < length)
	{
	var c = sql[i];
	// Skip single-quoted string literals (with '' escapes).
	if (c == '\'')
	{
	i++;
	while (i < length)
	{
	if (sql[i] == '\'')
	{
	i++;
	// Escaped quote: '' -> stay inside the string.
	if (i < length && sql[i] == '\'')
	{
	i++;
	continue;
	}
	break;
	}
	i++;
	}
	continue;
	}
	// Skip double-quoted identifiers.
	if (c == '\"')
	{
	i++;
	while (i < length)
	{
	if (sql[i] == '\"')
	{
	i++;
	break;
	}
	i++;
	}
	continue;
	}
	// Skip bracketed identifiers: [ ... ].
	if (c == '[')
	{
	i++;
	while (i < length && sql[i] != ']')
	i++;
	if (i < length)
	i++;
	continue;
	}
	// Skip line comments: -- ... end of line.
	if (c == '-' && i + 1 < length && sql[i + 1] == '-')
	{
	i += 2;
	while (i < length && sql[i] != '\n' && sql[i] != '\r')
	i++;
	continue;
	}
	// Skip block comments: /* ... */.
	if (c == '/' && i + 1 < length && sql[i + 1] == '*')
	{
	i += 2;
	while (i + 1 < length && !(sql[i] == '*' && sql[i + 1] == '/'))
	i++;
	if (i + 1 < length)
	i += 2;
	continue;
	}
	// Check for standalone OFFSET keyword.
	if (c == 'O' || c == 'o')
	{
	if (IsStandaloneOffsetKeyword(sql, i))
	{
	var afterOffset = i + 6; // length of "OFFSET"
	while (afterOffset < length && sql[afterOffset] == ' ')
	afterOffset++;
	if (afterOffset < length && sql[afterOffset] == '$')
	{
	var numStart = afterOffset + 1;
	var numEnd = numStart;
	while (numEnd < length && char.IsDigit(sql[numEnd]))
	numEnd++;
	if (numEnd > numStart &&
	int.TryParse(sql.Substring(numStart, numEnd - numStart), out var paramIndex) &&
	paramMap.TryGetValue(paramIndex, out var param) &&
	param.Value is IConvertible conv)
	{
	try
	{
	var val = conv.ToInt64(null);
	if (val < 0)
	param.Value = 0L;
	}
	catch
	{
	// not numeric — leave as-is
	}
	}
	}
	// Continue scanning after the OFFSET token and any following spaces.
	i = Math.Max(i + 6, afterOffset);
	continue;
	}
	}
	i++;

[Copilot]

HasTablesAsync is marked async but doesn’t await anything and ignores the cancellationToken. Consider removing async and returning Task.FromResult(HasTables()) (or honoring cancellation by checking cancellationToken before calling HasTables).

Suggested change

	public override async Task<bool> HasTablesAsync(CancellationToken cancellationToken = default)
	=> HasTables();
	public override Task<bool> HasTablesAsync(CancellationToken cancellationToken = default)
	{
	cancellationToken.ThrowIfCancellationRequested();
	return Task.FromResult(HasTables());
	}

[Copilot]

Composite ON CONFLICT index lookup uses idx.columns == targetCols, which is case-sensitive and can miss the correct unique index under the new case-insensitive identifier semantics. Normalize column names (and ideally the table name) before comparing idx.columns to targetCols.

[Copilot]

UseNodaTime changes DateTime/DateTimeOffset storage from Unix-milliseconds (used in the base provider) to UtcTicks (0001-01-01 epoch). This creates inconsistent semantics across providers and can break existing databases/queries when toggling NodaTime. Consider keeping DateTime/DateTimeOffset mapping consistent with DecentDBTypeMappingSource (Unix ms) and only changing NodaTime-specific types (Instant/LocalDate/LocalDateTime).

[Copilot]

This repro test hardcodes a /tmp path and calls removeFile(dbPath) unconditionally (throws if missing; not portable). Also, tests under tests/ aren’t run by nimble test_nim (tests/nim/*.nim only). Consider migrating to a proper unittest with temp paths + fileExists checks, or relocating to examples/ if it’s just a manual reproducer.

[Copilot]

StripUpdateDeleteAlias rewrites by string Replace(alias + ".", tableName + ".") on the whole SQL text. This can also rewrite occurrences inside string literals/comments, and the quoted-identifier parsing via IndexOf('"') doesn’t handle escaped quotes (""). Consider a small tokenizer that skips strings/comments and rewrites only identifier tokens, including proper handling of doubled quotes in quoted identifiers.

[Copilot]

ON CONFLICT unique-target matching uses idx.columns == conflictTargetCols, which is case-sensitive. Since identifiers are now resolved case-insensitively, this can fail to match an existing unique index when quoting/casing differs. Compare normalized column names (e.g., map both lists through normalizedName) before equality checking.