feat(web): OPFS-backed SQLite driver + local dev script#74
Merged
Conversation
Replaces the browser demo's hard-coded IN_MEMORY + DemoFileSystem stack with a real durable storage layer using @sqlite.org/sqlite-wasm and the OPFS SAH-pool VFS, so graph data persists across browser sessions. - Add WasmOpfsSqlDriver: full SqlDriver backed by a JS Web Worker running @sqlite.org/sqlite-wasm with opfs-sahpool; all ops return QueryResult.AsyncValue - Add SqliteWorkerInterop.kt, JsBindCollector, JsRowCursor supporting files - Add sqlite-stelekit-worker.js: module worker initialising opfs-sahpool, falls back to :memory: if OPFS is unavailable (private browsing) - Add OpfsInterop.kt: suspend helpers for OPFS directory walk, file read/write - Rewrite PlatformFileSystem (wasmJs): in-memory cache pre-loaded from OPFS at startup; writes are cache-first with async OPFS write-through - Rewrite browser/Main.kt: async OPFS init outside Compose tree; sets window.__stelekit_ready for Playwright test synchronisation - Update DriverFactory.js.kt: createDriverAsync() caches the pre-init driver so that createDriver() (called by RepositoryFactory) returns it - Make MigrationRunner.applyAll() suspend; wrap JVM/Android call sites in runBlocking to stay backwards-compatible - Enable generateAsync = true in SQLDelight to support async SqlDriver - Add @sqlite.org/sqlite-wasm@3.46.1 npm dep to wasmJsMain - Add scripts/serve-web.sh: single command to build and preview locally - Extend e2e/tests/demo.spec.ts: OPFS persistence test + beforeEach cleanup Note: after the first build, verify sqlite-stelekit-worker.js and sqlite3.wasm appear in kmp/build/dist/wasmJs/productionExecutable/. If the worker path differs, update workerScriptPath in DriverFactory.js.kt. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Planning artifacts for the OPFS SQLite browser storage feature: requirements, architecture research (stack, features, pitfalls), implementation plan, and test validation plan. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds durable, browser-persistent storage for the WASM target by introducing an OPFS-backed SQLite worker/driver, replaces the previous demo-only in-memory stack, and includes a convenience script for local web dev.
Changes:
- Add a Web Worker (
@sqlite.org/sqlite-wasm) using OPFS SAH pool VFS and a Kotlin/WASMSqlDriverbridge returning asyncQueryResults. - Replace WASM stubs with OPFS-backed
PlatformFileSystem+ async driver initialization before starting the Compose tree. - Make
MigrationRunner.applyAll()suspend and adapt JVM/Android call sites; add Playwright OPFS persistence coverage and a local serve script.
Reviewed changes
Copilot reviewed 23 out of 23 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| scripts/serve-web.sh | Build WASM distribution and start local server pointing at the dist dir. |
| kmp/build.gradle.kts | Add sqlite-wasm npm dep and enable SQLDelight async result generation. |
| kmp/src/wasmJsMain/resources/sqlite-stelekit-worker.js | New sqlite-wasm worker using OPFS SAH pool VFS with in-memory fallback. |
| kmp/src/wasmJsMain/kotlin/dev/stapler/stelekit/db/SqliteWorkerInterop.kt | JS interop helpers for worker messaging and row/value access. |
| kmp/src/wasmJsMain/kotlin/dev/stapler/stelekit/db/JsBindCollector.kt | Prepared-statement binder collecting parameters into a JS array. |
| kmp/src/wasmJsMain/kotlin/dev/stapler/stelekit/db/JsRowCursor.kt | Cursor implementation over worker-returned JS row objects. |
| kmp/src/wasmJsMain/kotlin/dev/stapler/stelekit/db/WasmOpfsSqlDriver.kt | New SQLDelight driver that proxies SQL execution to the worker. |
| kmp/src/wasmJsMain/kotlin/dev/stapler/stelekit/db/DriverFactory.js.kt | Replace stub with cached async driver init + schema/migrations. |
| kmp/src/wasmJsMain/kotlin/dev/stapler/stelekit/platform/OpfsInterop.kt | OPFS read/write/delete helpers via navigator.storage.getDirectory(). |
| kmp/src/wasmJsMain/kotlin/dev/stapler/stelekit/platform/PlatformFileSystem.kt | Replace stub with OPFS-preloaded in-memory cache and async write-through. |
| kmp/src/wasmJsMain/kotlin/dev/stapler/stelekit/browser/Main.kt | Perform OPFS preload + driver init before Compose window starts. |
| kmp/src/commonMain/kotlin/dev/stapler/stelekit/db/MigrationRunner.kt | Make applyAll suspend and await async SQLDelight operations. |
| kmp/src/jvmMain/kotlin/dev/stapler/stelekit/db/DriverFactory.jvm.kt | Wrap schema create + migrations in runBlocking for suspend migration runner. |
| kmp/src/androidMain/kotlin/dev/stapler/stelekit/db/DriverFactory.android.kt | Wrap migrations in runBlocking for suspend migration runner. |
| kmp/src/androidMain/kotlin/dev/stapler/stelekit/db/DriverFactory.android.kt.tmp.4573.1775983611619 | Delete stray temporary file from the repo. |
| e2e/tests/demo.spec.ts | Clear OPFS between tests and add persistence-across-reload test. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| actual override fun readFile(path: String): String? = cache[path] | ||
| actual override fun fileExists(path: String): Boolean = cache.containsKey(path) | ||
| actual override fun listFiles(path: String): List<String> = | ||
| cache.keys.filter { it.startsWith("$path/") && !it.removePrefix("$path/").contains('/') } |
Owner
Author
There was a problem hiding this comment.
Fixed: listFiles() now strips the path prefix and returns just names via .map { it.removePrefix("$path/") }.
| .map { it.removePrefix("$path/").substringBefore('/') } | ||
| .filter { it.isNotEmpty() && cache.keys.any { k -> k.startsWith("$path/$it/") } } | ||
| .distinct() | ||
| .map { "$path/$it" } |
Owner
Author
There was a problem hiding this comment.
Fixed: listDirectories() now returns directory names only, matching the JVM implementation.
Comment on lines
71
to
72
| actual override fun directoryExists(path: String): Boolean = true | ||
| actual override fun createDirectory(path: String): Boolean = true |
Owner
Author
There was a problem hiding this comment.
Fixed: directoryExists() now checks the cache (path == homeDir || cache.keys.any { it.startsWith("$path/") }) instead of always returning true.
Comment on lines
+21
to
+25
| SteleDatabase.Schema.create(driver).await() | ||
| MigrationRunner.applyAll(driver) | ||
| cachedDriver = driver | ||
| return driver | ||
| } |
Owner
Author
There was a problem hiding this comment.
Fixed: SteleDatabase.Schema.create(driver) is now wrapped in a try/catch — the exception is swallowed as benign when tables already exist on a persisted OPFS database.
Comment on lines
+29
to
35
| val backend = try { | ||
| driverFactory.createDriverAsync(graphId) | ||
| GraphBackend.SQLDELIGHT | ||
| } catch (e: Throwable) { | ||
| println("[SteleKit] OPFS driver init failed, using IN_MEMORY: ${e.message}") | ||
| GraphBackend.IN_MEMORY | ||
| } |
Owner
Author
There was a problem hiding this comment.
Fixed: the code now checks driver.actualBackend == "memory" after a successful createDriverAsync() call and falls back to GraphBackend.IN_MEMORY when the worker silently fell back to :memory:.
Comment on lines
+8
to
+16
| // Clear OPFS stelekit directory before each test to prevent inter-test bleed | ||
| await page.goto('/'); | ||
| await page.evaluate(async () => { | ||
| try { | ||
| const root = await navigator.storage.getDirectory(); | ||
| await root.removeEntry('stelekit', { recursive: true }); | ||
| } catch { | ||
| // Directory may not exist on first run | ||
| } |
Owner
Author
There was a problem hiding this comment.
The beforeEach goto is only to gain same-origin access for the OPFS cleanup evaluate call — the app doesn't fully initialize there. The actual test's goto('/') starts the app under the listener. The ordering is intentional; leaving as-is.
Comment on lines
+71
to
+92
| override fun newTransaction(): QueryResult<Transacter.Transaction> = QueryResult.AsyncValue { | ||
| val id = nextMsgId() | ||
| val promise = createWorkerResponsePromise(worker, id) | ||
| workerPostMessage(worker, buildTransactionBeginMessage(id)) | ||
| promise.await() | ||
| object : Transacter.Transaction() { | ||
| override val enclosingTransaction: Transacter.Transaction? = null | ||
| override fun endTransaction(successful: Boolean): QueryResult<Unit> = | ||
| this@WasmOpfsSqlDriver.endTransaction(successful) | ||
| } | ||
| } | ||
|
|
||
| override fun endTransaction(successful: Boolean): QueryResult<Unit> = QueryResult.AsyncValue { | ||
| val id = nextMsgId() | ||
| val promise = createWorkerResponsePromise(worker, id) | ||
| workerPostMessage(worker, buildTransactionEndMessage(id, successful)) | ||
| promise.await() | ||
| Unit | ||
| } | ||
|
|
||
| override fun currentTransaction(): Transacter.Transaction? = null | ||
|
|
Owner
Author
There was a problem hiding this comment.
Good catch. Tracking the active transaction and wiring enclosingTransaction correctly requires more state management in the Wasm driver. Deferring to a follow-up to keep this PR focused on the initial OPFS integration.
…x Detekt violations - RestrictedDatabaseQueries: fix missing spaces in method names (fun→suspend fun propagation dropped the space character), add suspend modifier to all write methods to match generateAsync=true API (returns Long not QueryResult<Long>) - Propagate suspend up the call chain: OperationLogger, ChangelogRepository, DebugFlagRepository, HistogramRetentionJob, SqlDelightPageRepository.upsertPage, DatabaseWriteActor.logSaveBlocks — all callers are already in coroutine contexts - PooledJdbcSqliteDriverTest: wrap transaction call in runBlocking for test context - SearchSkeleton: add modifier param to SearchSkeletonList (ModifierMissing) - SearchDialog: extract Regex constants, fix SearchResultRow param order, add modifier to ActivePrefixChipRow, extract complex condition to named booleans - SearchViewModel: rethrow CancellationException before generic catch Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Contributor
JVM Load Benchmark (Desktop)Synthetic in-memory benchmark measuring load performance for the desktop (JVM) app.
Flamegraphs (this PR)**Allocation** — object allocation pressure (JDBC/SQLite churn)Alloc flamegraph not available CPU — method-level hotspots by on-CPU time CPU flamegraph not available Top SQL queries by total time (this PR)| table:operation | calls | p50 | p99 | max | total | |-----------------|-------|-----|-----|-----|-------| | `pages:select` | 2 | 1ms | 1ms | 1ms | 1ms |Top allocation hotspots (this PR)`67.6%` byte[]_[k] `4.5%` java.lang.Object[]_[k] `3.9%` jdk.internal.org.objectweb.asm.SymbolTable$Entry_[k] `2.8%` java.lang.String_[k] `2.2%` java.lang.StringBuilder_[k]Top CPU hotspots (this PR)`99.1%` /usr/lib/x86_64-linux-gnu/libc.so.6 `0.1%` SR_handler `0.1%` pthread_cond_broadcast `0.1%` java/lang/Boolean.booleanValue_[1] `0.1%` inflate_fast |
…nd fix param order With generateAsync=true, SteleDatabase.Schema is SqlSchema<QueryResult.AsyncValue<Unit>> but AndroidSqliteDriver requires SqlSchema<QueryResult.Value<Unit>>. Add async-extensions dependency and use .synchronous() adapter to bridge the type mismatch. Also fix SearchSkeletonList ComposableParamOrder: modifier must precede other optional params. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Remove invalid `override` from WasmOpfsSqlDriver.endTransaction — SqlDriver interface has no endTransaction method in SQLDelight 2.3.2 - Extract js() calls in OpfsInterop suspend functions to private non-suspend helpers (Kotlin/Wasm restriction: js() must be a single top-level expression) - Extract js() call in Main.kt launch block to top-level markSteleKitReady() for the same reason Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Kotlin/Wasm cannot infer T for Promise<JsAny>.await() when there is no expected type in context (variable without annotation, discarded result). Add explicit `: JsAny` type on stored results and @Suppress UNUSED_VARIABLE for discarded awaits. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- PlatformFileSystem.listFiles/listDirectories: return names not full paths, matching JVM behavior (callers re-prefix with the parent path) - PlatformFileSystem.directoryExists: check cache instead of always true - DriverFactory.js: guard Schema.create in try/catch so persisted OPFS databases don't fail on second load when tables already exist - Main.kt: check driver.actualBackend == "memory" after createDriverAsync to detect silent worker fallback and switch to IN_MEMORY backend Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…AndroidSqliteDriver crash AndroidSqliteDriver.executeQuery calls QueryResult.getValue() on the mapper result. The previous mapper returned QueryResult.AsyncValue, which inherits DefaultImpls.getValue() that throws "The driver used with SQLDelight is asynchronous". AndroidCursor.next() and JsRowCursor.next() both return QueryResult.Value, so cursor.next().value is safe on all platforms. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Contributor
Android Load BenchmarkInstrumented benchmark on an API 30 x86_64 emulator — 500-page synthetic graph. Comparing Graph Load
Interactive Write Latency (during Phase 3)
SAF I/O Overhead (ContentProvider vs direct File read)Measures Binder IPC cost added by ContentResolver per readFile() call.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What?
Implements durable browser storage for the SteleKit WASM target, replacing the hard-coded
IN_MEMORY + DemoFileSystemdemo stack with a real SQLite database persisted in the browser's Origin Private File System (OPFS). Also adds a single-command local dev script.Why?
The browser app previously lost all data on every page reload — the
DriverFactory.js.ktstub threwUnsupportedOperationExceptionwith a "Phase B" TODO. This PR completes Phase B: graph data now survives reloads and is isolated per origin. Additionally, there was no convenient way to build and preview the WASM app locally.How?
Storage architecture:
sqlite-stelekit-worker.js— ES module Web Worker that initialises@sqlite.org/sqlite-wasmwith theopfs-sahpoolVFS (3–4× faster than the async proxy VFS, no SharedArrayBuffer required, Safari 16.4+ compatible)WasmOpfsSqlDriver— customSqlDriverbridging Kotlin/WASM ↔ worker viapostMessage/Promise. All ops returnQueryResult.AsyncValue(requiresgenerateAsync = true)DriverFactory.js.kt—createDriverAsync()pre-initialises the driver before the Compose tree starts; caches it socreateDriver()(called byRepositoryFactory) returns the same instancePlatformFileSystem(wasmJs) — in-memory cache pre-loaded from OPFS at startup; writes are cache-first with async OPFS write-throughbackend: 'memory'and the app falls back toIN_MEMORYsilentlyMigrationRunner:
applyAll()is nowsuspend; JVM/Android call sites wrapped inrunBlocking(no behaviour change on those platforms).Local dev:
scripts/serve-web.shbuilds:kmp:wasmJsBrowserDistribution -PenableJs=truethen startsnode e2e/server.mjspointing at the output.Testing
New e2e tests (
e2e/tests/demo.spec.ts)beforeEachclears the OPFSstelekitdirectory for test isolation/, waits forwindow.__stelekit_ready, reloads, waits again, asserts the OPFSstelekitdirectory exists (proves the driver wrote to OPFS, not just memory)Existing test
The original canvas-renders test is unchanged and still runs first.
How to test locally
Manual verification
http://localhost:8787Type of Change
Follow-up Required (R2 risk)
After the first CI build, verify that both of these files appear in
kmp/build/dist/wasmJs/productionExecutable/:sqlite-stelekit-worker.js(fromwasmJsMain/resources/)sqlite3.wasm(from@sqlite.org/sqlite-wasmnpm package)If the worker path is different in the webpack output, update the
workerScriptPathliteral inDriverFactory.js.kt.Files Changed
scripts/serve-web.shkmp/build.gradle.kts@sqlite.org/sqlite-wasm@3.46.1npm dep; enablegenerateAsync = truekmp/src/wasmJsMain/resources/sqlite-stelekit-worker.jskmp/src/wasmJsMain/.../db/SqliteWorkerInterop.ktjs(...)interop helperskmp/src/wasmJsMain/.../db/JsBindCollector.ktSqlPreparedStatementimplkmp/src/wasmJsMain/.../db/JsRowCursor.ktSqlCursorimpl over JS row objectskmp/src/wasmJsMain/.../db/WasmOpfsSqlDriver.ktSqlDrivervia workerkmp/src/wasmJsMain/.../db/DriverFactory.js.ktkmp/src/wasmJsMain/.../platform/OpfsInterop.ktkmp/src/wasmJsMain/.../platform/PlatformFileSystem.ktkmp/src/wasmJsMain/.../browser/Main.ktkmp/src/commonMain/.../db/MigrationRunner.ktapplyAll→suspend funkmp/src/jvmMain/.../db/DriverFactory.jvm.ktapplyAllinrunBlockingkmp/src/androidMain/.../db/DriverFactory.android.ktapplyAllinrunBlockinge2e/tests/demo.spec.ts🤖 Generated with Claude Code