From 06d708c3d21a77b2c798d52533e2672c37da9544 Mon Sep 17 00:00:00 2001 From: Gabe Kangas Date: Thu, 30 Jul 2026 16:45:21 -0700 Subject: [PATCH] docs(plugins): document private SQL storage Document the storage.sql permission, JavaScript and Python APIs, query limits, security boundary, and scenario test behavior. --- docs/plugins/apis.mdx | 88 ++++++++++++++++++++++++++++++++++++- docs/plugins/permissions.md | 13 +++++- docs/plugins/testing.mdx | 2 + 3 files changed, 100 insertions(+), 3 deletions(-) diff --git a/docs/plugins/apis.mdx b/docs/plugins/apis.mdx index be3bf01ac6..b0d0f8ee91 100644 --- a/docs/plugins/apis.mdx +++ b/docs/plugins/apis.mdx @@ -347,7 +347,7 @@ Requires `storage.upload`. ### `owncast.fs.*` -A private, sandboxed filesystem at `data/plugin-data//`. Unlike `owncast.storage.upload`, these files stay server-side: they're never served over HTTP. Paths are relative to your sandbox root. The host confines every path to your own directory (a plugin cannot read another plugin's files, and `../` or absolute paths collapse back inside the sandbox). Parent directories are created as needed on write. +A private, sandboxed filesystem at `data/plugin-storage//files/`. Unlike `owncast.storage.upload`, these files stay server-side: they're never served over HTTP. Paths are relative to your sandbox root. The host confines every path to your own directory (a plugin cannot read another plugin's files, and `../` or absolute paths collapse back inside the sandbox). Parent directories are created as needed on write. | Method | Returns | | ---------------------- | -------------------------------------------- | @@ -381,6 +381,91 @@ text = owncast.fs.read_text("notes/log.txt") Requires `storage.fs`. +### `owncast.sql.*` + +One private SQLite database per plugin, at `data/plugin-storage//db/plugin.db`, separate from Owncast's own database and from the [`storage.fs`](/docs/plugins/permissions#storagefs) sandbox. The sandbox is rooted at `files/`, so `db/` is not a path `owncast.fs.*` refuses but one it cannot express, and the filesystem quota walk covers `files/` only, so the two quotas stay independent. Reach for this over [`storage.kv`](/docs/plugins/permissions#storagekv) when you need to sort, filter, or aggregate rather than just remember a value. + +| Method | Returns | +| ---------------------------- | ---------------------------------------------------- | +| `sql.exec(sql, params?)` | `{ ok, error?, rowsAffected, lastInsertId }` | +| `sql.query(sql, params?)` | rows as objects keyed by column name | +| `sql.queryRow(sql, params?)` | the first row object, or `null` when nothing matched | + +In Python `queryRow` is `query_row`, rows come back as dicts, and `query_row` returns `None` when nothing matched. An error throws in JavaScript and raises `RuntimeError` in Python, so you don't have to check `ok` yourself. Parameters are `null`/`None`, booleans, numbers, or strings. Anything else is refused. + + + + +```js +owncast.sql.exec(`CREATE TABLE IF NOT EXISTS chatters ( + user_id TEXT PRIMARY KEY, + messages INTEGER NOT NULL DEFAULT 0 +)`); + +owncast.sql.exec( + `INSERT INTO chatters (user_id, messages) VALUES (?, 1) + ON CONFLICT (user_id) DO UPDATE SET messages = messages + 1`, + [msg.user.id], +); + +const top = owncast.sql.query( + 'SELECT user_id, messages FROM chatters ORDER BY messages DESC LIMIT ?', + [5], +); +const mine = owncast.sql.queryRow('SELECT messages FROM chatters WHERE user_id = ?', [msg.user.id]); +``` + + + + +```python +owncast.sql.exec(""" + CREATE TABLE IF NOT EXISTS chatters ( + user_id TEXT PRIMARY KEY, + messages INTEGER NOT NULL DEFAULT 0 + ) +""") + +owncast.sql.exec( + """ + INSERT INTO chatters (user_id, messages) VALUES (?, 1) + ON CONFLICT (user_id) DO UPDATE SET messages = messages + 1 + """, + [msg.user.id], +) + +top = owncast.sql.query( + "SELECT user_id, messages FROM chatters ORDER BY messages DESC LIMIT ?", + [5], +) +mine = owncast.sql.query_row("SELECT messages FROM chatters WHERE user_id = ?", [msg.user.id]) +``` + + + + +Each `exec` call runs as one host-owned transaction. A multi-statement batch commits whole or leaves the database untouched, so a schema migration can't half-apply. A plugin cannot leave a transaction open across calls, so there's nothing to clean up either. + +`query` never hands back a silently short result. A query that overruns the row cap or the result budget is an **error** telling you to add a `LIMIT`, so write the bound you actually want when a table grows with your audience. `queryRow` reads a single row, which keeps it cheap on a table `query` is too big for. + +| Limit | Value | +| --------------------- | ----------------------------- | +| Encoded request | 64 KiB total JSON | +| Bound parameters | 64 per call | +| Returned column value | 1 MiB | +| Encoded query result | 1 MiB | +| Rows returned | 10000 | +| Call duration | 2 seconds | +| Database size | 128 MiB | + +Ordinary SQL is unaffected: DDL, DML, indexes, views, triggers, `ORDER BY`, recursive CTEs, subqueries, `UNION`, and the json1 functions all work. Refused in every host: `ATTACH`, `DETACH`, every `PRAGMA` (reads included), temporary-schema DDL both as keywords (`CREATE TEMP TABLE` / `INDEX` / `TRIGGER` / `VIEW`) and schema-qualified (`CREATE TABLE temp.x`), `load_extension()`, `VACUUM` and `VACUUM INTO`, and transaction controls (`BEGIN`, `COMMIT`, `END`, `ROLLBACK`, `SAVEPOINT`, and `RELEASE`). `exec` already owns the transaction around the whole batch. + +::::caution[JavaScript loses precision above 2^53] +Parameters and results cross the host boundary as JSON. Python can bind and read exact 64-bit SQLite integers. JavaScript loses unsafe integers before `JSON.stringify` on writes and during `JSON.parse` on reads. Store values above `Number.MAX_SAFE_INTEGER` (2^53 - 1) as TEXT when a JavaScript plugin needs them to remain exact. +:::: + +Requires `storage.sql`. For a worked example, the `chat-leaderboard` plugin covers schema creation in one atomic `exec`, an `ON CONFLICT` upsert, a bounded ranked `query`, and a single-row read, in both [JavaScript](https://github.com/owncast/plugin-sdk/tree/main/examples/js/chat-leaderboard) and [Python](https://github.com/owncast/plugin-sdk/tree/main/examples/python/chat-leaderboard). It contrasts with `message-counter`, which keeps the same counts in [`storage.kv`](/docs/plugins/permissions#storagekv) and cannot rank. + ## Config ### `owncast.config.get(key, fallback?)` @@ -737,6 +822,7 @@ Method names below are the JavaScript (camelCase) form. The Python equivalents a | `owncast.kv.get` / `.set` / `.getJSON` / `.setJSON` | `storage.kv` | | `owncast.storage.upload` | `storage.upload` | | `owncast.fs.read` / `.readText` / `.write` / `.list` / `.delete` / `.exists` | `storage.fs` | +| `owncast.sql.exec` / `.query` / `.queryRow` | `storage.sql` | | `owncast.http.fetch` | `network.fetch` | | `owncast.events.emit` | `events.emit` | | `owncast.stream.current` | `server.read` | diff --git a/docs/plugins/permissions.md b/docs/plugins/permissions.md index f49bc1f716..6f85f2e6e5 100644 --- a/docs/plugins/permissions.md +++ b/docs/plugins/permissions.md @@ -137,10 +137,18 @@ Grants `owncast.storage.upload(name, bytes)`: upload a file to Owncast's public ### `storage.fs` -Grants `owncast.fs.*`: a private, sandboxed filesystem at `data/plugin-data//` that your plugin can read, write, list, and delete within. Useful for caches, generated data files, append-style logs, or anything you need to persist as real files rather than key/value strings. +Grants `owncast.fs.*`: a private, sandboxed filesystem at `data/plugin-storage//files/` that your plugin can read, write, list, and delete within. Useful for caches, generated data files, append-style logs, or anything you need to persist as real files rather than key/value strings. Unlike `storage.upload`, these files stay **server-side**: they're never served over HTTP. Every path is confined to your plugin's own directory: a plugin cannot read another plugin's files or escape its sandbox (`../` and absolute paths are collapsed back inside). +### `storage.sql` + +Grants `owncast.sql.*`: one private SQLite database per plugin, at `data/plugin-storage//db/plugin.db`. `owncast.sql.exec(sql, params?)` runs statements, `owncast.sql.query(sql, params?)` returns matching rows, and `owncast.sql.queryRow(sql, params?)` reads a single row. Reach for this instead of `storage.kv` when you need to sort, filter, or aggregate rather than just remember a value. See [`owncast.sql.*`](/docs/plugins/apis#owncastsql) for the methods in both languages, the per-call limits, and the SQL the host refuses. + +The database is private to your plugin and separate from Owncast's own database. The `storage.fs` sandbox is rooted at `files/`, so `db/` is not a path `owncast.fs.*` refuses but one it **cannot express**, and the filesystem quota walk covers `files/` only, so the two quotas stay independent: the database has its own 128 MiB cap, and files written through `storage.fs` count against a separate 256 MiB quota. + +Plugin databases are **not** included in Owncast's database backups, so treat the contents as rebuildable or export what matters yourself. SQL data is retained when a plugin is uninstalled, the same as its config and its `storage.fs` files, so a reinstall finds its tables where it left them. An admin who wants the space back deletes `data/plugin-storage//`. + ### `network.fetch` Grants `owncast.http.fetch(url, opts?)`: synchronous outbound HTTP. @@ -252,7 +260,8 @@ None of the four viewer-injection fields require `http.serve`, and neither do th | `auth.gate` | `owncast.auth.grantSession`, `.endSession`, and the `onAuthCheck` handler: be the site's auth gate | | `storage.kv` | Per-plugin namespaced key/value store | | `storage.upload` | Upload files to Owncast's public file area | -| `storage.fs` | Private, sandboxed server-side filesystem at `data/plugin-data//` | +| `storage.fs` | Private, sandboxed server-side filesystem at `data/plugin-storage//files/` | +| `storage.sql` | Private per-plugin SQLite database at `data/plugin-storage//db/plugin.db` | | `network.fetch` | Outbound HTTP. Also requires `network.allowedHosts` | | `events.emit` | Emit custom events for other plugins | | `http.serve` | Serve HTTP at `/plugins//*` | diff --git a/docs/plugins/testing.mdx b/docs/plugins/testing.mdx index 753ddd0601..3f142f5d9d 100644 --- a/docs/plugins/testing.mdx +++ b/docs/plugins/testing.mdx @@ -225,6 +225,8 @@ The scenario's top-level `expect` checks what happened across the whole run: `owncast.fs.*` (the `storage.fs` sandbox) has no dedicated assertion: the runtime backs it with a real in-memory sandbox during tests, so test it the way you'd use it: drive your plugin's own endpoints (or handlers) and assert on what they return. For example, `POST` a file through your upload endpoint, then `GET` your list endpoint and assert the response includes it. The [`file-manager`](https://github.com/owncast/plugin-sdk/tree/main/examples) example does exactly this. +[`owncast.sql.*`](/docs/plugins/apis#owncastsql) works the same way. The test runner and the dev server give each plugin a real in-memory SQLite database, so there's no SQL assertion and no `given.sql`: every scenario starts with an empty database and your plugin creates its own schema on first use. Drive the handlers or commands that write, then assert on what the ones that read send back. The same statements are refused there as on a real server and the same per-call limits apply, so a scenario that passes runs the same SQL in production. The `chat-leaderboard` example ([JavaScript](https://github.com/owncast/plugin-sdk/tree/main/examples/js/chat-leaderboard), [Python](https://github.com/owncast/plugin-sdk/tree/main/examples/python/chat-leaderboard)) is tested exactly this way: chat events count messages, then `!top` and `!rank` report the standings. + Example exercising several: ```json