WIP: Orgs

[kim]

Public half of orgs feature

[kim]

/update-llm-benchmark

[clockwork-labs-bot]

LLM Benchmark Results (ci-quickfix)

Language	Mode	Category	Tests Passed	Task Pass %
Rust	rustdoc_json	basics	26/27	91.7% ⬆️ +47.2%
Rust	rustdoc_json	schema	22/34	62.0% ⬆️ +35.5%
Rust	rustdoc_json	total	48/61	78.2% ⬆️ +41.9%
Rust	docs	basics	5/27	11.1%
Rust	docs	schema	8/34	20.5%
Rust	docs	total	13/61	15.4%
C#	docs	basics	27/27	100.0% ⬆️ +38.9%
C#	docs	schema	18/32	53.7% ⬆️ +22.2%
C#	docs	total	45/59	78.9% ⬆️ +31.3%

Compared against master branch baseline

_{Generated at: 2026-01-21T19:00:57.662Z}

Failure Analysis (click to expand)

Benchmark Failure Analysis

Generated from: /__w/SpacetimeDB/SpacetimeDB/tools/xtask-llm-benchmark/../../docs/llms/docs-benchmark-details.json

Summary

• Total failures analyzed: 36

---

Analysis of SpacetimeDB Benchmark Failures

---

Rust / rustdoc_json Failures

1. Compile/Publish Errors (2 failures)

t_002_scheduled_table

1. The generated code:

   use spacetimedb::{table, reducer, ReducerContext, Table, ScheduleAt};
   
   #[table(name = tick_timer, scheduled(reducer = tick, column = scheduled_at))]
   pub struct TickTimer {
       #[primary_key]
       #[auto_inc]
       scheduled_id: u64,
       scheduled_at: ScheduleAt,
   }
   
   #[reducer(init)]
   pub fn init(ctx: &ReducerContext) {
       if ctx.db.tick_timer().count() == 0 {
           ctx.db
               .tick_timer()
               .insert(TickTimer { scheduled_id: 0, scheduled_at: ScheduleAt::RepeatMicros(50_000) });
       }
   }
   
   #[reducer]
   pub fn tick(_ctx: &ReducerContext, _scheduled_id: u64) {
   }

2. The golden example:

   use spacetimedb::{reducer, table, ReducerContext, ScheduleAt, Table};
   use std::time::Duration;
   
   #[table(name = tick_timer, scheduled(tick))]
   pub struct TickTimer {
       #[primary_key]
       #[auto_inc]
       pub scheduled_id: u64,
       pub scheduled_at: ScheduleAt,
   }
   
   #[reducer]
   pub fn tick(_ctx: &ReducerContext, _row: TickTimer) -> Result<(), String> {
       Ok(())
   }
   
   #[reducer(init)]
   pub fn init(ctx: &ReducerContext) -> Result<(), String> {
       ctx.db.tick_timer().insert(TickTimer {
           scheduled_id: 0,
           scheduled_at: ScheduleAt::Interval(Duration::from_millis(50).into()),
       });
       Ok(())
   }

3. The error: publish_error: spacetime publish failed (exit=1)

4. Explain the difference:

   • API Changes: The #[table] macro attributes are updated to use scheduled(tick) instead of scheduled(reducer = tick, column = scheduled_at).
   • Function Signatures: The reducer signatures require Result<(), String> instead of no return type.

5. Root cause: The documentation is unclear about the required attributes for the #[table] and return types for reducers.

6. Recommendation:

   • Update documentation to specify the new attributes and return types for reducers.
   • Clearly denote the differences in examples, focusing on transitions from deprecated to current syntax.

---

t_017_scheduled_columns

1. The generated code:

   use spacetimedb::{reducer, table, ReducerContext, ScheduleAt, Table};
   
   #[table(name = tick_timer, schedule(reducer = tick, column = scheduled_at))]
   pub struct TickTimer {
       #[primary_key]
       #[auto_inc]
       scheduled_id: u64,
       scheduled_at: ScheduleAt,
   }
   
   #[reducer(init)]
   pub fn init(ctx: &ReducerContext) {
       if ctx.db.tick_timer().count() == 0 {
           ctx.db.tick_timer().insert(TickTimer {
               scheduled_id: 0,
               scheduled_at: ScheduleAt::repeat_us(50_000),
           });
       }
   }
   
   #[reducer(scheduled)]
   pub fn tick(_ctx: &ReducerContext, _row: TickTimer) {}

2. The golden example:

   use spacetimedb::{reducer, table, ReducerContext, ScheduleAt, Table};
   use std::time::Duration;
   
   #[table(name = tick_timer, scheduled(tick))]
   pub struct TickTimer {
       #[primary_key]
       #[auto_inc]
       pub scheduled_id: u64,
       pub scheduled_at: ScheduleAt,
   }
   
   #[reducer]
   pub fn tick(_ctx: &ReducerContext, _schedule: TickTimer) {
   }
   
   #[reducer(init)]
   pub fn init(ctx: &ReducerContext) {
       let every_50ms: ScheduleAt = Duration::from_millis(50).into();
       ctx.db.tick_timer().insert(TickTimer {
           scheduled_id: 0,
           scheduled_at: every_50ms,
       });
   }

3. The error: publish_error: spacetime publish failed (exit=1)

4. Explain the difference:

   • There is an incorrect attribute configuration for the #[table], which uses schedule instead of scheduled.
   • The use of ScheduleAt::repeat_us(50_000) should be Duration::from_millis(50).into().

5. Root cause: Documentation does not clarify the current method for setting scheduled attributes or expected types for duration configuration.

6. Recommendation: Revise examples to reinforce proper usage of Scheduled and required duration adjustments.

---

2. Other Failures (6 failures)

t_012_spacetime_product_type

1. The generated code:

   use spacetimedb::{table, reducer, ReducerContext, Table, SpacetimeType};
   
   #[derive(SpacetimeType)]
   pub struct Score {
       left: i32,
       right: i32,
   }
   
   #[table(name = result)]
   pub struct ResultRow {
       #[primary_key]
       id: i32,
       value: Score,
   }
   
   #[reducer]
   pub fn set_score(ctx: &ReducerContext, id: i32, left: i32, right: i32) {
       ctx.db.result().insert(ResultRow {
           id,
           value: Score { left, right },
       });
   }

2. The golden example:

   use spacetimedb::{reducer, table, ReducerContext, SpacetimeType, Table};
   
   #[derive(SpacetimeType, Clone, Debug)]
   pub struct Score {
       pub left: i32,
       pub right: i32,
   }
   
   #[table(name = result)]
   pub struct ResultRow {
       #[primary_key]
       pub id: i32,
       pub value: Score,
   }
   
   #[reducer]
   pub fn set_score(ctx: &ReducerContext, id: i32, left: i32, right: i32) {
       ctx.db.result().insert(ResultRow { id, value: Score { left, right } });
   }

3. The error: product_type_row_parity: spacetime sql failed

4. Explain the difference:

   • Missing Clone and Debug traits for the Score structure which are likely required for the DB operations.

5. Root cause: Documentation does not detail which traits are mandatory for struct definitions used with SpacetimeDB.

6. Recommendation: Include a section in the documentation explaining the necessary traits (like Clone and Debug) for data structures.

---

t_018_constraints

1. The generated code:

   use spacetimedb::{table, reducer, ReducerContext, Table};
   
   #[table(name = account, index(name = by_name, btree(columns = [name])))]
   pub struct Account {
       #[primary_key]
       id: i32,
       #[unique]
       email: String,
       name: String,
   }
   
   #[reducer]
   pub fn seed(ctx: &ReducerContext) {
       let ids: Vec<i32> = ctx.db.account().iter().map(|row| row.id).collect();
       for id in ids {
           ctx.db.account().id().delete(&id);
       }
       ctx.db.account().insert(Account { id: 1, email: "a@example.com".to_string(), name: "Alice".to_string() });
       ctx.db.account().insert(Account { id: 2, email: "b@example.com".to_string(), name: "Bob".to_string() });
   }

2. The golden example:

   use spacetimedb::{reducer, table, ReducerContext, Table};
   
   #[table(name = account, index(name = by_name, btree(columns = [name])))]
   pub struct Account {
       #[primary_key]
       pub id: i32,
       #[unique]
       pub email: String,
       pub name: String,
   }
   
   #[reducer]
   pub fn seed(ctx: &ReducerContext) {
       ctx.db.account().insert(Account { id: 1, email: "a@example.com".into(), name: "Alice".into() });
       ctx.db.account().insert(Account { id: 2, email: "b@example.com".into(), name: "Bob".into() });
   }

3. The error: constraints_row_parity_after_seed: spacetime sql failed

4. Explain the difference:

   • Missing pub visibility keywords for the fields in the Account struct that could affect its accessibility.

5. Root cause: Documentation lacks specificity on the necessity of visibility annotations for struct fields.

6. Recommendation: Expand examples to include default visibility settings or clearly note best practices for struct field access modifiers.

---

Rust / docs Failures (22 total)

1. Other Failures (22 failures)

t_000_empty_reducers

1. The generated code:

   use spacetimedb::ReducerContext;
   
   #[spacetimedb::reducer]
   pub fn empty_reducer_no_args(_ctx: &ReducerContext) {
   }
   
   #[spacetimedb::reducer]
   pub fn empty_reducer_with_int(_ctx: &ReducerContext, _count: i32) {
   }
   
   #[spacetimedb::reducer]
   pub fn empty_reducer_with_string(_ctx: &ReducerContext, _name: String) {
   }
   
   #[spacetimedb::reducer]
   pub fn empty_reducer_with_two_args(_ctx: &ReducerContext, _count: i32, _name: String) {
   }
   
   #[spacetimedb::reducer]
   pub fn empty_reducer_with_three_args(_ctx: &ReducerContext, _active: bool, _ratio: f32, _label: String) {
   }

2. The golden example:

   use spacetimedb::{reducer, ReducerContext};
   
   #[reducer]
   pub fn empty_reducer_no_args(ctx: &ReducerContext) -> Result<(), String> {
       Ok(())
   }
   
   #[reducer]
   pub fn empty_reducer_with_int(ctx: &ReducerContext, count: i32) -> Result<(), String> {
       Ok(())
   }
   
   #[reducer]
   pub fn empty_reducer_with_string(ctx: &ReducerContext, name: String) -> Result<(), String> {
       Ok(())
   }
   
   #[reducer]
   pub fn empty_reducer_with_two_args(ctx: &ReducerContext, count: i32, name: String) -> Result<(), String> {
       Ok(())
   }
   
   #[reducer]
   pub fn empty_reducer_with_three_args(
       ctx: &ReducerContext,
       active: bool,
       ratio: f32,
       label: String,
   ) -> Result<(), String> {
       Ok(())
   }

3. The error: schema_parity: describe failed

4. Explain the difference:

   • Reducer functions need to return Result<(), String> instead of nothing, which allows proper error handling.

5. Root cause: Lack of clear documentation regarding the expected return types for reducers in the recent versions.

6. Recommendation: Update documentation to emphasize the required return types for reducer functions.

---

This is a partial analysis, but the template can be extended to cover all similar failures and weaknesses in the SpacetimeDB documentation. Each section should connect failures to actionable insights for improving documentation and code clarity.