Skip to content

Conversation

@coolreader18
Copy link
Collaborator

@coolreader18 coolreader18 commented Jan 20, 2026

Description of Changes

This release has a couple of patches I've been waiting on for a bit:

Expected complexity level and risk

2: v8 is a very big and important dependency, but also very stable.

Testing

  • No change to behavior; automated testing is sufficient.

@coolreader18 coolreader18 requested a review from Centril January 20, 2026 21:47
Copy link
Contributor

@Centril Centril left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved modulo the 2 requested changes below.

coolreader18 and others added 3 commits January 21, 2026 09:39
Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
Signed-off-by: Noa <coolreader18@gmail.com>
@coolreader18
Copy link
Collaborator Author

/update-llm-benchmark

@clockwork-labs-bot
Copy link
Collaborator

LLM Benchmark Results (ci-quickfix)

Language Mode Category Tests Passed Task Pass %
Rust rustdoc_json basics 22/27 74.3% ⬆️ +29.9%
Rust rustdoc_json schema 26/34 75.3% ⬆️ +48.8%
Rust rustdoc_json total 48/61 74.8% ⬆️ +38.5%
Rust docs basics 5/27 11.1%
Rust docs schema 8/34 20.5%
Rust docs total 13/61 15.4%
C# docs basics 24/27 91.7% ⬆️ +30.6%
C# docs schema 25/34 73.7% ⬆️ +42.2%
C# docs total 49/61 83.5% ⬆️ +35.8%

Compared against master branch baseline

Generated at: 2026-01-21T19:26:17.891Z

Failure Analysis (click to expand)

Benchmark Failure Analysis

Generated from: /__w/SpacetimeDB/SpacetimeDB/tools/xtask-llm-benchmark/../../docs/llms/docs-benchmark-details.json

Summary

  • Total failures analyzed: 35

Analysis of SpacetimeDB Benchmark Failures

In this analysis, I've categorized the SpacetimeDB benchmark test failures by language and mode, detailing the differences between the generated code and expected outputs, and providing actionable insights for documentation changes.

Rust / rustdoc_json Failures

Common Errors: Schema Parity and Publishing Issues

Failures: t_002_scheduled_table, t_017_scheduled_columns

  1. The generated code:

    use spacetimedb::{table, reducer, ReducerContext, Table, ScheduleAt};
    
    #[table(name = tick_timer, scheduled(reducer = tick, column = scheduled_at))]
    pub struct TickTimer {
        #[primary_key]
        #[auto_inc]
        scheduled_id: u64,
        scheduled_at: ScheduleAt,
    }
    
    //...
    #[reducer(init)]
    pub fn init(ctx: &ReducerContext) {
        if ctx.db.tick_timer().count() == 0 {
            ctx.db.tick_timer().insert(TickTimer {
                scheduled_id: 0,
                scheduled_at: ScheduleAt::RepeatMicros(50_000),
            });
        }
    }
  2. The golden example:

    use spacetimedb::{reducer, table, ReducerContext, ScheduleAt, Table};
    use std::time::Duration;
    
    #[table(name = tick_timer, scheduled(tick))]
    pub struct TickTimer {
        #[primary_key]
        #[auto_inc]
        pub scheduled_id: u64,
        pub scheduled_at: ScheduleAt,
    }
    
    #[reducer(init)]
    pub fn init(ctx: &ReducerContext) {
        let every_50ms: ScheduleAt = Duration::from_millis(50).into();
        ctx.db.tick_timer().insert(TickTimer {
            scheduled_id: 0,
            scheduled_at: every_50ms,
        });
    }
  3. The error:

    • Publishing failed due to schema disparity and incorrectly configured attributes.
  4. Differences:

    • The scheduled_at attribute in the generated code incorrectly uses RepeatMicros, while the expected uses Interval.
    • The scheduled annotation is incorrect in the generated code.
  5. Root cause:

    • Documentation does not clearly explain the difference between scheduled_at syntax in various contexts and how to correctly initialize ScheduleAt.
  6. Recommendation:

    • Update documentation to clarify the usage of ScheduleAt and the implementation of the scheduled attribute by providing clear examples. Mention the importance of using Interval correctly.

Specific Reducer and Table Definitions

Failures: t_003_struct_in_table

  1. The generated code:

    use spacetimedb::{table, reducer, ReducerContext, Table, SpacetimeType};
    
    #[derive(SpacetimeType)]
    pub struct Position {
        x: i32,
        y: i32,
    }
    
    #[table(name = entity)]
    pub struct Entity {
        #[primary_key]
        id: i32,
        pos: Position,
    }
    
    #[reducer(init)]
    pub fn init(_ctx: &ReducerContext) {}
  2. The golden example:

    use spacetimedb::{table, SpacetimeType};
    
    #[derive(SpacetimeType, Clone, Debug)]
    pub struct Position {
        pub x: i32,
        pub y: i32,
    }
    
    #[table(name = entity)]
    pub struct Entity {
        #[primary_key]
        pub id: i32,
        pub pos: Position,
    }
  3. The error:

    • Reducers differ in expected output and generated code.
  4. Differences:

    • Missing pub accessors on the struct fields, which are necessary for proper serialization and schema definition in SpacetimeDB.
  5. Root cause:

    • The documentation may not explicitly state the importance of public fields for generated types that are part of the schema.
  6. Recommendation:

    • Update documentation to clarify that all struct fields used in SpacetimeDB schemas should be declared as pub and provide an example demonstrating this requirement.

CRUD and User Insertion Errors

Failures: t_004_insert, t_006_delete, t_007_crud

  1. The generated code:

    use spacetimedb::{table, reducer, ReducerContext, Table};
    
    #[table(name = user)]
    pub struct User {
        #[primary_key]
        id: i32,
        name: String,
        age: i32,
        active: bool,
    }
    
    #[reducer]
    pub fn insert_user(ctx: &ReducerContext, id: i32, name: String, age: i32, active: bool) {
        ctx.db.user().insert(User { id, name, age, active });
    }
  2. The golden example:

    use spacetimedb::{reducer, table, ReducerContext, Table};
    
    #[table(name = user)]
    pub struct User {
        #[primary_key]
        pub id: i32,
        pub name: String,
        pub age: i32,
        pub active: bool,
    }
    
    #[reducer]
    pub fn insert_user(ctx: &ReducerContext, id: i32, name: String, age: i32, active: bool) -> Result<(), String> {
        ctx.db.user().insert(User { id, name, age, active });
        Ok(())
    }
  3. The error:

    • Schema errors and missing return types for CRUD operations.
  4. Differences:

    • The lack of pub keywords in the struct fields prevents external access.
    • The missing return type leading to general errors during operation.
  5. Root cause:

    • Documentation lacks emphasis on return types for reducer functions and public attributes for structs.
  6. Recommendation:

    • Reinforce the necessity for pub keywords in struct definitions and encourage clear function signatures by providing comprehensive guidelines on how to define CRUD operations.

C# / docs Failures

Index Lookup Issues

Failures: t_008_index_lookup

  1. The generated code:

    using SpacetimeDB;
    
    public static partial class Module
    {
        [SpacetimeDB.Table(Name = "User", Public = true)]
        public partial struct User
        {
            [SpacetimeDB.PrimaryKey]
            public int Id;
            public string Name;
            public int Age;
            public bool Active;
        }
    
        [SpacetimeDB.Reducer]
        public static void LookupUserName(ReducerContext ctx, int id)
        {
            var user = ctx.Db.User.Id.Find(id);
            if (user is null)
            {
                return;
            }
            //...
        }
    }
  2. The golden example:

    using SpacetimeDB;
    
    public static partial class Module
    {
        [Table(Name = "User")]
        public partial struct User
        {
            [PrimaryKey] public int Id;
            public string Name;
            public int Age;
            public bool Active;
        }
    
        [Reducer]
        public static void LookupUserName(ReducerContext ctx, int id)
        {
            var u = ctx.Db.User.Id.Find(id);
            if (u.HasValue)
            {
                //...
            }
        }
    }
  3. The error:

    • Build errors due to problematic struct definitions and missing access modifiers.
  4. Differences:

    • Unnecessary Public = true in the attribute causing complexity.
    • Incorrect handling of nullable types during user lookups.
  5. Root cause:

    • Lack of clarity in documentation regarding nullable types and access modifiers.
  6. Recommendation:

    • Simplify C# struct declarations in the documentation and clarify correct nullable handling and access levels for properties.

This analysis provides insights into the recurring issues across the benchmark failures and identifies actionable steps for documentation improvements to prevent similar failures in the future.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants