Templates naming standarization

[drogus]

Description of Changes

This PR renames the templates to always use shorthand for the language, specify a framework (or console) if necessary, and shorten the naming in general

Expected complexity level and risk

1

Testing

I've tested generating templates manually

[drogus]

/update-llm-benchmark

[clockwork-labs-bot]

LLM Benchmark Results (ci-quickfix)

Language	Mode	Category	Tests Passed	Pass %	Task Pass %
Rust	rustdoc_json	basics	0/12	0.0%	0.0%
Rust	rustdoc_json	schema	0/10	0.0%	0.0%
Rust	rustdoc_json	total	0/22	0.0%	0.0%
C#	docs	basics	0/12	0.0%	0.0%
C#	docs	schema	0/10	0.0%	0.0%
C#	docs	total	0/22	0.0%	0.0%

_{Generated at: 2026-01-15T16:50:45.748Z}

[drogus]

/update-llm-benchmark

[drogus]

/update-llm-benchmark

[clockwork-labs-bot]

LLM Benchmark Results (ci-quickfix)

Language	Mode	Category	Tests Passed	Task Pass %
Rust	rustdoc_json	basics	20/27	68.8% ⬇️ -14.6%
Rust	rustdoc_json	schema	29/34	80.0%
Rust	rustdoc_json	total	49/61	73.9% ⬇️ -8.0%
C#	docs	basics	12/27	61.1%
C#	docs	schema	10/34	31.5%
C#	docs	total	22/61	47.7%

Compared against master branch baseline

_{Generated at: 2026-01-20T13:07:38.079Z}

Failure Analysis (click to expand)

Benchmark Failure Analysis

Generated from: /__w/SpacetimeDB/SpacetimeDB/tools/xtask-llm-benchmark/../../docs/llms/docs-benchmark-details.json

Summary

• Total failures analyzed: 25

Analysis

Analysis of SpacetimeDB Benchmark Test Failures

Rust Failures

1. Root Causes

• Compile/Publish Errors: Issues generally stem from incorrect or incomplete structures defined in the code which lead to problems during compilation and publishing.

  • Documentation Issues:
    • Lack of clear guidelines on defining complex types (e.g., using SpacetimeType).
    • Missing explanations on required table and reducer annotations, especially for those that dictate publish behaviors.

• Other Failures: Failures were primarily due to discrepancies in reducer expectations and missing or improperly defined table structures, which resulted in various SQL runtime errors.

  • Documentation Issues:
    • Insufficient clarity on schema changes and correct reducer use.
    • Ambiguity around the use of pub visibility and needs for struct fields in the context of SpacetimeDB.

2. Recommendations

Documentation Files/Sections to Update

• Table Definition Guide: Update to include detailed explanations and examples of table definitions, emphasizing the use of pub for fields and structures.

  • Exact Changes:
    • Clearly state all required annotations (like #[table(name = ...)], #[primary_key]) and provide exhaustive examples for complex nested types.

• Reducer Expectations Section: Enhance sections covering reducers; discuss expected signatures, common pitfalls (e.g. mismatched expected reducers), and details on initialization.

  • Exact Changes:
    • Include tables that link reducer names and expected behaviors directly correlating with example codes to form a clear, actionable reference.

3. Priority

• High Impact: Enhance the Table Definition Guide and Reducer Expectations Section, as these areas directly affect the most frequent issues related to compiler errors and SQL failures.

---

C# Failures

1. Root Causes

• Issues with Table and Index Definitions: Failure to correctly define tables and their properties resulted in SQL errors due to missing entities.

  • Documentation Issues:
    • Incomplete explanations about marking public or requiring specific attributes for table definitions can lead to failure in recognizing tables.

• Reducer Implementation Errors: Failures often came from using outdated or incorrect approaches to structuring reducer methods, leading to incorrect database operations.

  • Documentation Issues:
    • Lack of examples showcasing how reducers should be structured, especially with regard to the use of parameters and return types.

2. Recommendations

Documentation Files/Sections to Update

• C# Table and Struct Documentation: Revise documentation covering attributes and definitions for structs and tables.

  • Exact Changes:
    • Include a section dedicated to outlining which attributes should be used for which programming contexts (e.g., Public vs. non-Public) and examples that correctly use them.

• C# Reducer Documentation: Enhance sections discussing reducer methods—emphasizing the correct signature and return types based on different operational contexts.

  • Exact Changes:
    • Provide example codes demonstrating several reducer use cases, especially showing initializations and updates correctly.

3. Priority

• High Impact: Focus on updating the C# Table and Struct Documentation and C# Reducer Documentation, as these areas directly influence many of the error sources impacting SQL executions, which are fundamental to SpacetimeDB operations.

This orchestrated approach to refining documentation should significantly reduce the number of benchmark test failures in both Rust and C#, improving overall developer experience and efficiency in using SpacetimeDB.

[drogus]

/update-llm-benchmark

[clockwork-labs-bot]

LLM Benchmark Results (ci-quickfix)

Language	Mode	Category	Tests Passed	Task Pass %
Rust	rustdoc_json	basics	23/27	82.6% ⬆️ +38.2%
Rust	rustdoc_json	schema	25/34	72.0% ⬆️ +45.5%
Rust	rustdoc_json	total	48/61	77.8% ⬆️ +41.5%
Rust	docs	basics	5/27	11.1%
Rust	docs	schema	7/34	15.5% ⬇️ -5.0%
Rust	docs	total	12/61	13.1% ⬇️ -2.3%
C#	docs	basics	27/27	100.0% ⬆️ +38.9%
C#	docs	schema	20/30	58.7% ⬆️ +27.2%
C#	docs	total	47/57	81.2% ⬆️ +33.6%

Compared against master branch baseline

_{Generated at: 2026-01-21T11:42:04.144Z}

Failure Analysis (click to expand)

Benchmark Failure Analysis

Generated from: /__w/SpacetimeDB/SpacetimeDB/tools/xtask-llm-benchmark/../../docs/llms/docs-benchmark-details.json

Summary

• Total failures analyzed: 34

---

Analysis of SpacetimeDB Benchmark Failures

Each section below analyzes specific failures grouped by language and mode, providing code examples, expected results, explanations of differences, root causes, and actionable recommendations.

---

Rust / rustdoc_json Failures

Compile/Publish Errors (3 failures)

t_002_scheduled_table

1. The generated code:

   // LLM Output
   #[table(name = tick_timer, schedule(reducer = tick, column = scheduled_at))]
   pub struct TickTimer {
       #[primary_key]
       #[auto_inc]
       scheduled_id: u64,
       scheduled_at: ScheduleAt,
   }

2. The golden example:

   // Expected (golden)
   #[table(name = tick_timer, scheduled(tick))]
   pub struct TickTimer {
       #[primary_key]
       #[auto_inc]
       pub scheduled_id: u64,
       pub scheduled_at: ScheduleAt,
   }

3. The error: publish_error: spacetime publish failed (exit=1)

4. Explain the difference:

   • The schedule attribute syntax is incorrect. It should use scheduled(tick) instead of schedule(reducer = tick, column = scheduled_at).

5. Root cause: The documentation likely lacks clarity on how to properly apply the scheduled syntax in the attribute.

6. Recommendation: Update the documentation to clearly show the correct way to declare scheduled tables. For example:

   #[table(name = tick_timer, scheduled(tick))]

t_016_sum_type_columns

1. The generated code:

   // LLM Output
   #[derive(SpacetimeType, Clone)]
   pub struct Rect {
       width: i32,
       height: i32,
   }

2. The golden example:

   // Expected (golden)
   #[derive(SpacetimeType, Clone, Debug)]
   pub struct Rect {
       pub width: i32,
       pub height: i32,
   }

3. The error: publish_error: spacetime publish failed (exit=1)

4. Explain the difference:

   • Missing visibility specifiers such as pub led to potential access violations when the struct is instantiated.

5. Root cause: Documentation does not emphasize the importance of specifying visibility for fields.

6. Recommendation: Revise documentation to illustrate the necessity of using pub for fields in structs intended for use outside their module.

t_017_scheduled_columns

1. The generated code:

   #[table(name = tick_timer, schedule(reducer = tick, column = scheduled_at))]
   pub struct TickTimer {
       #[primary_key]
       #[auto_inc]
       scheduled_id: u64,
       scheduled_at: ScheduleAt,
   }

2. The golden example:

   #[table(name = tick_timer, scheduled(tick))]
   pub struct TickTimer {
       #[primary_key]
       #[auto_inc]
       pub scheduled_id: u64,
       pub scheduled_at: ScheduleAt,
   }

3. The error: publish_error: spacetime publish failed (exit=1)

4. Explain the difference:

   • Similar to the previous failure, the syntax for scheduling is incorrect. The expected format uses scheduled(tick) instead.

5. Root cause: Lack of clarity on syntax requirements for scheduling in the documentation.

6. Recommendation: Enhance documentation with examples that show correct syntax for all table attributes, particularly for scheduling.

---

Other Failures (4 failures)

t_004_insert

1. The generated code:

   #[table(name = user)]
   pub struct User {
       #[primary_key]
       id: i32,
       name: String,
       age: i32,
       active: bool,
   }

2. The golden example:

   #[table(name = user)]
   pub struct User {
       #[primary_key]
       pub id: i32,
       pub name: String,
       pub age: i32,
       pub active: bool,
   }

3. The error: no such table: user

4. Explain the difference:

   • Missing visibility specifiers (pub) for the fields would prevent correct access, leading to runtime errors during insert operations.

5. Root cause: Similar to the previous cases, visibility is crucial but not adequately addressed in documentation.

6. Recommendation: Stress the importance of visibility in struct field definitions within the documentation.

t_007_crud

1. The generated code:

   user.insert(User { id: 1, name: "Alice".to_string(), age: 30, active: true });

2. The golden example:

   ctx.db.user().insert(User { id: 1, name: "Alice".into(), age: 30, active: true });

3. The error: no such table: user

4. Explain the difference:

   • The access method for the user table needs to use the context object for database operations.

5. Root cause: Documentation should clarify how to access the database context for CRUD operations.

6. Recommendation: Include a specific section with examples for performing CRUD operations, emphasizing the use of context for accessing tables.

---

C# / docs Failures (5 total)

Other Failures (5 failures)

t_014_elementary_columns

1. The generated code:

   [SpacetimeDB.Table(Name = "Primitive", Public = true)]
   public partial struct Primitive
   {
       [SpacetimeDB.PrimaryKey]
       public int Id;
       public int Count;
       public long Total;
       public float Price;
       public double Ratio;
       public bool Active;
       public string Name;
   }

2. The golden example:

   [Table(Name = "Primitive")]
   public partial struct Primitive
   {
       [PrimaryKey] public int Id;
       public int Count;
       public long Total;
       public float Price;
       public double Ratio;
       public bool Active;
       public string Name;
   }

3. The error: no such table: primitive

4. Explain the difference:

   • Misuse of Public in the attribute declaration, which isn't required in the expected syntax.

5. Root cause: Lack of guidelines in the documentation regarding when to use public access modifiers in table struct definitions.

6. Recommendation: Remove unnecessary access modifiers from examples to avoid confusion; revise documentation language to clarify use cases.

---

This structured analysis highlights specific actionable insights that can be employed to clear up failure sources and enhance the documentation for better developer comprehensibility.

[drogus]

/update-llm-benchmark

[clockwork-labs-bot]

LLM Benchmark Results (ci-quickfix)

Language	Mode	Category	Tests Passed	Task Pass %
Rust	rustdoc_json	basics	21/27	75.0% ⬆️ +30.6%
Rust	rustdoc_json	schema	21/34	58.7% ⬆️ +32.2%
Rust	rustdoc_json	total	42/61	67.6% ⬆️ +31.3%
Rust	docs	basics	5/27	11.1%
Rust	docs	schema	4/30	12.5% ⬇️ -8.0%
Rust	docs	total	9/57	11.7% ⬇️ -3.6%
C#	docs	basics	27/27	100.0% ⬆️ +38.9%
C#	docs	schema	24/30	71.7% ⬆️ +40.2%
C#	docs	total	51/57	87.1% ⬆️ +39.5%

Compared against master branch baseline

_{Generated at: 2026-01-21T12:43:31.105Z}

Failure Analysis (click to expand)

Benchmark Failure Analysis

Generated from: /__w/SpacetimeDB/SpacetimeDB/tools/xtask-llm-benchmark/../../docs/llms/docs-benchmark-details.json

Summary

• Total failures analyzed: 35

---

Analysis of SpacetimeDB Benchmark Test Failures

Rust / rustdoc_json Failures

Compile/Publish Errors (4 failures)

Failure: t_002_scheduled_table

1. Generated Code:

   use spacetimedb::{table, reducer, ReducerContext, Table, UniqueColumn, ScheduleAt};
   
   #[table(name = tick_timer, scheduled(reducer = tick, column = scheduled_at))]
   pub struct TickTimer {
       #[primary_key]
       #[auto_inc]
       scheduled_id: u64,
       scheduled_at: ScheduleAt,
   }
   
   #[reducer(init)]
   pub fn init(ctx: &ReducerContext) {
       if ctx.db.tick_timer().count() == 0 {
           ctx.db.tick_timer().insert(TickTimer {
               scheduled_id: 0,
               scheduled_at: ScheduleAt::RepeatMicros(50_000),
           });
       }
   }
   
   #[reducer]
   pub fn tick(ctx: &ReducerContext, scheduled_id: u64) {
       if let Some(mut row) = ctx.db.tick_timer().scheduled_id().find(scheduled_id) {
           row.scheduled_at = ScheduleAt::RepeatMicros(50_000);
           ctx.db.tick_timer().scheduled_id().update(row);
       }
   }

2. Golden Example:

   use spacetimedb::{reducer, table, ReducerContext, ScheduleAt, Table};
   use std::time::Duration;
   
   #[table(name = tick_timer, scheduled(tick))]
   pub struct TickTimer {
       #[primary_key]
       #[auto_inc]
       scheduled_id: u64,
       scheduled_at: ScheduleAt,
   }
   
   #[reducer]
   pub fn tick(_ctx: &ReducerContext, _row: TickTimer) -> Result<(), String> {
       Ok(())
   }
   
   #[reducer(init)]
   pub fn init(ctx: &ReducerContext) -> Result<(), String> {
       ctx.db.tick_timer().insert(TickTimer {
           scheduled_id: 0,
           scheduled_at: ScheduleAt::Interval(Duration::from_millis(50).into()),
       });
       Ok(())
   }

3. Error: publish_error: spacetime publish failed (exit=1)

4. Explain the Difference:

   • The LLM output incorrectly includes scheduled(reducer = tick, column = scheduled_at) which is not valid and leads to failures.
   • The expected output correctly uses simply scheduled(tick).

5. Root Cause: The documentation may not clarify the correct usage of the scheduled attribute and additional parameters which can lead to syntax issues.

6. Recommendation: Update the documentation to clarify the scheduled attribute syntax and provide examples of valid usages without extra parameters.

---

Failure: t_007_crud

1. Generated Code:

   use spacetimedb::{reducer, table, ReducerContext};
   
   #[table(name = user)]
   pub struct User {
       #[primary_key]
       id: i32,
       name: String,
       age: i32,
       active: bool,
   }
   
   #[reducer]
   pub fn crud(ctx: &ReducerContext) {
       let users = ctx.db.user();
   
       users.insert(User {
           id: 1,
           name: "Alice".to_string(),
           age: 30,
           active: true,
       });
       users.insert(User {
           id: 2,
           name: "Bob".to_string(),
           age: 22,
           active: false,
       });
   
       if let Some(mut u) = users.id().find(1i32) {
           u.name = "Alice2".to_string();
           u.age = 31;
           u.active = false;
           let _ = users.id().update(u);
       }
   
       users.id().delete(&2i32);
   }

2. Golden Example:

   use spacetimedb::{reducer, table, ReducerContext, Table};
   
   #[table(name = user)]
   pub struct User {
       #[primary_key]
       pub id: i32,
       pub name: String,
       pub age: i32,
       pub active: bool,
   }
   
   #[reducer]
   pub fn crud(ctx: &ReducerContext) {
       ctx.db.user().insert(User { id: 1, name: "Alice".into(), age: 30, active: true });
       ctx.db.user().insert(User { id: 2, name: "Bob".into(),   age: 22, active: false });
       ctx.db.user().id().update(User { id: 1, name: "Alice2".into(), age: 31, active: false });
       ctx.db.user().id().delete(2);
   }

3. Error: publish_error: spacetime publish failed (exit=1)

4. Explain the Difference:

   • The LLM did not use pub access modifier for struct fields which caused data access errors.
   • The update command for id was incorrectly constructed, leading to data updates for non-existent fields.

5. Root Cause: A lack of emphasis in the documentation on using pub for struct fields and the correct syntax for data updating operations.

6. Recommendation: Update the documentation to emphasize the necessity of pub before fields when designing a table struct and provide clearer examples of inserting and updating records.

---

Other Failures (5 failures)

Failure: t_003_struct_in_table

1. Generated Code:

   use spacetimedb::{table, reducer, ReducerContext, Table, SpacetimeType};
   
   #[derive(SpacetimeType)]
   pub struct Position {
       x: i32,
       y: i32,
   }
   
   #[table(name = entity)]
   pub struct Entity {
       #[primary_key]
       id: i32,
       pos: Position,
   }
   
   #[reducer]
   pub fn add_entity(ctx: &ReducerContext, id: i32, x: i32, y: i32) {
       ctx.db.entity().insert(Entity { id, pos: Position { x, y } });
   }

2. Golden Example:

   use spacetimedb::{table, SpacetimeType};
   
   #[derive(SpacetimeType, Clone, Debug)]
   pub struct Position {
       pub x: i32,
       pub y: i32,
   }
   
   #[table(name = entity)]
   pub struct Entity {
       #[primary_key]
       pub id: i32,
       pub pos: Position,
   }

3. Error: schema_parity: reducers differ - expected [], got ["add_entity()"]

4. Explain the Difference:

   • LLM fails to include the pub keyword for struct attributes causing schema discrepancies.
   • The reducer function signature does not match the expected pattern for initialization.

5. Root Cause: The documentation does not sufficiently cover requirements for struct member visibility and reducer requirements.

6. Recommendation: Augment documentation to include visibility rules for fields in structs and clarify what constitutes a valid reducer function signature.

---

Rust / docs Failures (22 failures)

Compile/Publish Errors (1 failure)

Failure: t_019_many_to_many

1. Generated Code:

   use spacetimedb::{ReducerContext, Table, TableWithPrimaryKey};
   
   #[spacetimedb::table(name = user)]
   pub struct User {
       #[primary_key]
       user_id: i32,
       name: String,
   }
   
   #[spacetimedb::table(name = group)]
   pub struct Group {
       #[primary_key]
       group_id: i32,
       title: String,
   }
   
   #[spacetimedb::table(
       name = membership,
       index(name = by_user, btree(columns = [user_id])),
       index(name = by_group, btree(columns = [group_id]))
   )]
   pub struct Membership {
       #[primary_key]
       id: i32,
       user_id: i32,
       group_id: i32,
   }
   
   #[spacetimedb::reducer]
   pub fn seed(ctx: &ReducerContext) {
       // Clear existing rows to ensure we end up with exactly the specified rows.
       for m in ctx.db.membership().iter() {
           ctx.db.membership().id().delete(m.id);
       }
       for u in ctx.db.user().iter() {
           ctx.db.user().user_id().delete(u.user_id);
       }
       for g in ctx.db.group().iter() {
           ctx.db.group().group_id().delete(g.group_id);
       }
   
       // Insert users
       ctx.db.user().insert(User { user_id: 1, name: "Alice".to_string() });
       ctx.db.user().insert(User { user_id: 2, name: "Bob".to_string() });
   
       // Insert groups
       ctx.db.group().insert(Group { group_id: 10, title: "Admin".to_string() });
       ctx.db.group().insert(Group { group_id: 20, title: "Dev".to_string() }); 
   }

2. Golden Example:

   use spacetimedb::{reducer, table, ReducerContext, Table};
   
   #[table(name = user)]
   pub struct User {
       #[primary_key]
       pub user_id: i32,
       pub name: String,
   }
   
   #[table(name = group)]
   pub struct Group {
       #[primary_key]
       pub group_id: i32,
       pub title: String,
   }
   
   #[table(
       name = membership,
       index(name = by_user,  btree(columns = [user_id])),
       index(name = by_group, btree(columns = [group_id]))
   )]
   pub struct Membership {
       #[primary_key]
       pub id: i32,
       pub user_id: i32,
       pub group_id: i32,
   }
   
   #[reducer]
   pub fn seed(ctx: &ReducerContext) {
       ctx.db.user().insert(User  { user_id: 1, name: "Alice".into() });
       ctx.db.user().insert(User  { user_id: 2, name: "Bob".into()   });
       ctx.db.group().insert(Group { group_id: 10, title: "Admin".into() });
       ctx.db.group().insert(Group { group_id: 20, title: "Dev".into()   });
       ctx.db.membership().insert(Membership { id: 1, user_id: 1, group_id: 10 });
   }

3. Error: publish_error: spacetime publish failed (exit=1)

4. Explain the Difference:

   • The LLM output does not use pub for struct fields, creating access issues.
   • Details for inserting data into group and membership are improperly handled.

5. Root Cause: The documentation might lack clarity on access modifiers for struct members and their implications.

6. Recommendation: Update examples and clarify the importance of using pub for struct members in documentation.

---

Timeout Issues (1 failures)

• Task: t_020_ecs

Root Cause Analysis:
The timeout may originate from unoptimized queries or excessive data processing inside the reducer functions.

Recommendation:
Monitor the efficiency of reducer logic, optimize data access patterns, and provide timeout guidelines in the documentation.

---

C# / docs Failures (4 total)

Timeout Issues (1 failures)

• Task: t_020_ecs

Root Cause Analysis: Similar to Rust, unlikely efficient operations or heavy processing in the reducer function.

Recommendation: Add performance optimization guidance regarding reducer implementations in the documentation.

---

Other Failures (3 failures)

Failure: t_014_elementary_columns

1. Generated Code:

   using SpacetimeDB;
   
   public static partial class Module
   {
       [SpacetimeDB.Table(Name = "Primitive", Public = true)]
       public partial struct Primitive
       {
           [SpacetimeDB.PrimaryKey]
           public int Id;
           public int Count;
           public long Total;
           public float Price;
           public double Ratio;
           public bool Active;
           public string Name;
       }
   
       [SpacetimeDB.Reducer]
       public static void Seed(ReducerContext ctx)
       {
           ctx.Db.Primitive.Insert(new Primitive
           {
               Id = 1,
               Count = 2,
               Total = 3000000000L,
               Price = 1.5f,
               Ratio = 2.25,
               Active = true,
               Name = "Alice",
           });
       }
   }

2. Golden Example:

   using SpacetimeDB;
   
   public static partial class Module
   {
       [Table(Name = "Primitive")]
       public partial struct Primitive
       {
           [PrimaryKey] public int Id;
           public int Count;
           public long Total;
           public float Price;
           public double Ratio;
           public bool Active;
           public string Name;
       }
   
       [Reducer]
       public static void Seed(ReducerContext ctx)
       {
           ctx.Db.Primitive.Insert(new Primitive {
               Id = 1,
               Count = 2,
               Total = 3000000000,
               Price = 1.5f,
               Ratio = 2.25,
               Active = true,
               Name = "Alice"
           });
       }
   }

3. Error: Error: no such table: primitive.

4. Explain the Difference:

   • The Public = true attribute in the LLM response is incorrect and could lead to compilation failures.
   • Various member variables lack pub access modifier, leading to access violations.

5. Root Cause: Documentation may not sufficiently cover the requirements for table visibility.

6. Recommendation: Update documentation to clarify correct attributes and access modifiers for struct members.

---

Summary of Documentation Recommendations

1. Field Visibility: Emphasize the requirement for pub on struct fields within the documentation.
2. Reducer Patterns: Clarify proper syntax and structure of reducers in documentation.
3. Scheduled and Index Attributes: Provide examples of correct use for scheduling and indices.
4. Error Handling: Recommend analysis of API return values for more effective debugging and unexpected errors.
5. Performance Optimization: Include best practices for performance tuning, especially for reducers that handle large datasets.

This structured approach will help prevent similar errors in the future and improve the clarity of the documentation overall.