Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,14 +61,14 @@ Source generation shifts work from run time to build time: you pay a little up f
<!-- benchmarks:start -->
| Scenario | TUnit (AOT) | TUnit | xUnit v3 | NUnit | MSTest |
|----------|---|---|---|---|---|
| Data-driven tests | 16.65 ms | 265.70 ms | 523.90 ms | 523.01 ms | 529.13 ms |
| Async-heavy tests | 113.1 ms | 360.0 ms | 605.6 ms | 608.6 ms | 676.0 ms |
| Matrix combinations | 119.6 ms | 474.1 ms | 1,469.2 ms | 1,468.9 ms | 1,515.3 ms |
| Large suites (scale) | 29.38 ms | 263.55 ms | 504.14 ms | 488.30 ms | 495.94 ms |
| Massive parallelism | 217.6 ms | 467.8 ms | 2,945.7 ms | 1,107.5 ms | 2,993.9 ms |
| Setup/teardown lifecycle | — | 335.8 ms | 1,050.2 ms | 1,008.8 ms | 1,099.9 ms |

<sub>Mean wall-clock time to run the same test suite. TUnit (AOT) 1.58.0 · TUnit 1.58.0 · xUnit v3 3.2.2 · NUnit 4.6.1 · MSTest 4.3.0. .NET SDK 10.0.301, .NET 10.0.9 (10.0.9, 10.0.926.27113), X64 RyuJIT x86-64-v4. Updated 2026-07-12 — regenerated weekly by the [Speed Comparison workflow](https://github.com/thomhurst/TUnit/actions/workflows/speed-comparison.yml). Full results and methodology: [tunit.dev/docs/benchmarks](https://tunit.dev/docs/benchmarks/).</sub>
| Data-driven tests | 18.46 ms | 333.55 ms | 612.50 ms | 588.63 ms | 643.67 ms |
| Async-heavy tests | 117.0 ms | 375.9 ms | 607.7 ms | 599.8 ms | 659.9 ms |
| Matrix combinations | 118.1 ms | 388.2 ms | 1,491.5 ms | 1,452.1 ms | 1,541.1 ms |
| Large suites (scale) | 18.25 ms | 320.96 ms | 520.49 ms | 520.19 ms | 566.98 ms |
| Massive parallelism | 217.5 ms | 530.9 ms | 2,975.1 ms | 1,122.5 ms | 3,029.7 ms |
| Setup/teardown lifecycle | 73.35 ms | 458.30 ms | 1,186.16 ms | 1,147.56 ms | 1,279.43 ms |

<sub>Mean wall-clock time to run the same test suite. TUnit (AOT) 1.60.0 · TUnit 1.60.0 · xUnit v3 3.2.2 · NUnit 4.6.1 · MSTest 4.3.2. .NET SDK 10.0.302, .NET 10.0.10 (10.0.10, 10.0.1026.32716), X64 RyuJIT x86-64-v4. Updated 2026-07-16 — regenerated weekly by the [Speed Comparison workflow](https://github.com/thomhurst/TUnit/actions/workflows/speed-comparison.yml). Full results and methodology: [tunit.dev/docs/benchmarks](https://tunit.dev/docs/benchmarks/).</sub>
<!-- benchmarks:end -->

## Getting Started
Expand Down
20 changes: 10 additions & 10 deletions docs/docs/benchmarks/engine/AsyncTests.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,20 +9,20 @@ sidebar_position: 3
> Realistic async/await patterns with I/O simulation

:::info Last Updated
This benchmark was automatically generated on **2026-07-12** from the latest CI run.
This benchmark was automatically generated on **2026-07-16** from the latest CI run.

**Environment:** Ubuntu Latest • .NET SDK 10.0.301
**Environment:** Ubuntu Latest • .NET SDK 10.0.302
:::

## 📊 Results

| Framework | Version | Mean | Median | StdDev |
|-----------|---------|------|--------|--------|
| **TUnit** | 1.58.0 | 360.0 ms | 353.0 ms | 18.09 ms |
| NUnit | 4.6.1 | 608.6 ms | 599.2 ms | 40.35 ms |
| MSTest | 4.3.0 | 676.0 ms | 672.3 ms | 18.78 ms |
| xUnit3 | 3.2.2 | 605.6 ms | 601.9 ms | 11.72 ms |
| **TUnit (AOT)** | 1.58.0 | 113.1 ms | 113.0 ms | 0.68 ms |
| **TUnit** | 1.60.0 | 375.9 ms | 373.8 ms | 11.50 ms |
| NUnit | 4.6.1 | 599.8 ms | 597.0 ms | 18.50 ms |
| MSTest | 4.3.2 | 659.9 ms | 662.1 ms | 7.23 ms |
| xUnit3 | 3.2.2 | 607.7 ms | 607.6 ms | 8.83 ms |
| **TUnit (AOT)** | 1.60.0 | 117.0 ms | 115.7 ms | 2.60 ms |

## 📈 Visual Comparison

Expand Down Expand Up @@ -60,8 +60,8 @@ This benchmark was automatically generated on **2026-07-12** from the latest CI
xychart-beta
title "AsyncTests Performance Comparison"
x-axis ["TUnit", "NUnit", "MSTest", "xUnit3", "TUnit_AOT"]
y-axis "Time (ms)" 0 --> 812
bar [360, 608.6, 676, 605.6, 113.1]
y-axis "Time (ms)" 0 --> 792
bar [375.9, 599.8, 659.9, 607.7, 117]
```

## 🎯 Key Insights
Expand All @@ -74,4 +74,4 @@ This benchmark compares TUnit's performance against NUnit, MSTest, xUnit3 using
View the [benchmarks overview](/docs/benchmarks) for methodology details and environment information.
:::

*Last generated: 2026-07-12T00:38:25.258Z*
*Last generated: 2026-07-16T16:49:09.139Z*
18 changes: 9 additions & 9 deletions docs/docs/benchmarks/engine/BuildTime.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,9 @@ sidebar_position: 9
> Compilation time from a clean build across frameworks — how long it takes to build an identical test project.

:::info Last Updated
This benchmark was automatically generated on **2026-07-12** from the latest CI run.
This benchmark was automatically generated on **2026-07-16** from the latest CI run.

**Environment:** Ubuntu Latest • .NET SDK 10.0.301
**Environment:** Ubuntu Latest • .NET SDK 10.0.302
:::

## 📊 Results
Expand All @@ -20,10 +20,10 @@ Compilation time comparison across frameworks:

| Framework | Version | Mean | Median | StdDev |
|-----------|---------|------|--------|--------|
| **TUnit** | 1.58.0 | 1,146.4 ms | 1,144.9 ms | 57.04 ms |
| Build_NUnit | 4.6.1 | 1,083.5 ms | 1,081.5 ms | 19.23 ms |
| Build_MSTest | 4.3.0 | 986.7 ms | 990.0 ms | 26.70 ms |
| Build_xUnit3 | 3.2.2 | 817.4 ms | 817.5 ms | 8.10 ms |
| **TUnit** | 1.60.0 | 931.9 ms | 920.1 ms | 48.38 ms |
| Build_NUnit | 4.6.1 | 901.1 ms | 891.3 ms | 31.51 ms |
| Build_MSTest | 4.3.2 | 1,046.9 ms | 1,060.8 ms | 76.11 ms |
| Build_xUnit3 | 3.2.2 | 857.4 ms | 858.7 ms | 7.54 ms |

## 📈 Visual Comparison

Expand Down Expand Up @@ -61,8 +61,8 @@ Compilation time comparison across frameworks:
xychart-beta
title "Build Time Comparison"
x-axis ["Build_TUnit", "Build_NUnit", "Build_MSTest", "Build_xUnit3"]
y-axis "Time (ms)" 0 --> 1376
bar [1146.4, 1083.5, 986.7, 817.4]
y-axis "Time (ms)" 0 --> 1257
bar [931.9, 901.1, 1046.9, 857.4]
```

---
Expand All @@ -71,4 +71,4 @@ xychart-beta
View the [benchmarks overview](/docs/benchmarks) for methodology details and environment information.
:::

*Last generated: 2026-07-12T00:38:25.259Z*
*Last generated: 2026-07-16T16:49:09.141Z*
20 changes: 10 additions & 10 deletions docs/docs/benchmarks/engine/DataDrivenTests.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,20 +9,20 @@ sidebar_position: 4
> Parameterized tests with multiple data sources

:::info Last Updated
This benchmark was automatically generated on **2026-07-12** from the latest CI run.
This benchmark was automatically generated on **2026-07-16** from the latest CI run.

**Environment:** Ubuntu Latest • .NET SDK 10.0.301
**Environment:** Ubuntu Latest • .NET SDK 10.0.302
:::

## 📊 Results

| Framework | Version | Mean | Median | StdDev |
|-----------|---------|------|--------|--------|
| **TUnit** | 1.58.0 | 265.70 ms | 264.36 ms | 9.348 ms |
| NUnit | 4.6.1 | 523.01 ms | 519.57 ms | 14.061 ms |
| MSTest | 4.3.0 | 529.13 ms | 527.68 ms | 34.022 ms |
| xUnit3 | 3.2.2 | 523.90 ms | 517.07 ms | 24.917 ms |
| **TUnit (AOT)** | 1.58.0 | 16.65 ms | 16.68 ms | 1.655 ms |
| **TUnit** | 1.60.0 | 333.55 ms | 329.63 ms | 26.217 ms |
| NUnit | 4.6.1 | 588.63 ms | 588.62 ms | 30.349 ms |
| MSTest | 4.3.2 | 643.67 ms | 644.95 ms | 34.220 ms |
| xUnit3 | 3.2.2 | 612.50 ms | 613.31 ms | 23.558 ms |
| **TUnit (AOT)** | 1.60.0 | 18.46 ms | 18.50 ms | 1.089 ms |

## 📈 Visual Comparison

Expand Down Expand Up @@ -60,8 +60,8 @@ This benchmark was automatically generated on **2026-07-12** from the latest CI
xychart-beta
title "DataDrivenTests Performance Comparison"
x-axis ["TUnit", "NUnit", "MSTest", "xUnit3", "TUnit_AOT"]
y-axis "Time (ms)" 0 --> 635
bar [265.7, 523.01, 529.13, 523.9, 16.65]
y-axis "Time (ms)" 0 --> 773
bar [333.55, 588.63, 643.67, 612.5, 18.46]
```

## 🎯 Key Insights
Expand All @@ -74,4 +74,4 @@ This benchmark compares TUnit's performance against NUnit, MSTest, xUnit3 using
View the [benchmarks overview](/docs/benchmarks) for methodology details and environment information.
:::

*Last generated: 2026-07-12T00:38:25.258Z*
*Last generated: 2026-07-16T16:49:09.140Z*
20 changes: 10 additions & 10 deletions docs/docs/benchmarks/engine/MassiveParallelTests.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,20 +9,20 @@ sidebar_position: 5
> Parallel execution stress tests

:::info Last Updated
This benchmark was automatically generated on **2026-07-12** from the latest CI run.
This benchmark was automatically generated on **2026-07-16** from the latest CI run.

**Environment:** Ubuntu Latest • .NET SDK 10.0.301
**Environment:** Ubuntu Latest • .NET SDK 10.0.302
:::

## 📊 Results

| Framework | Version | Mean | Median | StdDev |
|-----------|---------|------|--------|--------|
| **TUnit** | 1.58.0 | 467.8 ms | 467.2 ms | 7.40 ms |
| NUnit | 4.6.1 | 1,107.5 ms | 1,109.8 ms | 26.10 ms |
| MSTest | 4.3.0 | 2,993.9 ms | 2,985.5 ms | 23.91 ms |
| xUnit3 | 3.2.2 | 2,945.7 ms | 2,945.5 ms | 13.92 ms |
| **TUnit (AOT)** | 1.58.0 | 217.6 ms | 217.7 ms | 0.42 ms |
| **TUnit** | 1.60.0 | 530.9 ms | 525.1 ms | 20.42 ms |
| NUnit | 4.6.1 | 1,122.5 ms | 1,114.3 ms | 21.95 ms |
| MSTest | 4.3.2 | 3,029.7 ms | 3,024.8 ms | 30.72 ms |
| xUnit3 | 3.2.2 | 2,975.1 ms | 2,971.1 ms | 44.00 ms |
| **TUnit (AOT)** | 1.60.0 | 217.5 ms | 217.1 ms | 1.20 ms |

## 📈 Visual Comparison

Expand Down Expand Up @@ -60,8 +60,8 @@ This benchmark was automatically generated on **2026-07-12** from the latest CI
xychart-beta
title "MassiveParallelTests Performance Comparison"
x-axis ["TUnit", "NUnit", "MSTest", "xUnit3", "TUnit_AOT"]
y-axis "Time (ms)" 0 --> 3593
bar [467.8, 1107.5, 2993.9, 2945.7, 217.6]
y-axis "Time (ms)" 0 --> 3636
bar [530.9, 1122.5, 3029.7, 2975.1, 217.5]
```

## 🎯 Key Insights
Expand All @@ -74,4 +74,4 @@ This benchmark compares TUnit's performance against NUnit, MSTest, xUnit3 using
View the [benchmarks overview](/docs/benchmarks) for methodology details and environment information.
:::

*Last generated: 2026-07-12T00:38:25.258Z*
*Last generated: 2026-07-16T16:49:09.140Z*
20 changes: 10 additions & 10 deletions docs/docs/benchmarks/engine/MatrixTests.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,20 +9,20 @@ sidebar_position: 6
> Combinatorial test generation and execution

:::info Last Updated
This benchmark was automatically generated on **2026-07-12** from the latest CI run.
This benchmark was automatically generated on **2026-07-16** from the latest CI run.

**Environment:** Ubuntu Latest • .NET SDK 10.0.301
**Environment:** Ubuntu Latest • .NET SDK 10.0.302
:::

## 📊 Results

| Framework | Version | Mean | Median | StdDev |
|-----------|---------|------|--------|--------|
| **TUnit** | 1.58.0 | 474.1 ms | 482.5 ms | 60.45 ms |
| NUnit | 4.6.1 | 1,468.9 ms | 1,470.3 ms | 25.32 ms |
| MSTest | 4.3.0 | 1,515.3 ms | 1,514.5 ms | 14.28 ms |
| xUnit3 | 3.2.2 | 1,469.2 ms | 1,470.9 ms | 8.94 ms |
| **TUnit (AOT)** | 1.58.0 | 119.6 ms | 119.5 ms | 2.14 ms |
| **TUnit** | 1.60.0 | 388.2 ms | 389.1 ms | 3.28 ms |
| NUnit | 4.6.1 | 1,452.1 ms | 1,452.6 ms | 7.35 ms |
| MSTest | 4.3.2 | 1,541.1 ms | 1,541.2 ms | 6.08 ms |
| xUnit3 | 3.2.2 | 1,491.5 ms | 1,489.1 ms | 12.51 ms |
| **TUnit (AOT)** | 1.60.0 | 118.1 ms | 117.9 ms | 0.96 ms |

## 📈 Visual Comparison

Expand Down Expand Up @@ -60,8 +60,8 @@ This benchmark was automatically generated on **2026-07-12** from the latest CI
xychart-beta
title "MatrixTests Performance Comparison"
x-axis ["TUnit", "NUnit", "MSTest", "xUnit3", "TUnit_AOT"]
y-axis "Time (ms)" 0 --> 1819
bar [474.1, 1468.9, 1515.3, 1469.2, 119.6]
y-axis "Time (ms)" 0 --> 1850
bar [388.2, 1452.1, 1541.1, 1491.5, 118.1]
```

## 🎯 Key Insights
Expand All @@ -74,4 +74,4 @@ This benchmark compares TUnit's performance against NUnit, MSTest, xUnit3 using
View the [benchmarks overview](/docs/benchmarks) for methodology details and environment information.
:::

*Last generated: 2026-07-12T00:38:25.258Z*
*Last generated: 2026-07-16T16:49:09.140Z*
20 changes: 10 additions & 10 deletions docs/docs/benchmarks/engine/ScaleTests.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,20 +9,20 @@ sidebar_position: 7
> Large test suites (150+ tests) measuring scalability

:::info Last Updated
This benchmark was automatically generated on **2026-07-12** from the latest CI run.
This benchmark was automatically generated on **2026-07-16** from the latest CI run.

**Environment:** Ubuntu Latest • .NET SDK 10.0.301
**Environment:** Ubuntu Latest • .NET SDK 10.0.302
:::

## 📊 Results

| Framework | Version | Mean | Median | StdDev |
|-----------|---------|------|--------|--------|
| **TUnit** | 1.58.0 | 263.55 ms | 263.67 ms | 1.995 ms |
| NUnit | 4.6.1 | 488.30 ms | 487.42 ms | 7.690 ms |
| MSTest | 4.3.0 | 495.94 ms | 495.27 ms | 8.060 ms |
| xUnit3 | 3.2.2 | 504.14 ms | 499.49 ms | 13.636 ms |
| **TUnit (AOT)** | 1.58.0 | 29.38 ms | 29.52 ms | 2.269 ms |
| **TUnit** | 1.60.0 | 320.96 ms | 320.67 ms | 12.233 ms |
| NUnit | 4.6.1 | 520.19 ms | 513.89 ms | 15.712 ms |
| MSTest | 4.3.2 | 566.98 ms | 568.49 ms | 35.283 ms |
| xUnit3 | 3.2.2 | 520.49 ms | 516.76 ms | 19.655 ms |
| **TUnit (AOT)** | 1.60.0 | 18.25 ms | 17.91 ms | 1.478 ms |

## 📈 Visual Comparison

Expand Down Expand Up @@ -60,8 +60,8 @@ This benchmark was automatically generated on **2026-07-12** from the latest CI
xychart-beta
title "ScaleTests Performance Comparison"
x-axis ["TUnit", "NUnit", "MSTest", "xUnit3", "TUnit_AOT"]
y-axis "Time (ms)" 0 --> 605
bar [263.55, 488.3, 495.94, 504.14, 29.38]
y-axis "Time (ms)" 0 --> 681
bar [320.96, 520.19, 566.98, 520.49, 18.25]
```

## 🎯 Key Insights
Expand All @@ -74,4 +74,4 @@ This benchmark compares TUnit's performance against NUnit, MSTest, xUnit3 using
View the [benchmarks overview](/docs/benchmarks) for methodology details and environment information.
:::

*Last generated: 2026-07-12T00:38:25.259Z*
*Last generated: 2026-07-16T16:49:09.140Z*
20 changes: 10 additions & 10 deletions docs/docs/benchmarks/engine/SetupTeardownTests.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,20 +9,20 @@ sidebar_position: 8
> Expensive test fixtures with setup/teardown overhead

:::info Last Updated
This benchmark was automatically generated on **2026-07-12** from the latest CI run.
This benchmark was automatically generated on **2026-07-16** from the latest CI run.

**Environment:** Ubuntu Latest • .NET SDK 10.0.301
**Environment:** Ubuntu Latest • .NET SDK 10.0.302
:::

## 📊 Results

| Framework | Version | Mean | Median | StdDev |
|-----------|---------|------|--------|--------|
| **TUnit** | 1.58.0 | 335.8 ms | 335.5 ms | 1.81 ms |
| NUnit | 4.6.1 | 1,008.8 ms | 1,007.3 ms | 6.40 ms |
| MSTest | 4.3.0 | 1,099.9 ms | 1,099.0 ms | 5.80 ms |
| xUnit3 | 3.2.2 | 1,050.2 ms | 1,049.8 ms | 5.27 ms |
| **TUnit (AOT)** | 1.58.0 | NA | NA | NA |
| **TUnit** | 1.60.0 | 458.30 ms | 463.95 ms | 35.039 ms |
| NUnit | 4.6.1 | 1,147.56 ms | 1,149.59 ms | 37.016 ms |
| MSTest | 4.3.2 | 1,279.43 ms | 1,280.80 ms | 24.024 ms |
| xUnit3 | 3.2.2 | 1,186.16 ms | 1,185.53 ms | 29.989 ms |
| **TUnit (AOT)** | 1.60.0 | 73.35 ms | 73.16 ms | 1.190 ms |

## 📈 Visual Comparison

Expand Down Expand Up @@ -60,8 +60,8 @@ This benchmark was automatically generated on **2026-07-12** from the latest CI
xychart-beta
title "SetupTeardownTests Performance Comparison"
x-axis ["TUnit", "NUnit", "MSTest", "xUnit3", "TUnit_AOT"]
y-axis "Time (ms)" 0 --> 1320
bar [335.8, 1008.8, 1099.9, 1050.2, 0]
y-axis "Time (ms)" 0 --> 1536
bar [458.3, 1147.56, 1279.43, 1186.16, 73.35]
```

## 🎯 Key Insights
Expand All @@ -74,4 +74,4 @@ This benchmark compares TUnit's performance against NUnit, MSTest, xUnit3 using
View the [benchmarks overview](/docs/benchmarks) for methodology details and environment information.
:::

*Last generated: 2026-07-12T00:38:25.259Z*
*Last generated: 2026-07-16T16:49:09.141Z*
14 changes: 7 additions & 7 deletions docs/docs/benchmarks/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,9 @@ sidebar_position: 1
# Performance Benchmarks

:::info Last Updated
These benchmarks were automatically generated on **2026-07-12** from the latest CI run.
These benchmarks were automatically generated on **2026-07-16** from the latest CI run.

**Environment:** Ubuntu Latest • .NET SDK 10.0.301
**Environment:** Ubuntu Latest • .NET SDK 10.0.302
:::

## 🚀 Runtime Benchmarks
Expand Down Expand Up @@ -37,10 +37,10 @@ These benchmarks compare TUnit against the most popular .NET testing frameworks:

| Framework | Version Tested |
|-----------|----------------|
| **TUnit** | 1.58.0 |
| **TUnit** | 1.60.0 |
| **xUnit v3** | 3.2.2 |
| **NUnit** | 4.6.1 |
| **MSTest** | 4.3.0 |
| **MSTest** | 4.3.2 |

### Test Scenarios

Expand All @@ -56,8 +56,8 @@ The benchmarks measure real-world testing patterns:
### Environment

- **OS**: Ubuntu Latest (GitHub Actions)
- **Runtime**: .NET 10.0.9 (10.0.9, 10.0.926.27113), X64 RyuJIT x86-64-v4
- **SDK**: .NET SDK 10.0.301
- **Runtime**: .NET 10.0.10 (10.0.10, 10.0.1026.32716), X64 RyuJIT x86-64-v4
- **SDK**: .NET SDK 10.0.302
- **Hardware**: GitHub Actions Standard Runner (Ubuntu)
- **Tool**: BenchmarkDotNet v0.15.8, Linux Ubuntu 24.04.4 LTS (Noble Numbat)

Expand All @@ -80,4 +80,4 @@ These benchmarks run automatically daily via [GitHub Actions](https://github.com
Each benchmark runs multiple iterations with statistical analysis to ensure accuracy. Results may vary based on hardware and test characteristics.
:::

*Last generated: 2026-07-12T00:38:25.260Z*
*Last generated: 2026-07-16T16:49:09.141Z*
Loading
Loading