Adds a streaming based attachment functionality to NServiceBus.
See Milestones for release notes.
Already a Patron? skip past this section
It is expected that all developers become a Patron to use NServiceBus Community Extensions. Go to licensing FAQ
Support this project by becoming a Sponsor. The company avatar will show up here with a website link. The avatar will also be added to all GitHub repositories under the NServiceBusCommunity organization.
Thanks to all the backing developers. Support this project by becoming a patron.
- https://www.nuget.org/packages/NServiceBus.Community.Attachments.FileShare
- https://www.nuget.org/packages/NServiceBus.Community.Attachments.FileShare.Raw
- https://www.nuget.org/packages/NServiceBus.Community.Attachments.Sql
- https://www.nuget.org/packages/NServiceBus.Community.Attachments.Sql.Raw
This project delivers similar functionality to the DataBus. However it does have some different behavior:
With the DataBus all binary data is read every message received. This is irrespective of if the receiving endpoint requires that data. With NServiceBus.Attachments data is explicitly read on demand, so if data is not required there is no performance impact. NServiceBus.Attachments also supports processing all data items via an IAsyncEnumerable.
With the DataBus all data items are placed into byte arrays. This means that memory needs to be allocated to store those arrays on either reading or writing. With NServiceBus.Attachments data can be streamed and processed in an async manner. This can significantly decrease the memory pressure on an endpoint.
With the DataBus the only interaction is via byte arrays. NServiceBus.Attachments supports reading and writing using streams, byte arrays, or string.
Benchmark results for attachment persister operations at different data sizes. Results collected using BenchmarkDotNet.
- BenchmarkDotNet v0.15.8, Windows 10 (10.0.19045.7058/22H2/2022Update)
- Intel Core i9-9880H CPU 2.30GHz, 1 CPU, 16 logical and 8 physical cores
- .NET SDK 10.0.200
Uses LocalDb with a per-benchmark isolated database.
| Method | DataSize | Mean | Allocated |
|---|---|---|---|
| SaveStream | 1 KB | 11.34 ms | 25.30 KB |
| SaveBytes | 1 KB | 11.36 ms | 21.10 KB |
| SaveAndGetBytes | 1 KB | 16.87 ms | 38.67 KB |
| SaveAndCopyTo | 1 KB | 17.01 ms | 42.41 KB |
| SaveAndGetStream | 1 KB | 16.90 ms | 41.16 KB |
| SaveStream | 100 KB | 13.24 ms | 129.63 KB |
| SaveBytes | 100 KB | 13.57 ms | 124.77 KB |
| SaveAndGetBytes | 100 KB | 19.29 ms | 340.77 KB |
| SaveAndCopyTo | 100 KB | 19.00 ms | 152.20 KB |
| SaveAndGetStream | 100 KB | 18.01 ms | 146.01 KB |
| SaveStream | 1 MB | 37.78 ms | 1097.60 KB |
| SaveBytes | 1 MB | 37.37 ms | 1082.72 KB |
| SaveAndGetBytes | 1 MB | 44.25 ms | 3147.55 KB |
| SaveAndCopyTo | 1 MB | 45.02 ms | 1306.87 KB |
| SaveAndGetStream | 1 MB | 43.94 ms | 1111.88 KB |
| SaveStream | 10 MB | 251.07 ms | 10884.69 KB |
| SaveBytes | 10 MB | 256.15 ms | 10729.69 KB |
| SaveAndGetBytes | 10 MB | 278.18 ms | 31194.30 KB |
| SaveAndCopyTo | 10 MB | 291.02 ms | 11560.28 KB |
| SaveAndGetStream | 10 MB | 282.84 ms | 10790.12 KB |
| Method | DataSize | Mean | Allocated |
|---|---|---|---|
| SaveStream | 1 KB | 0.71 ms | 68.76 KB |
| SaveBytes | 1 KB | 0.66 ms | 68.70 KB |
| SaveAndGetBytes | 1 KB | 0.81 ms | 71.39 KB |
| SaveAndCopyTo | 1 KB | 1.00 ms | 70.63 KB |
| SaveAndGetStream | 1 KB | 0.81 ms | 70.13 KB |
| SaveStream | 100 KB | 0.64 ms | 103.60 KB |
| SaveBytes | 100 KB | 0.67 ms | 103.54 KB |
| SaveAndGetBytes | 100 KB | 0.89 ms | 205.10 KB |
| SaveAndCopyTo | 100 KB | 1.04 ms | 105.34 KB |
| SaveAndGetStream | 100 KB | 0.88 ms | 104.84 KB |
| SaveStream | 1 MB | 1.11 ms | 1027.47 KB |
| SaveBytes | 1 MB | 1.10 ms | 1027.41 KB |
| SaveAndGetBytes | 1 MB | 1.68 ms | 2053.10 KB |
| SaveAndCopyTo | 1 MB | 1.79 ms | 1029.34 KB |
| SaveAndGetStream | 1 MB | 1.33 ms | 1028.84 KB |
| SaveStream | 10 MB | 4.90 ms | 10243.47 KB |
| SaveBytes | 10 MB | 5.23 ms | 10243.41 KB |
| SaveAndGetBytes | 10 MB | 8.49 ms | 20485.09 KB |
| SaveAndCopyTo | 10 MB | 8.48 ms | 10245.34 KB |
| SaveAndGetStream | 10 MB | 5.34 ms | 10244.84 KB |
- FileShare is ~15-50x faster than SQL for raw operations, as expected for local file I/O vs database round-trips.
- Streaming avoids double allocation on read: At 10 MB,
SaveAndGetStreamallocates ~10 MB (the data itself), whileSaveAndGetBytesallocates ~20-31 MB (data copied into a byte array on top of the original). - SQL save cost scales with data size: ~11ms for 1 KB, ~13ms for 100 KB, ~38ms for 1 MB, ~254ms for 10 MB. FileShare stays under 6ms for all sizes up to 10 MB.
SaveStreamvsSaveBytes: Nearly identical performance in both implementations.
Gecko designed by Alex Podolsky from The Noun Project.
