S3 External Storage Driver #783
Description
Is your feature request related to a problem? Please describe.
The external payload storage foundation described in #765 enables payloads to be externally stored for various customer reasons. But there are no pre-built storage drivers for customers to use, thus requiring customers to write their own drivers. The SDK should provide ones with sufficient demand, such as an AWS S3 driver.
Describe the solution you'd like
A pre-built storage driver that can handle most customers needs for storing their large payloads in AWS S3.
- Can store and retrieve from buckets that are statically named or dynamically determined based on payload information and execution context.
- Stores payloads with object keys in a hierarchy that allows customers to implement lifecycle management easily (e.g. delete all keys that correspond to run X of workflow Y).
- Stores payloads with object keys to maximize the deduplication of storing the same payload data within certain context levels. For example, a large payload that is passed in and out via activity inputs and results within the same workflow is ideally deduplicated to a single object key to optimize storage and prevent unnecessary transfers.
- Driver is a separate module/package/bundle that doesn't pull in S3 dependencies unless actually required based on inclusion of the driver.
Per-SDK Tickets
- S3 External Storage Driver - Go sdk-go#2251
- Java -
- TypeScript -
- S3 External Storage Driver - Python sdk-python#1390
- .NET -
- Ruby -
- Rust -
- PHP -
- Temporal CLI -