From 435fd1a875ac8bf01a8d7e4b88e4e304f66b31c3 Mon Sep 17 00:00:00 2001 From: "Artur M. Wolff" Date: Wed, 5 Nov 2025 16:50:47 +0100 Subject: [PATCH 1/6] app/(docs)/dcs/api/s3/s3-compatibility: add storj-vs-self-managed-encryption-s3-compatibility-differences --- app/(docs)/dcs/api/s3/s3-compatibility/page.md | 2 +- .../page.md | 5 +++++ 2 files changed, 6 insertions(+), 1 deletion(-) create mode 100644 app/(docs)/dcs/api/s3/s3-compatibility/storj-vs-self-managed-encryption-s3-compatibility-differences/page.md diff --git a/app/(docs)/dcs/api/s3/s3-compatibility/page.md b/app/(docs)/dcs/api/s3/s3-compatibility/page.md index ffcf0d508..b995c00a6 100644 --- a/app/(docs)/dcs/api/s3/s3-compatibility/page.md +++ b/app/(docs)/dcs/api/s3/s3-compatibility/page.md @@ -1,5 +1,5 @@ --- -title: S3 Compatibility +title: S3-Compatibility docId: eZ4caegh9queuQuaazoo weight: 1 redirects: diff --git a/app/(docs)/dcs/api/s3/s3-compatibility/storj-vs-self-managed-encryption-s3-compatibility-differences/page.md b/app/(docs)/dcs/api/s3/s3-compatibility/storj-vs-self-managed-encryption-s3-compatibility-differences/page.md new file mode 100644 index 000000000..2e5ebfb57 --- /dev/null +++ b/app/(docs)/dcs/api/s3/s3-compatibility/storj-vs-self-managed-encryption-s3-compatibility-differences/page.md @@ -0,0 +1,5 @@ +--- +title: Storj Managed vs. Self-Managed Encryption S3-Compatibility Differences +docId: oozaauQueuq9hgeac4Ze +weight: 1 +--- From 67d3cbe41e0d473bbc415737b8686368de289879 Mon Sep 17 00:00:00 2001 From: "Artur M. Wolff" Date: Tue, 25 Nov 2025 19:37:07 +0100 Subject: [PATCH 2/6] write the page --- .../page.md | 39 +++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/app/(docs)/dcs/api/s3/s3-compatibility/storj-vs-self-managed-encryption-s3-compatibility-differences/page.md b/app/(docs)/dcs/api/s3/s3-compatibility/storj-vs-self-managed-encryption-s3-compatibility-differences/page.md index 2e5ebfb57..7e18e2c0b 100644 --- a/app/(docs)/dcs/api/s3/s3-compatibility/storj-vs-self-managed-encryption-s3-compatibility-differences/page.md +++ b/app/(docs)/dcs/api/s3/s3-compatibility/storj-vs-self-managed-encryption-s3-compatibility-differences/page.md @@ -3,3 +3,42 @@ title: Storj Managed vs. Self-Managed Encryption S3-Compatibility Differences docId: oozaauQueuq9hgeac4Ze weight: 1 --- + +## Listing endpoints behavior + +If your project is using Self-Managed Encryption, a bucket's paths are +end-to-end encrypted, which means that it's impossible to always list a +bucket in lexicographical order (as per S3 specification). For requests +that come with forward-slash-terminated prefix and/or forward-slash +delimiter, listing endpoints return results in lexicographical order, +but for encrypted paths (which is often very different from the expected +order for decrypted paths). Ideally, clients shouldn't care about +ordering in those cases. For requests that come with +non-forward-slash-terminated prefix and/or non-forward-slash delimiter, +listing endpoints perform exhaustive listing, which filters paths +server-side before they are returned to the caller. In this case, +listing results are in lexicographical order. + +Projects using Self-Managed Encryption created after November 30, 2025 +are excluded from the usage of arbitrary prefixes and/or delimiters to +perform exhaustive listing. Such requests will immediately be rejected +by the listing endpoints with the 501 Not Implemented S3-compatible HTTP +error code. Users who need to create new projects utilizing exhaustive +listing are encouraged to create projects using Storj Managed +Encryption. + +Exhaustive listing won't work for buckets containing hundreds of +thousands of objects. Users who need to perform exhaustive listing with +such buckets are encouraged to start utilizing projects using Storj +Managed Encryption. + +### Storj Managed Encryption + +If your project is using Storj Managed Encryption, a bucket's paths are +kept in plaintext. This means that the behavior of Storj's S3 listing +endpoints is fully S3-compatible. Requests can come with an arbitrary +prefix and/or delimiter and they will take a preferable execution path, +resulting in superior listing times compared to listing over buckets in +any projects using Self-Managed Encryption. Storj recommends all new +projects to be projects using Storj Managed Encryption for users that +care about S3-compatibility, robustness, and performance. From b6e5b37bfa5e5c1758900e7a774c5fbf345bf4b3 Mon Sep 17 00:00:00 2001 From: "Artur M. Wolff" Date: Tue, 9 Dec 2025 17:23:55 +0100 Subject: [PATCH 3/6] Update page.md --- .../page.md | 52 ++++++------------- 1 file changed, 17 insertions(+), 35 deletions(-) diff --git a/app/(docs)/dcs/api/s3/s3-compatibility/storj-vs-self-managed-encryption-s3-compatibility-differences/page.md b/app/(docs)/dcs/api/s3/s3-compatibility/storj-vs-self-managed-encryption-s3-compatibility-differences/page.md index 7e18e2c0b..6d3685cbd 100644 --- a/app/(docs)/dcs/api/s3/s3-compatibility/storj-vs-self-managed-encryption-s3-compatibility-differences/page.md +++ b/app/(docs)/dcs/api/s3/s3-compatibility/storj-vs-self-managed-encryption-s3-compatibility-differences/page.md @@ -4,41 +4,23 @@ docId: oozaauQueuq9hgeac4Ze weight: 1 --- -## Listing endpoints behavior - -If your project is using Self-Managed Encryption, a bucket's paths are -end-to-end encrypted, which means that it's impossible to always list a -bucket in lexicographical order (as per S3 specification). For requests -that come with forward-slash-terminated prefix and/or forward-slash -delimiter, listing endpoints return results in lexicographical order, -but for encrypted paths (which is often very different from the expected -order for decrypted paths). Ideally, clients shouldn't care about -ordering in those cases. For requests that come with -non-forward-slash-terminated prefix and/or non-forward-slash delimiter, -listing endpoints perform exhaustive listing, which filters paths -server-side before they are returned to the caller. In this case, -listing results are in lexicographical order. - -Projects using Self-Managed Encryption created after November 30, 2025 -are excluded from the usage of arbitrary prefixes and/or delimiters to -perform exhaustive listing. Such requests will immediately be rejected -by the listing endpoints with the 501 Not Implemented S3-compatible HTTP -error code. Users who need to create new projects utilizing exhaustive -listing are encouraged to create projects using Storj Managed -Encryption. - -Exhaustive listing won't work for buckets containing hundreds of -thousands of objects. Users who need to perform exhaustive listing with -such buckets are encouraged to start utilizing projects using Storj -Managed Encryption. +## Listing Endpoints Behavior + +The behavior of listing endpoints depends on the encryption model configured for your project. + +### Self-Managed Encryption +When using **Self-Managed Encryption**, bucket paths are end-to-end encrypted. This prevents the listing endpoint from seeing the plaintext paths, which affects sorting and filtering. + +* **Standard Listing:** For requests using a forward-slash (`/`) as a prefix terminator or delimiter, results are returned in the lexicographical order of the *encrypted* paths. This often differs from the expected alphabetical order of the decrypted keys. Clients should not rely on ordering in these cases. +* **Exhaustive Listing:** For requests using non-standard prefixes or delimiters (not ending in `/`), the endpoint performs an "exhaustive listing." This filters paths server-side to return results in the correct (decrypted) lexicographical order. + +**Limitations:** +1. **New Projects:** Projects created after **November 30, 2025**, cannot use exhaustive listing. Requests triggering this behavior will fail with a `501 Not Implemented` error. +2. **Large Buckets:** Exhaustive listing does not work for buckets containing hundreds of thousands of objects. + +Users requiring these features should use Storj Managed Encryption. ### Storj Managed Encryption +If your project uses **Storj Managed Encryption**, bucket paths are stored in plaintext. This ensures that Storj's S3 listing endpoints are **fully S3-compatible**. -If your project is using Storj Managed Encryption, a bucket's paths are -kept in plaintext. This means that the behavior of Storj's S3 listing -endpoints is fully S3-compatible. Requests can come with an arbitrary -prefix and/or delimiter and they will take a preferable execution path, -resulting in superior listing times compared to listing over buckets in -any projects using Self-Managed Encryption. Storj recommends all new -projects to be projects using Storj Managed Encryption for users that -care about S3-compatibility, robustness, and performance. +Requests can use arbitrary prefixes and/or delimiters. These requests utilize an optimized execution path, offering superior performance compared to Self-Managed Encryption. Storj recommends Storj Managed Encryption for all new projects requiring high performance and strict S3 compatibility. From da855ae7bdaf33041e5d916f3095b811412d6d68 Mon Sep 17 00:00:00 2001 From: "Artur M. Wolff" Date: Tue, 9 Dec 2025 17:28:40 +0100 Subject: [PATCH 4/6] use "s3 compatibility" as a noun properly --- app/(docs)/dcs/api/s3/s3-compatibility/page.md | 2 +- .../page.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/app/(docs)/dcs/api/s3/s3-compatibility/page.md b/app/(docs)/dcs/api/s3/s3-compatibility/page.md index b995c00a6..ffcf0d508 100644 --- a/app/(docs)/dcs/api/s3/s3-compatibility/page.md +++ b/app/(docs)/dcs/api/s3/s3-compatibility/page.md @@ -1,5 +1,5 @@ --- -title: S3-Compatibility +title: S3 Compatibility docId: eZ4caegh9queuQuaazoo weight: 1 redirects: diff --git a/app/(docs)/dcs/api/s3/s3-compatibility/storj-vs-self-managed-encryption-s3-compatibility-differences/page.md b/app/(docs)/dcs/api/s3/s3-compatibility/storj-vs-self-managed-encryption-s3-compatibility-differences/page.md index 6d3685cbd..ed4f568e3 100644 --- a/app/(docs)/dcs/api/s3/s3-compatibility/storj-vs-self-managed-encryption-s3-compatibility-differences/page.md +++ b/app/(docs)/dcs/api/s3/s3-compatibility/storj-vs-self-managed-encryption-s3-compatibility-differences/page.md @@ -1,5 +1,5 @@ --- -title: Storj Managed vs. Self-Managed Encryption S3-Compatibility Differences +title: Storj Managed vs. Self-Managed Encryption S3 Compatibility Differences docId: oozaauQueuq9hgeac4Ze weight: 1 --- From 7c09110b1b6954796e6cf58793d0bb06a001e743 Mon Sep 17 00:00:00 2001 From: "Artur M. Wolff" Date: Wed, 10 Dec 2025 11:57:56 +0100 Subject: [PATCH 5/6] clarify how bucket paths are encrypted --- .../page.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/app/(docs)/dcs/api/s3/s3-compatibility/storj-vs-self-managed-encryption-s3-compatibility-differences/page.md b/app/(docs)/dcs/api/s3/s3-compatibility/storj-vs-self-managed-encryption-s3-compatibility-differences/page.md index ed4f568e3..88a91067b 100644 --- a/app/(docs)/dcs/api/s3/s3-compatibility/storj-vs-self-managed-encryption-s3-compatibility-differences/page.md +++ b/app/(docs)/dcs/api/s3/s3-compatibility/storj-vs-self-managed-encryption-s3-compatibility-differences/page.md @@ -9,7 +9,7 @@ weight: 1 The behavior of listing endpoints depends on the encryption model configured for your project. ### Self-Managed Encryption -When using **Self-Managed Encryption**, bucket paths are end-to-end encrypted. This prevents the listing endpoint from seeing the plaintext paths, which affects sorting and filtering. +When using **Self-Managed Encryption**, bucket paths are end-to-end encrypted. This prevents the listing endpoint from deciphering path names, which affects sorting and filtering. * **Standard Listing:** For requests using a forward-slash (`/`) as a prefix terminator or delimiter, results are returned in the lexicographical order of the *encrypted* paths. This often differs from the expected alphabetical order of the decrypted keys. Clients should not rely on ordering in these cases. * **Exhaustive Listing:** For requests using non-standard prefixes or delimiters (not ending in `/`), the endpoint performs an "exhaustive listing." This filters paths server-side to return results in the correct (decrypted) lexicographical order. @@ -21,6 +21,6 @@ When using **Self-Managed Encryption**, bucket paths are end-to-end encrypted. T Users requiring these features should use Storj Managed Encryption. ### Storj Managed Encryption -If your project uses **Storj Managed Encryption**, bucket paths are stored in plaintext. This ensures that Storj's S3 listing endpoints are **fully S3-compatible**. +If your project uses **Storj Managed Encryption**, bucket paths are **encrypted using server-managed keys**. This allows the system to process path names directly, ensuring that Storj's S3 listing endpoints are **fully S3-compatible**. Requests can use arbitrary prefixes and/or delimiters. These requests utilize an optimized execution path, offering superior performance compared to Self-Managed Encryption. Storj recommends Storj Managed Encryption for all new projects requiring high performance and strict S3 compatibility. From 5b11df42a96a29d7d86b66feb877130b8ad8f831 Mon Sep 17 00:00:00 2001 From: "Artur M. Wolff" Date: Wed, 10 Dec 2025 12:03:06 +0100 Subject: [PATCH 6/6] swap paths for objects keys and add at-a-glance table --- .../page.md | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/app/(docs)/dcs/api/s3/s3-compatibility/storj-vs-self-managed-encryption-s3-compatibility-differences/page.md b/app/(docs)/dcs/api/s3/s3-compatibility/storj-vs-self-managed-encryption-s3-compatibility-differences/page.md index 88a91067b..8f7c3e051 100644 --- a/app/(docs)/dcs/api/s3/s3-compatibility/storj-vs-self-managed-encryption-s3-compatibility-differences/page.md +++ b/app/(docs)/dcs/api/s3/s3-compatibility/storj-vs-self-managed-encryption-s3-compatibility-differences/page.md @@ -8,11 +8,16 @@ weight: 1 The behavior of listing endpoints depends on the encryption model configured for your project. +| Encryption Type | S3 Compatibility | +| :--- | :--- | +| **Self-Managed Encryption** | **Limited** (Restricted sorting/filtering) | +| **Storj Managed Encryption** | **Full** (Fully S3-compatible) | + ### Self-Managed Encryption -When using **Self-Managed Encryption**, bucket paths are end-to-end encrypted. This prevents the listing endpoint from deciphering path names, which affects sorting and filtering. +When using **Self-Managed Encryption**, object keys are end-to-end encrypted. This prevents the listing endpoint from deciphering the keys, which affects sorting and filtering. -* **Standard Listing:** For requests using a forward-slash (`/`) as a prefix terminator or delimiter, results are returned in the lexicographical order of the *encrypted* paths. This often differs from the expected alphabetical order of the decrypted keys. Clients should not rely on ordering in these cases. -* **Exhaustive Listing:** For requests using non-standard prefixes or delimiters (not ending in `/`), the endpoint performs an "exhaustive listing." This filters paths server-side to return results in the correct (decrypted) lexicographical order. +* **Standard Listing:** For requests using a forward-slash (`/`) as a prefix terminator or delimiter, results are returned in the lexicographical order of the *encrypted* object keys. This often differs from the expected alphabetical order of the decrypted keys. Clients should not rely on ordering in these cases. +* **Exhaustive Listing:** For requests using non-standard prefixes or delimiters (not ending in `/`), the endpoint performs an "exhaustive listing." This filters object keys server-side to return results in the correct (decrypted) lexicographical order. **Limitations:** 1. **New Projects:** Projects created after **November 30, 2025**, cannot use exhaustive listing. Requests triggering this behavior will fail with a `501 Not Implemented` error. @@ -21,6 +26,6 @@ When using **Self-Managed Encryption**, bucket paths are end-to-end encrypted. T Users requiring these features should use Storj Managed Encryption. ### Storj Managed Encryption -If your project uses **Storj Managed Encryption**, bucket paths are **encrypted using server-managed keys**. This allows the system to process path names directly, ensuring that Storj's S3 listing endpoints are **fully S3-compatible**. +If your project uses **Storj Managed Encryption**, object keys are **encrypted using server-managed keys**. This allows the system to process object keys directly, ensuring that Storj's S3 listing endpoints are **fully S3-compatible**. Requests can use arbitrary prefixes and/or delimiters. These requests utilize an optimized execution path, offering superior performance compared to Self-Managed Encryption. Storj recommends Storj Managed Encryption for all new projects requiring high performance and strict S3 compatibility.