Summary
When a ListObjectsV2 page returned by the backend contains a single object whose key has an empty path segment (a //), multistore fails the entire list request with 503 ServiceUnavailable instead of listing the other (valid) keys. One malformed object key takes out the whole prefix.
object_store's strict Path parser rejects empty segments, and multistore propagates that as a fatal backend error for the page rather than skipping the offending key.
Observed error
Proxy log:
[WARN] multistore::proxy: request failed {
error=backend error: Encountered object with invalid path:
Path "raw//" contained empty path segment,
status=503, s3_code=ServiceUnavailable
}
Client just sees the generic:
An error occurred (ServiceUnavailable) when calling the ListObjectsV2 operation
(reached max retries: 2): Service unavailable
Reproduction
An object genuinely exists in S3 with a // in its key (double slash after raw/):
s3://.../raw//b.e21.BW.f09_g17.SSP245-TSMLT-GAUSS-DELAYED-2045.001/atm/proc/tseries/day_1/....nc
aws s3 ls --no-sign-request against the raw bucket lists it fine (S3 keys are opaque byte strings and // is legal). Listing the same prefix through multistore 503s.
Only prefixes that contain such a key fail; sibling prefixes list normally.
Impact
A single stray upload with a double slash (or a zero-byte folder-marker object ending in //) makes an entire prefix unlistable through the gateway, with an error that gives the operator no hint about which object or why. S3 itself accepts and serves these keys, so the gateway is stricter than the backend it fronts.
Suggested fix
On list, don't let one unparseable key fail the page. Options:
- Skip keys that fail strict
Path::parse, ideally logging the raw key at warn so operators can find and fix it; or
- Parse list keys leniently (
Path::from / from_url_path, which tolerate empty segments) instead of strict parse.
At minimum, surface the offending raw key in the error so it's diagnosable without correlating backend logs.
Environment
- multistore 0.6.3 (features:
azure), via multistore-cf-workers on Cloudflare Workers
- Backend: S3
Summary
When a ListObjectsV2 page returned by the backend contains a single object whose key has an empty path segment (a
//),multistorefails the entire list request with503 ServiceUnavailableinstead of listing the other (valid) keys. One malformed object key takes out the whole prefix.object_store's strictPathparser rejects empty segments, and multistore propagates that as a fatal backend error for the page rather than skipping the offending key.Observed error
Proxy log:
Client just sees the generic:
Reproduction
An object genuinely exists in S3 with a
//in its key (double slash afterraw/):aws s3 ls --no-sign-requestagainst the raw bucket lists it fine (S3 keys are opaque byte strings and//is legal). Listing the same prefix through multistore 503s.Only prefixes that contain such a key fail; sibling prefixes list normally.
Impact
A single stray upload with a double slash (or a zero-byte folder-marker object ending in
//) makes an entire prefix unlistable through the gateway, with an error that gives the operator no hint about which object or why. S3 itself accepts and serves these keys, so the gateway is stricter than the backend it fronts.Suggested fix
On list, don't let one unparseable key fail the page. Options:
Path::parse, ideally logging the raw key atwarnso operators can find and fix it; orPath::from/from_url_path, which tolerate empty segments) instead of strictparse.At minimum, surface the offending raw key in the error so it's diagnosable without correlating backend logs.
Environment
azure), viamultistore-cf-workerson Cloudflare Workers