Skip to content

Fix s3 listing bug#147

Closed
CannonLock wants to merge 1 commit into
mainfrom
fix-url-decoding
Closed

Fix s3 listing bug#147
CannonLock wants to merge 1 commit into
mainfrom
fix-url-decoding

Conversation

@CannonLock
Copy link
Copy Markdown

@CannonLock CannonLock commented Mar 11, 2026

Fixes an issue with AmazonS3List::Results where it does not url decode the results from S3 despite requesting them url encoded.

This was entirely vibe coded, but the fix is pretty small. I had it give me a breakdown of what went wrong and it makes sense to me.

This is a interesting item though. We are url encoding the object names because they could have XML breaking values in them. But whatever is reading these values is not expecting them to be url encoded.

I think until we can have the upstream be able to handle url encoded object names we ought to just accept that the XML can be broken with wild object names.

Closes #146

Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves S3-backed directory listing behavior when S3 returns URL-encoded keys/prefixes (via encoding-type=url), ensuring downstream consumers (e.g., WebDAV PROPFIND responses) use decoded, correct path components.

Changes:

  • Decode URL-encoded <Key> and <CommonPrefixes><Prefix> values when parsing S3 ListBucket XML results.
  • Add a unit test covering decoding behavior in S3 list parsing.
  • Extend the S3 integration test setup and test script to validate PROPFIND href paths for a populated pseudo-directory.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
test/s3-test.sh Adds an integration check that PROPFIND responses contain expected decoded href paths and do not include %2F-encoded directory prefixes.
test/s3-setup.sh Uploads two additional objects under testfolder/ to support the new PROPFIND integration assertions.
test/s3_tests.cc Adds a unit test verifying URL-decoding of object keys and common prefixes from ListBucket XML.
src/S3Commands.hh Exposes a static helper for parsing ListBucket results to enable direct unit testing.
src/S3Commands.cc Implements URL-decoding and factors ListBucket XML parsing into the new helper, decoding keys and common prefixes.

Comment thread src/S3Commands.hh
Fixes an issue with AmazonS3List::Results where it does not url decode the results from S3 despite requesting them url encoded.

Closes #146
Comment thread src/S3Commands.cc
@jhiemstrawisc
Copy link
Copy Markdown
Member

Superseded by PR #151

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PROPFIND broken on S3 backed Origin

3 participants