Filesystem: Add read support for more fsspec-based filesystems by amotl · Pull Request #254 · panodata/omniload

amotl · 2026-07-25T17:28:48Z

About

A few more filesystem wrappers based on fsspec: Databricks files, Dropbox, FTP, Google Drive, HDFS, OCI, OneDrive, OSS, R2, SharePoint, SMB, WebDAV, WebHDFS.
Filesystem documentation improvements across the board.

Documentation

Preview: https://omniload--254.org.readthedocs.build/supported-sources/filesystem.html#supported-filesystems

References

- Add support for Databricks files, Dropbox, FTP, Google Drive, HDFS, OCI, OneDrive, OSS, R2, SharePoint, SMB, WebDAV, WebHDFS. - Filesystem documentation improvements across the board.

coderabbitai · 2026-07-25T17:28:54Z

Warning

Review limit reached

@amotl, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 48 seconds

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: ae59b4bd-bc73-4e03-aa41-409e633a13ad

📥 Commits

Reviewing files that changed from the base of the PR and between 7b24bb8 and 7b4e5be.

⛔ Files ignored due to path filters (1)

tests/assets/privatekey.pem is excluded by !**/*.pem

📒 Files selected for processing (73)

.gitleaksignore
docs/changelog.md
docs/conf.py
docs/supported-sources/azure-blob-storage.md
docs/supported-sources/azure-data-lake-storage.md
docs/supported-sources/azure-storage.md
docs/supported-sources/cbor.md
docs/supported-sources/databricks-files.md
docs/supported-sources/databricks-warehouse.md
docs/supported-sources/dropbox.md
docs/supported-sources/filesystem.md
docs/supported-sources/ftp.md
docs/supported-sources/google-cloud-storage.md
docs/supported-sources/google-drive.md
docs/supported-sources/hdfs.md
docs/supported-sources/msgpack.md
docs/supported-sources/nextcloud.md
docs/supported-sources/ods.md
docs/supported-sources/onedrive.md
docs/supported-sources/oracle-oci.md
docs/supported-sources/oss.md
docs/supported-sources/r2.md
docs/supported-sources/s3.md
docs/supported-sources/sftp.md
docs/supported-sources/sharepoint.md
docs/supported-sources/smb.md
docs/supported-sources/webdav.md
docs/supported-sources/webhdfs.md
docs/supported-sources/xlsx.md
docs/supported-sources/xml.md
docs/supported-sources/yaml.md
pyproject.toml
src/dlt_filesystem/error.py
src/dlt_filesystem/source/adapter.py
src/dlt_filesystem/source/api.py
src/dlt_filesystem/source/base.py
src/dlt_filesystem/source/core.py
src/dlt_filesystem/source/error.py
src/dlt_filesystem/source/fsspec/databricks.py
src/dlt_filesystem/source/fsspec/dropbox.py
src/dlt_filesystem/source/fsspec/ftp.py
src/dlt_filesystem/source/fsspec/gdrive.py
src/dlt_filesystem/source/fsspec/hdfs.py
src/dlt_filesystem/source/fsspec/local.py
src/dlt_filesystem/source/fsspec/oci.py
src/dlt_filesystem/source/fsspec/onedrive.py
src/dlt_filesystem/source/fsspec/oss.py
src/dlt_filesystem/source/fsspec/r2.py
src/dlt_filesystem/source/fsspec/sharepoint.py
src/dlt_filesystem/source/fsspec/smb.py
src/dlt_filesystem/source/fsspec/webdav.py
src/dlt_filesystem/source/fsspec/webhdfs.py
src/dlt_filesystem/source/impl/remote.py
src/dlt_filesystem/source/model.py
src/dlt_filesystem/source/router.py
src/dlt_filesystem/util/fsspec.py
src/dlt_filesystem/util/python.py
src/dlt_filesystem/util/web.py
src/omniload/core/factory.py
src/omniload/core/registry.py
tests/assets/privatekey-fingerprint.txt
tests/dlt_filesystem/format/test_bson.py
tests/dlt_filesystem/format/test_cbor.py
tests/dlt_filesystem/format/test_msgpack.py
tests/dlt_filesystem/format/test_xml.py
tests/dlt_filesystem/format/test_yaml.py
tests/dlt_filesystem/test_source_hints.py
tests/dlt_filesystem/test_source_incremental.py
tests/dlt_filesystem/test_source_local.py
tests/dlt_filesystem/test_source_remote.py
tests/main/filesystem/test_file_incremental.py
tests/main/filesystem/test_local_read.py
tests/main/filesystem/test_remote_read.py

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch filesystems-read-202607b

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

read-the-docs-community · 2026-07-25T17:29:52Z

Documentation build overview

📚 omniload | 🛠️ Build #33758063 | 📁 Comparing 7b4e5be against latest (7b24bb8)

🔍 Preview build

30 files changed · + 16 added · ± 11 modified · - 3 deleted

+ Added

± Modified

- Deleted

codecov · 2026-07-25T17:38:27Z

Codecov Report

❌ Patch coverage is 90.90909% with 48 lines in your changes missing coverage. Please review.
✅ Project coverage is 57.21%. Comparing base (7b24bb8) to head (7b4e5be).

Files with missing lines	Patch %	Lines
src/dlt_filesystem/util/python.py	74.35%	10 Missing ⚠️
src/dlt_filesystem/util/fsspec.py	76.92%	9 Missing ⚠️
src/dlt_filesystem/source/fsspec/webdav.py	84.84%	5 Missing ⚠️
src/dlt_filesystem/source/impl/remote.py	87.17%	5 Missing ⚠️
src/dlt_filesystem/source/core.py	92.85%	2 Missing ⚠️
src/dlt_filesystem/source/fsspec/hdfs.py	92.30%	2 Missing ⚠️
src/dlt_filesystem/source/model.py	97.14%	2 Missing ⚠️
src/dlt_filesystem/source/router.py	84.61%	2 Missing ⚠️
src/dlt_filesystem/source/fsspec/databricks.py	95.45%	1 Missing ⚠️
src/dlt_filesystem/source/fsspec/dropbox.py	94.44%	1 Missing ⚠️
... and 9 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #254      +/-   ##
==========================================
+ Coverage   55.71%   57.21%   +1.50%     
==========================================
  Files         210      227      +17     
  Lines        9894    10444     +550     
==========================================
+ Hits         5512     5976     +464     
- Misses       4382     4468      +86

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

amotl added 5 commits July 25, 2026 19:23

Router: Don't signal scheme://bucket-name/file-glob as deprecated

b3c60b3

Tests: Improve parameterized test output

78d8145

Tests: Validate initialization of existing remote filesystems

49a43bc

Core: Raise NotImplementedError when using unknown pipeline elements

2e14f64

Filesystem: Add read support for more fsspec-based filesystems

7b4e5be

- Add support for Databricks files, Dropbox, FTP, Google Drive, HDFS, OCI, OneDrive, OSS, R2, SharePoint, SMB, WebDAV, WebHDFS. - Filesystem documentation improvements across the board.

amotl mentioned this pull request Jul 25, 2026

Filesystem: Add read support for more fsspec-based filesystems (workbench) #236

Closed

amotl added the coderabbit-review Tell CodeRabbit to review the patch. label Jul 25, 2026

amotl marked this pull request as ready for review July 25, 2026 17:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Filesystem: Add read support for more fsspec-based filesystems#254

Filesystem: Add read support for more fsspec-based filesystems#254
amotl wants to merge 5 commits into
mainfrom
filesystems-read-202607b

amotl commented Jul 25, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Jul 25, 2026 •

edited

Loading

Review limit reached

Uh oh!

read-the-docs-community Bot commented Jul 25, 2026

Uh oh!

codecov Bot commented Jul 25, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

amotl commented Jul 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

About

Documentation

References

Uh oh!

coderabbitai Bot commented Jul 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review limit reached

Uh oh!

read-the-docs-community Bot commented Jul 25, 2026

Documentation build overview

Uh oh!

codecov Bot commented Jul 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

amotl commented Jul 25, 2026 •

edited

Loading

coderabbitai Bot commented Jul 25, 2026 •

edited

Loading

codecov Bot commented Jul 25, 2026 •

edited

Loading