Skip to content

Filesystem: Add read support for more fsspec-based filesystems #254

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
amotl wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Filesystem: Add read support for more fsspec-based filesystems #254

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
b3c60b3
Router: Don't signal `scheme://bucket-name/file-glob` as deprecated
amotl Jul 16, 2026
78d8145
Tests: Improve parameterized test output
amotl Jul 16, 2026
49a43bc
Tests: Validate initialization of existing remote filesystems
amotl Jul 16, 2026
2e14f64
Core: Raise `NotImplementedError` when using unknown pipeline elements
amotl Jul 17, 2026
7b4e5be
Filesystem: Add read support for more fsspec-based filesystems
amotl Jul 16, 2026
3b9b008
Filesystem: Increase test coverage for error cases and utilities
amotl Jul 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .gitleaksignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,11 @@ omniload/src/telemetry/event.py:generic-api-key:17
omniload/src/testdata/fakebqcredentials.json:private-key:5
docs/supported-sources/shopify.md:generic-api-key:26
docs/supported-sources/smartsheets.md:generic-api-key:38
tests/assets/privatekey.pem:private-key:1
tests/main/filesystem/test_remote_read.py:generic-api-key:43
tests/main/filesystem/test_remote_read.py:generic-api-key:55
tests/main/filesystem/test_remote_read.py:generic-api-key:56
tests/main/filesystem/test_remote_read.py:generic-api-key:58
Comment on lines +6 to +9

@coderabbitai coderabbitai Bot Jul 25, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

Add the failing fixture to the allowlist.

The new entries stop at Line 58, but Gitleaks still reports generic-api-key at tests/main/filesystem/test_remote_read.py Line 59, blocking CI. Add the current fixture location (or rewrite that fake credential fixture so it does not match the rule).

Proposed fix 
 tests/main/filesystem/test_remote_read.py:generic-api-key:58
+tests/main/filesystem/test_remote_read.py:generic-api-key:59
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
tests/main/filesystem/test_remote_read.py:generic-api-key:43
tests/main/filesystem/test_remote_read.py:generic-api-key:55
tests/main/filesystem/test_remote_read.py:generic-api-key:56
tests/main/filesystem/test_remote_read.py:generic-api-key:58
tests/main/filesystem/test_remote_read.py:generic-api-key:43
tests/main/filesystem/test_remote_read.py:generic-api-key:55
tests/main/filesystem/test_remote_read.py:generic-api-key:56
tests/main/filesystem/test_remote_read.py:generic-api-key:58
tests/main/filesystem/test_remote_read.py:generic-api-key:59
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.gitleaksignore around lines 6 - 9, Extend the .gitleaksignore entries for
tests/main/filesystem/test_remote_read.py to include the generic-api-key finding
at line 59, or adjust the fake credential fixture so it no longer matches the
rule while preserving the test’s behavior.

Source: Pipeline failures

tests/saas/test_google_analytics.py:generic-api-key:45
tests/saas/test_google_analytics.py:generic-api-key:51
tests/saas/test_google_analytics.py:generic-api-key:58
2 changes: 2 additions & 0 deletions docs/changelog.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@
## in progress

- Filesystem: Migrated local filesystem access to Apache Arrow.
- Filesystem: Added support for reading from
Dropbox, FTP, HDFS, OCI, OneDrive, OSS, R2, SharePoint, SMB, WebDAV.
Comment on lines +6 to +7

@coderabbitai coderabbitai Bot Jul 25, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Include all connectors added by this PR in the changelog.

The bullet omits Databricks Files, Google Drive, and WebHDFS. Add them or explicitly make the wording non-exhaustive.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/changelog.md` around lines 6 - 7, Update the Filesystem changelog entry
to include Databricks Files, Google Drive, and WebHDFS, or revise its wording to
clearly indicate the list is non-exhaustive while preserving the existing
connector entries.


## 2026/07/16 v0.7.0

Expand Down
9 changes: 2 additions & 7 deletions docs/conf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,13 +73,7 @@

# -- Intersphinx ----------------------------------------------------------

intersphinx_mapping = {
"crash": ("https://cratedb.com/docs/crate/crash/en/latest/", None),
"cloud": ("https://cratedb.com/docs/cloud/en/latest/", None),
"croud": ("https://cratedb.com/docs/cloud/cli/en/latest/", None),
"guide": ("https://cratedb.com/docs/guide/", None),
"influxio": ("https://influxio.readthedocs.io/", None),
}
intersphinx_mapping = {}
linkcheck_ignore = [
r"https://pulse.internetsociety.org/",
r"https://www.g2.com/",
Expand All @@ -101,6 +95,7 @@
r"https://github.com/",
r"https://web.archive.org/",
r"https://images.minimus.io/",
r"https://medium.com/",

@coderabbitai coderabbitai Bot Jul 25, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "Changed lines files:"
git diff --stat || true
echo
if [ -f docs/conf.py ]; then
  echo "docs/conf.py outline:"
  ast-grep outline docs/conf.py || true
  echo
  echo "docs/conf.py relevant lines:"
  nl -ba docs/conf.py | sed -n '80,110p'
fi

echo
echo "Search for medium ignore pattern:"
rg -n "medium\.com" docs/conf.py . || true

Repository: panodata/omniload

Length of output: 1399

🏁 Script executed:

#!/bin/bash
set -euo pipefail

if [ -f docs/conf.py ]; then
  echo "docs/conf.py relevant linkcheck lines:"
  sed -n '77,105p' docs/conf.py
fi

echo
echo "Medium references:"
rg -n "medium\.com|linkcheck_ignore|linkcheck_anchors_ignore_for_url" docs/conf.py . || true

Repository: panodata/omniload

Length of output: 1719

Narrow the Medium link-check ignore pattern.

r"https://medium.com/" skips validation for every Medium URL. Since the only in-repo Medium link is https://medium.com/@pavithrasainath7/mastering-file-access-in-sharepoint-with-oauth-2.0-a-comprehensive-guide-0a6b2d53736a, ignore that exact URL instead of the whole domain.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/conf.py` at line 98, Update the Medium URL entry in the link-check
ignore configuration to match only the specific in-repo article URL, replacing
the broad domain pattern while preserving other ignore entries.

]
linkcheck_anchors_ignore_for_url = [
r"https://developers.zoom.us/",
Expand Down
165 changes: 0 additions & 165 deletions docs/supported-sources/azure-blob-storage.md
View file
Open in desktop

This file was deleted.

49 changes: 0 additions & 49 deletions docs/supported-sources/azure-data-lake-storage.md
View file
Open in desktop

This file was deleted.

Loading
Loading