Fix BigQuery parquet dump exports

[luccasmmg]

Summary

• use wildcard GCS URIs for parquet EXPORT DATA requests, matching BigQuery SQL requirements
  • Before the parquet uri was missing a "*" but that is required by BQ, even if in the end we cannot concatenate parquet shards together.
• cast native BigQuery JSON columns to strings for parquet exports
  • When i fixed the error above i found another one, which was if the table had json columns the export would break so now we cast those to string
• reject multi-shard parquet exports with 413 because parquet shards cannot be byte-concatenated into one valid file
  • Not a whole lot we can do here, other than maybe doing the concatenation server side, but then that breaks the streaming idea

Tests

• pytest tests/test_datastore_dump.py

Summary by CodeRabbit

• Bug Fixes

  • Improved Parquet exports containing JSON fields for more reliable output.
  • Multi-shard Parquet exports now return a clear 413 error with guidance to choose another format.
  • Single-shard exports continue to redirect to signed download URLs, while CSV and NDJSON multi-shard exports stream efficiently.

• Documentation

  • Clarified export behavior, shard limitations, and download handling for supported formats.

[coderabbitai]

Caution

Review failed

Failed to post review comments.

GitHub was unavailable or timed out while CodeRabbit was posting the review. Please request a new review later if the pull request still needs one. This happened while posting 1 inline comment. Use @coderabbitai full review to retry the review.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: bee6021f-228c-43ab-8585-cd2c5e58fd44

📥 Commits

Reviewing files that changed from the base of the PR and between 6af16af and d988a24.

📒 Files selected for processing (3)

• datastore/api/endpoints/dump.py
• datastore/infrastructure/engines/bigquery/backend.py
• tests/test_datastore_dump.py

🧰 Additional context used

📓 Path-based instructions (7)

datastore/api/endpoints/**/*.py

📄 CodeRabbit inference engine (CLAUDE.md)

datastore/api/endpoints/**/*.py: All endpoint routes must call context.authorize(...) before delegating to services. Authorization must run for every request, and the context's authorize method handles provider-agnostic boundary policy and delegation to the active AuthProvider.
Endpoints must never execute raw SQL or make direct engine calls. All SQL, engine calls, and business logic orchestration must live in datastore/services/. Endpoints only handle request parsing, authorization, and response building.

Files:

• datastore/api/endpoints/dump.py

tests/**/*.py

📄 CodeRabbit inference engine (CLAUDE.md)

Test structure must mirror production code: tests/auth/<name>/ mirrors datastore/auth/<name>/; tests/engines/<name>/ mirrors datastore/infrastructure/engines/<name>/. Use tests/conftest.py for shared fixtures (FakeCKAN, InMemoryCache, TestClient).

Files:

• tests/test_datastore_dump.py

datastore/{services,infrastructure,core,auth}/**/*.py

📄 CodeRabbit inference engine (CLAUDE.md)

Only datastore/api/ and datastore/main.py may import from fastapi or starlette. All FastAPI/Starlette dependencies (Request, Response, StreamingResponse, middleware, status codes, routing, Depends) must be confined to these locations.

Files:

• datastore/infrastructure/engines/bigquery/backend.py

datastore/{services,infrastructure,auth}/**/*.py

📄 CodeRabbit inference engine (CLAUDE.md)

Only datastore/schemas/ and datastore/core/config.py may import from pydantic or pydantic_settings. Engines, services, and auth providers must pass plain dicts, tuples, and dataclasses instead of Pydantic models.

Files:

• datastore/infrastructure/engines/bigquery/backend.py

datastore/infrastructure/engines/**/*.py

📄 CodeRabbit inference engine (CLAUDE.md)

Engines must return a lazy row iterator of tuples, never list[dict]. This keeps peak memory ≈ 1 row regardless of result size for streaming search responses.

Files:

• datastore/infrastructure/engines/bigquery/backend.py

datastore/infrastructure/engines/**/backend.py

📄 CodeRabbit inference engine (CLAUDE.md)

datastore/infrastructure/engines/**/backend.py: Storage engines are plugins loaded via DATASTORE_ENGINE environment variable. Adding a new engine requires creating a folder under datastore/infrastructure/engines/<name>/ with __init__.py exporting Backend = <ConcreteClass>, a backend.py implementing DatastoreBackend, and optional client.py/lib.py for backend-specific helpers. Include allowed_functions.txt for SQL function allow-list.
Implement healthcheck() on every storage backend. The /ready endpoint calls both rw and ro engine healthchecks and returns 503 Service Unavailable if either fails.
Store Frictionless schema and unique_key as JSON in the backend's native table metadata (e.g., BigQuery table description OPTION) for round-trip without a separate metadata table. On datastore_info, return the exact schema that was supplied at create time.

Files:

• datastore/infrastructure/engines/bigquery/backend.py

datastore/infrastructure/engines/bigquery/backend.py

📄 CodeRabbit inference engine (CLAUDE.md)

For GET /datastore/dump/{resource_id}, read table.modified from the backend, compute a cache-key revision, and check GCS for cached shards before running EXPORT DATA. After a successful export, GC older revisions in the same (rid, fmt) prefix.

Files:

• datastore/infrastructure/engines/bigquery/backend.py

📝 Walkthrough

Walkthrough

Parquet exports now use wildcard GCS URIs, convert JSON columns to strings, and reject multi-shard downloads with PayloadTooLargeError. Endpoint documentation describes the resulting shard-specific response behavior.

Changes

Parquet export handling

Layer / File(s)	Summary
Wildcard export and JSON conversion datastore/infrastructure/engines/bigquery/backend.py, tests/test_datastore_dump.py	Parquet exports use wildcard URIs, JSON fields are converted with TO_JSON_STRING, and SQL/URI generation is covered by tests.
Multi-shard Parquet rejection datastore/infrastructure/engines/bigquery/backend.py, datastore/api/endpoints/dump.py, tests/test_datastore_dump.py	Multiple Parquet shards raise PayloadTooLargeError, with endpoint documentation updated for shard-specific behavior.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant DumpEndpoint
  participant BigQueryBackend
  participant BigQuery
  participant GCS
  DumpEndpoint->>BigQueryBackend: Request Parquet dump
  BigQueryBackend->>BigQuery: Export to wildcard Parquet URI
  BigQuery->>GCS: Write Parquet shards
  BigQueryBackend->>GCS: List exported blobs
  GCS-->>BigQueryBackend: Return shard list
  BigQueryBackend-->>DumpEndpoint: Return export or raise PayloadTooLargeError

Loading

Possibly related PRs

• datopian/datastore#2: Updates the same BigQuery dump path with wildcard Parquet exports, shard rejection, and JSON conversion.
• datopian/datastore#3: Also modifies BigQueryBackend.dump and related export handling.

Suggested reviewers: sagargg

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main change: fixing BigQuery parquet dump exports.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix-bigquery-parquet-dump

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[sagargg]

Hi @luccasmmg Thanks for the PR. Do you have any idea how we can allow users to download large Parquet files?