Add gzip format for datastore dumps

[luccasmmg]

Summary

• add format=gzip support for /datastore/dump/{resource_id} as gzipped CSV
• export gzip dumps from BigQuery as CSV with compression='GZIP'
• stream multi-shard gzip dumps by decompressing shards, de-duping CSV headers, and recompressing as one gzip file

Tests

• uv run --extra dev pytest tests/test_datastore_dump.py
• uv run --extra dev ruff check datastore/api/endpoints/dump.py datastore/services/dump.py datastore/infrastructure/engines/bigquery/backend.py tests/test_datastore_dump.py

Summary by CodeRabbit

• New Features

  • Added support for exporting datastore resources as gzip-compressed CSV files.
  • Gzip exports support multi-shard results, combine data into a single download, and remove duplicate CSV headers.
  • Downloads use the appropriate .csv.gz filename and content type.

• Bug Fixes

  • Improved handling of oversized exports across supported export formats.

[coderabbitai]

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: a4073beb-c4b4-42ec-ab99-367572b9e046

📥 Commits

Reviewing files that changed from the base of the PR and between e1fcafd and 264f9cb.

📒 Files selected for processing (4)

• datastore/api/endpoints/dump.py
• datastore/infrastructure/engines/bigquery/backend.py
• datastore/services/dump.py
• tests/test_datastore_dump.py

 _____________________________________________
< My middle name? 'Bugfinder Extraordinaire.' >
 ---------------------------------------------
  \
   \   (\__/)
       (•ㅅ•)
       / 　 づ

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch add-gzip-dump-format

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Caution

Failed to replace (edit) comment. This is likely due to insufficient permissions or the comment being deleted.

Error details

putComment timed out