Skip to content

perf(exports): cap row count and stream body in chunks across xlsx/html/cases endpoints#35

Merged
jamofer merged 1 commit into
masterfrom
perf/exports-streaming-and-cap
May 24, 2026
Merged

perf(exports): cap row count and stream body in chunks across xlsx/html/cases endpoints#35
jamofer merged 1 commit into
masterfrom
perf/exports-streaming-and-cap

Conversation

@jamofer

@jamofer jamofer commented May 24, 2026

Copy link
Copy Markdown
Owner

Summary

  • routers/executions/exports.py wrapped responses in StreamingResponse(BytesIO(...)), which still buffers the entire workbook or HTML payload in one Python bytes object. A 1000-result execution with inline base64 attachments would blow the API process's memory before sending a single byte, with no way for an admin to short-circuit huge exports.
  • Two-part fix:
    • Configurable row cap. New MAX_EXPORT_RESULTS env var (default 5000) gates the three export endpoints (/executions/{id}/export/xlsx, /executions/{id}/export/html, /projects/{id}/cases/export/xlsx). Requests above the cap return 413 with the actual count and the configured ceiling so the caller knows how to narrow the scope (filter by version, environment, suite). The cases endpoint counts via SQL (func.count across TestCase joined to TestSuite) so we don't materialize cases just to reject them.
    • Chunked streaming. Helper _stream_bytes(payload) yields EXPORT_STREAM_CHUNK_BYTES (default 64 KiB) slices via generator. StreamingResponse now consumes the generator, so the kernel send queue drains as the worker iterates instead of holding the whole bytes blob alive across the await boundary.
  • New backend/tests/test_export_size_cap.py monkeypatches the cap to 3 and exercises all three endpoints: 413 above cap, 200 at cap (and the workbook contains exactly N data rows), HTML 413, cases-xlsx 413, plus a chunk-streaming test that asserts the body is delivered in multiple iter_bytes() chunks and starts with the ZIP magic bytes.
  • Full backend suite: 867 passing.

@jamofer jamofer merged commit e6765c0 into master May 24, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant