Skip to content

Add opt-in multipart file upload batching - #36

Open
greglum wants to merge 1 commit into
SEMOSS:mainfrom
greglum:feature/batch-file-uploads
Open

Add opt-in multipart file upload batching#36
greglum wants to merge 1 commit into
SEMOSS:mainfrom
greglum:feature/batch-file-uploads

Conversation

@greglum

@greglum greglum commented Jul 16, 2026

Copy link
Copy Markdown

Summary

  • add an opt-in batch_size parameter to ServerClient.upload_files while preserving one-file requests by default
  • send repeated multipart file fields in bounded groups and validate ordered server responses
  • forward upload_batch_size through VectorEngine.addDocument
  • close all file handles on successful and failed requests

Validation

  • six mocked unit tests covering default, four-file, remainder, validation, HTTP failure, malformed response, ordering, and handle closure
  • focused Ruff checks on changed code; the two existing bare-except diagnostics elsewhere in server_client.py remain unchanged
  • package source and wheel build
  • live four-file SEMOSS insight upload canary: one multipart request and four ordered results

Related work

SEMOSS/Semoss#2759 is the server-side counterpart to this change: this PR batches the client-to-server upload transport, while that PR batches the embedding-engine calls made during FAISS ingestion. The two changes are independent, touch different layers of the same ingestion flow, and neither depends on the other.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant