Upload sample file functionality - #123
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the ability upload a sample file of the dataset through the existing
create_submission_with_uploadfunction or a separate public functionupload_sample_file.Changes
upload.py: Single shared_upload_file(..., is_sample)function used by bothupload_dataset_fileand the new publicupload_sample_file(file_path, submission_id, state_path, show_progress, enable_logging, part_size)sits next to it. No signature changes to the existingupload_dataset_file._upload_base_url(submission_id, is_sample)picks between/uploadsand/submissions/{id}/sampleendpoints. All the sub-functions:_initiate_upload,_get_presigned_part_url, and_complete_uploadtakeis_sampleas arg and build their URLs from it, so the three new endpoints reuse the exact multipart upload flow.UploadStatehas a new fieldisSample: bool = Falseand_state_matchesnow rejects state from the other upload kind. Sample state files default to<filename>.mdc-sample-upload.jsonso sample and dataset resume state never collide.create_submission_with_uploadtakes optionalsample_file_pathandsample_state_path. A missing sample path raisesFileNotFoundErrorup front (before the draft is created or anything uploaded). Also the sample uploads after the dataset archive and before submit, so an interrupted big upload doesn't cause the sample to be re-uploaded.DatasetSubmissionnow declares the API-set, read-onlysampleFileReferenceId(it's already returned in submission payloads).Added both e2e live tests and smaller unit tests
Updated the docs in multiple places