feat(sync): refine cloud upload raw tags#95
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request Checklist
Please ensure your PR meets the following requirements:
Summary
This PR refines Keystone direct cloud upload raw tags so data-platform uploads include useful production context while omitting internal IDs and noisy sidecar task fields. It also reduces cloud upload progress log volume by logging every 10 uploaded parts and at completion.
Motivation
task.*fields are not useful as data-platform raw tags.Changes
Modified Files
internal/cloud/uploader.go- Throttle[CLOUD-UPLOAD] Progresslogs to every 10 parts and final completion.internal/services/dp_raw_tags.go- Remove internal/raw-ID tags and add contextual tags:sop_slug,sop_version,sop_description,scene,subscene,robot_type,data_collector_operator_id,data_collector_name,order_name, and publicbatch_id.internal/services/sync_worker.go- Load upload context from episode/task/SOP/workstation/robot/data collector/order/batch joins and pass it into raw tag construction.internal/services/sidecar_tags.go- Exclude top-level sidecartaskandtopics_summaryobjects from flattened raw tags.internal/services/dp_raw_tags_test.go- Update raw tag expectations for the new tag set and omitted internal fields.internal/services/sidecar_tags_test.go- Cover filtering of the entire sidecartaskobject.Added Files
Deleted Files
Type of Change
Impact Analysis
Breaking Changes
Cloud upload raw tags no longer include
keystone_episode_id,task_id,factory_id,organization_id, or any sidecartask.*fields. Consumers should use the new descriptive tags and publicbatch_idinstead.Backward Compatibility
Keystone API behavior and upload flow remain compatible. The raw tag payload shape changes for downstream data-platform consumers.
Testing
Test Environment
GOCACHE=/tmp/archebase-go-cachefor sandbox-compatible package testshttptestlisteners ininternal/servicesTest Cases
Manual Testing Steps
Not performed; this change is covered by package-level Go tests.
Test Coverage
Commands run:
Screenshots / Recordings
Not applicable.
Performance Impact
Documentation
Related Issues
Additional Notes
simplify sync progress log) that does not follow Conventional Commits; history was not rewritten.Reviewers
None specified.
Notes for Reviewers
internal/services/dp_raw_tags.go.internal/services/sync_worker.gomatch the intended production metadata semantics.Checklist for Reviewers