Skip to content

feat(sync): refine cloud upload raw tags#95

Merged
shark0F0497 merged 3 commits into
mainfrom
feat/tag-optimize
Jun 17, 2026
Merged

feat(sync): refine cloud upload raw tags#95
shark0F0497 merged 3 commits into
mainfrom
feat/tag-optimize

Conversation

@shark0F0497

Copy link
Copy Markdown
Collaborator

Pull Request Checklist

Please ensure your PR meets the following requirements:

  • Code follows the style guidelines
  • Tests pass locally
  • Code is formatted
  • Documentation updated if needed
  • Commit messages follow conventional commits
  • PR description is complete and clear

Summary

This PR refines Keystone direct cloud upload raw tags so data-platform uploads include useful production context while omitting internal IDs and noisy sidecar task fields. It also reduces cloud upload progress log volume by logging every 10 uploaded parts and at completion.


Motivation

  • Cloud raw tags should carry meaningful searchable production context such as SOP, scene, robot type, collector, order, and batch ID.
  • Internal Keystone IDs and broad sidecar task.* fields are not useful as data-platform raw tags.
  • Multipart cloud upload progress logs were too frequent for large MCAP files.

Changes

Modified Files

  • internal/cloud/uploader.go - Throttle [CLOUD-UPLOAD] Progress logs to every 10 parts and final completion.
  • internal/services/dp_raw_tags.go - Remove internal/raw-ID tags and add contextual tags: sop_slug, sop_version, sop_description, scene, subscene, robot_type, data_collector_operator_id, data_collector_name, order_name, and public batch_id.
  • internal/services/sync_worker.go - Load upload context from episode/task/SOP/workstation/robot/data collector/order/batch joins and pass it into raw tag construction.
  • internal/services/sidecar_tags.go - Exclude top-level sidecar task and topics_summary objects from flattened raw tags.
  • internal/services/dp_raw_tags_test.go - Update raw tag expectations for the new tag set and omitted internal fields.
  • internal/services/sidecar_tags_test.go - Cover filtering of the entire sidecar task object.

Added Files

  • None

Deleted Files

  • None

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update (documentation changes only)
  • Refactoring (code improvement without functional changes)
  • Performance improvement (code changes that improve performance)
  • Test changes (adding, modifying, or removing tests)

Impact Analysis

Breaking Changes

Cloud upload raw tags no longer include keystone_episode_id, task_id, factory_id, organization_id, or any sidecar task.* fields. Consumers should use the new descriptive tags and public batch_id instead.

Backward Compatibility

Keystone API behavior and upload flow remain compatible. The raw tag payload shape changes for downstream data-platform consumers.


Testing

Test Environment

  • Local Go test environment
  • GOCACHE=/tmp/archebase-go-cache for sandbox-compatible package tests
  • Elevated local test run for httptest listeners in internal/services

Test Cases

  • Unit tests pass locally
  • Integration tests pass locally
  • E2E tests pass (if applicable)
  • Manual testing completed

Manual Testing Steps

Not performed; this change is covered by package-level Go tests.

Test Coverage

  • New tests added
  • Existing tests updated
  • Coverage maintained or improved

Commands run:

env GOCACHE=/tmp/archebase-go-cache go test ./internal/services -run 'TestBuildDPDirectRawTags|TestFlattenSidecar'
env GOCACHE=/tmp/archebase-go-cache go test ./internal/cloud
go test ./internal/services
git diff --check main...HEAD

Screenshots / Recordings

Not applicable.


Performance Impact

  • Memory usage: No change
  • CPU usage: No change
  • Throughput: No change
  • Lock contention: No change
  • Logging volume: Decreased for multipart cloud uploads

Documentation


Related Issues

  • None

Additional Notes

  • The branch includes one existing commit message (simplify sync progress log) that does not follow Conventional Commits; history was not rewritten.

Reviewers

None specified.


Notes for Reviewers

  • Please focus on the raw tag schema change in internal/services/dp_raw_tags.go.
  • Verify the new context joins in internal/services/sync_worker.go match the intended production metadata semantics.

Checklist for Reviewers

  • Code changes are correct and well-implemented
  • Tests are adequate and pass
  • Documentation is updated and accurate
  • No unintended side effects
  • Performance impact is acceptable
  • Backward compatibility maintained (if applicable)

@shark0F0497 shark0F0497 merged commit fdf0a00 into main Jun 17, 2026
5 checks passed
@shark0F0497 shark0F0497 deleted the feat/tag-optimize branch June 17, 2026 07:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant