Skip to content

feat(sync): show episode upload progress#93

Merged
shark0F0497 merged 2 commits into
mainfrom
feat/sync-progress
Jun 16, 2026
Merged

feat(sync): show episode upload progress#93
shark0F0497 merged 2 commits into
mainfrom
feat/sync-progress

Conversation

@shark0F0497

Copy link
Copy Markdown
Collaborator

Pull Request Checklist

Please ensure your PR meets the following requirements:

  • Code follows the style guidelines
  • Tests pass locally
  • Code is formatted
  • Documentation updated if needed
  • Commit messages follow conventional commits
  • PR description is complete and clear

Summary

This PR adds runtime upload progress support for single episode cloud sync. Keystone now captures OSS multipart upload progress in memory, exposes it through sync status APIs, and documents the design tradeoffs.


Motivation

  • Operators can currently see pending, in_progress, completed, and failed, but not how far an active episode cloud sync has progressed.
  • The progress should reflect real OSS part upload completion without adding DB write pressure or schema changes.

Changes

Modified Files

  • internal/cloud/uploader.go - Adds an optional upload progress callback and reports progress after each successful OSS multipart part.
  • internal/cloud/uploader_test.go - Covers successful progress callbacks and ensures failed parts do not advance progress.
  • internal/services/sync_worker.go - Tracks in-memory per-episode sync progress and clears it when sync reaches a terminal state.
  • internal/services/sync_worker_test.go - Covers progress set/get and immediate cleanup behavior.
  • internal/api/handlers/sync.go - Extends single episode status responses and adds GET /api/v1/sync/episode-statuses?ids=... for batched status polling.
  • internal/api/handlers/sync_test.go - Covers batched status responses, partial missing IDs, and invalid ID validation.

Added Files

  • docs/designs/single-episode-cloud-sync-progress.zh.html - Design document for the runtime-only single episode cloud sync progress feature.

Deleted Files

  • None

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update (documentation changes only)
  • Refactoring (code improvement without functional changes)
  • Performance improvement (code changes that improve performance)
  • Test changes (adding, modifying, or removing tests)

Impact Analysis

Breaking Changes

None

Backward Compatibility

Fully backward compatible. Existing sync endpoints remain available, and progress is returned only as optional runtime metadata.


Testing

Test Environment

  • Local Go test run with GOCACHE=/tmp/go-build-cache
  • Local sandbox blocks tests that require httptest TCP listeners, so targeted package/test runs were used for the affected packages.

Test Cases

  • Unit tests pass locally
  • Integration tests pass locally
  • E2E tests pass (if applicable)
  • Manual testing completed

Manual Testing Steps

Not performed.

Test Coverage

  • New tests added
  • Existing tests updated
  • Coverage maintained or improved

Commands run:

env GOCACHE=/tmp/go-build-cache go test ./internal/cloud
env GOCACHE=/tmp/go-build-cache go test ./internal/services -run 'TestSyncWorkerEpisodeProgressSetGetAndFinish|TestMarkSyncCompleted_WritesExistingCloudFields|TestMarkSyncFailed_NonRetryableClearsNextRetry'
env GOCACHE=/tmp/go-build-cache go test ./internal/api/handlers -run 'TestSync|TestGetSync|TestListSync'

Full ./internal/services and ./internal/api/handlers package runs were not used because this sandbox rejects local TCP listeners used by existing WebSocket tests.


Screenshots / Recordings

Not applicable for backend changes.


Performance Impact

  • Memory usage: Slightly increased while episode syncs are active due to an in-memory progress map.
  • CPU usage: No meaningful change.
  • Throughput: No meaningful change.
  • Lock contention: Minimal; progress map uses a small mutex-protected map and updates once per successful OSS part.

Documentation


Related Issues

  • None

Additional Notes

  • Runtime progress is intentionally not persisted to sync_logs.
  • GET /api/v1/sync/episode-statuses is used instead of /sync/episodes/status to avoid route conflicts with existing :id routes.
  • This assumes the current single Keystone process deployment model; multi-instance deployments would need sticky routing or shared progress storage.

Reviewers


Notes for Reviewers

  • Please focus on the concurrency safety of the SyncWorker progress map.
  • Please verify the new batched status endpoint response shape works for the Synapse polling implementation.
  • Please confirm the route choice is acceptable given existing Gin dynamic routes.

Checklist for Reviewers

  • Code changes are correct and well-implemented
  • Tests are adequate and pass
  • Documentation is updated and accurate
  • No unintended side effects
  • Performance impact is acceptable
  • Backward compatibility maintained (if applicable)

@shark0F0497 shark0F0497 merged commit f2296c9 into main Jun 16, 2026
5 checks passed
@shark0F0497 shark0F0497 deleted the feat/sync-progress branch June 16, 2026 03:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant