Skip to content

Add bulk QA progress SSE#90

Merged
shark0F0497 merged 3 commits into
mainfrom
feat/qc-with-json
Jun 11, 2026
Merged

Add bulk QA progress SSE#90
shark0F0497 merged 3 commits into
mainfrom
feat/qc-with-json

Conversation

@shark0F0497

Copy link
Copy Markdown
Collaborator

Pull Request Checklist

  • Code follows the style guidelines
  • Tests pass locally
  • Code is formatted
  • Documentation updated if needed
  • Commit messages follow conventional commits
  • PR description is complete and clear

Summary

This PR adds a second built-in episode QA check for empty recordings and adds realtime bulk QA run progress over SSE for Data Ops bulk QA.


Motivation

  • Operators need QA to catch recordings that have a valid-looking MCAP file but no actual recorded messages/topics.
  • Bulk QA currently runs asynchronously without a durable run snapshot or realtime progress, so the UI cannot tell when a submitted batch has finished without refreshing.

Changes

Modified Files

  • [internal/api/handlers/data_ops.go](internal/api/handlers/data_ops.go) - wires Data Ops bulk action dependencies and bulk run routes.
  • [internal/api/handlers/data_ops_bulk.go](internal/api/handlers/data_ops_bulk.go) - changes bulk QA submission to create a tracked run and process episodes with progress accounting.
  • [internal/api/handlers/data_ops_test.go](internal/api/handlers/data_ops_test.go) - adds coverage for bulk run creation, active-run conflict, progress counts, stale-run interruption, and SSE snapshots.
  • [internal/server/server.go](internal/server/server.go) - interrupts stale queued/running bulk QA runs on service startup.
  • [docs/designs/data-ops-bulk-actions-api.zh.html](docs/designs/data-ops-bulk-actions-api.zh.html) - documents the run snapshot and SSE progress design.

Added Files

  • [internal/api/handlers/data_ops_bulk_run.go](internal/api/handlers/data_ops_bulk_run.go) - implements bulk QA run persistence, current-run lookup, run snapshot API, and SSE streaming.
  • [internal/storage/database/migrations/000006_bulk_runs.up.sql](internal/storage/database/migrations/000006_bulk_runs.up.sql) - creates the bulk run table.
  • [internal/storage/database/migrations/000006_bulk_runs.down.sql](internal/storage/database/migrations/000006_bulk_runs.down.sql) - drops the bulk run table.

Deleted Files

  • None.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update (documentation changes only)
  • Refactoring (code improvement without functional changes)
  • Performance improvement (code changes that improve performance)
  • Test changes (adding, modifying, or removing tests)

Impact Analysis

Breaking Changes

None.

Backward Compatibility

Fully backward compatible. Existing single-episode QA and bulk sync behavior remain unchanged. Bulk QA responses now include a run snapshot so clients can track progress.


Testing

Test Environment

Local Go test environment with GOCACHE=/tmp/go-cache.

Test Cases

  • Unit tests pass locally
  • Integration tests pass locally
  • E2E tests pass (if applicable)
  • Manual testing completed

Commands run:

git diff --check
GOCACHE=/tmp/go-cache go test ./internal/api/handlers -run 'Test(ParseDataOps|DataOps|PreviewBulk|BulkRun|StreamBulkRun|InterruptActive|CurrentBulkRun|GetBulkRun)' -count=1 -v
GOCACHE=/tmp/go-cache go test -race ./internal/api/handlers -run 'Test(BulkRun|StreamBulkRun|InterruptActive|CurrentBulkRun|GetBulkRun)' -count=1
GOCACHE=/tmp/go-cache go test ./internal/server ./internal/storage/database -count=1
GOCACHE=/tmp/go-cache go test ./internal/api/handlers -count=1

Manual Testing Steps

Not performed in browser from this repo.

Test Coverage

  • New tests added
  • Existing tests updated
  • Coverage maintained or improved

Screenshots / Recordings

N/A.


Performance Impact

  • Memory usage: Slight increase while an active bulk QA run is tracked and streamed.
  • CPU usage: No material change outside active bulk QA processing.
  • Throughput: No material change.
  • Lock contention: No material change expected.

Documentation

Generated Swagger files were not refreshed locally because the local swag setup was unavailable.


Related Issues

  • None.

Additional Notes

  • Only bulk QA gets realtime progress in this PR. Bulk sync keeps the existing accepted/background behavior.
  • The active run model is intentionally scoped to one global bulk_qa run at a time.

Reviewers

N/A.


Notes for Reviewers

  • Please review [internal/api/handlers/data_ops_bulk_run.go](internal/api/handlers/data_ops_bulk_run.go) for run lifecycle, SSE streaming, and reconnect behavior.
  • Please review the migration and startup interruption behavior for stale queued/running runs.

Checklist for Reviewers

  • Code changes are correct and well-implemented
  • Tests are adequate and pass
  • Documentation is updated and accurate
  • No unintended side effects
  • Performance impact is acceptable
  • Backward compatibility maintained (if applicable)

@shark0F0497 shark0F0497 merged commit 475884d into main Jun 11, 2026
5 checks passed
@shark0F0497 shark0F0497 deleted the feat/qc-with-json branch June 11, 2026 09:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant