feat(deploy): live SSE progress for deploy_component by kriszyp · Pull Request #531 · HarperFast/harper

kriszyp · 2026-05-14T04:16:14Z

Stacked on #530 — please review that one first; this PR will be retargeted to main after #530 lands.

Summary

Adds a ProgressEmitter and an Accept: text/event-stream path to the operations API for deploy_component.
deployComponent and Application.prepareApplication emit phase events at the extract → install → load → replicate → restart boundaries; install events stream each line of npm install/pnpm/yarn/custom-command stdout & stderr live; the stream terminates with done (operation result) or error.
The CLI requests SSE for deploy_component and renders a real upload progress bar (cli-progress in TTY, periodic text lines in non-TTY) followed by live phase and install events. Stdout still receives the final JSON/YAML result so existing CLI consumers don't break.
Non-SSE callers see zero behavior change: the emitter is undefined on that path and every emission is optional-chained.

First slice of #526.

Where to look

server/serverHelpers/progressEmitter.ts — small pub-sub + the createSSEResponseStream helper. The interesting subtlety is the subscribe() snapshot before iteration (so listeners that unsubscribe themselves during dispatch don't shift indexes), and the swallowed-exception policy so a buggy listener can never break the operation.
server/serverHelpers/serverHandlers.js — the new SSE branch in handlePostRequest. Headers (Content-Type, Cache-Control, X-Accel-Buffering) are set before the stream so reverse proxies and Fastify compression don't buffer the response.
components/operations.js + components/Application.ts — emission points. Application.prepareApplication emits the extract done → install start boundary because Application itself doesn't know which phase comes next; the rest is in deployComponent. installApplication builds an onLine callback that forwards each complete stdout/stderr line as an install SSE event; threaded through all three install code paths (custom command, devEngines packageManager, npm fallback).
components/Application.ts nonInteractiveSpawn — line-buffered onLine parameter so a chunk that splits mid-line doesn't fire a half-event; trailing partial lines flushed on close.
components/packageComponent.ts — getPackagedDirectorySize mirrors streamPackagedDirectory's skip predicates so the CLI's upload bar has a meaningful 100% target.
bin/deployRenderer.ts — owns the upload bar + SSE event rendering. tapUploadStream is an identity Transform that counts bytes for the progress bar; in non-TTY environments it falls back to periodic Uploaded X / ~Y (Z%) lines on each 10% step so logs stay grep-able. renderEvent handles phase, install, error, done.
bin/sseConsumer.ts — SSE record parser (handles split chunks, CRLF, comment lines, multi-line data:).
bin/cliOperations.ts — wires the new streamResponse option in httpRequest and only opts into SSE for operations in the allowlist (SSE_OPERATIONS).

Backward compatibility

The HTTP status stays 200 throughout an SSE deploy and failures are signaled in-band via an error event. The CLI converts an in-band error into a non-zero exit.
External callers that don't send Accept: text/event-stream still receive the buffered JSON/YAML response.
ProgressEmitter is stripped from req before replicateOperation(req) so peer nodes never see a serialized {listeners:[]} object that would otherwise throw TypeError: progress.emit is not a function.

Test plan

npx mocha unitTests/server/serverHelpers/progressEmitter.test.js — 6/6 pass.
npx mocha unitTests/bin/sseConsumer.test.js — 9/9 pass.
npx mocha unitTests/bin/deployRenderer.test.js — 6/6 pass (upload counting in non-TTY mode, endUpload idempotency, phase rendering with no duplicate-start, install line forwarding with manager header / stream prefix / line count, error & phase-error rendering).
npx mocha unitTests/components/packageDirectorySize.test.js — 5/5 pass (sum of sizes, skip_node_modules, .cache/webpack exclusion, empty dir, best-effort on missing path).
npx mocha unitTests/server/serverHelpers/multipartParser.test.js unitTests/bin/multipartBuilder.test.js unitTests/components/packageComponent.test.js unitTests/server/fastifyRoutes/operations.test.js — 38/38 pass (slice 1 + existing operations.test.js suite, unaffected).
End-to-end SSE deploy of a real component — not exercised locally; an integration test that exercises this path is a sensible follow-up.

Follow-ups (intentionally out of scope)

Re-emit per-peer SSE events on the origin stream once the direct-HTTPS replication relay lands (Support for deployments larger than 2GB #524 / harper-pro#146 follow-up).
Upload-byte-count upload events from the multipart parser layer — would let the CLI show a server-side received-bytes bar alongside the client-side sent-bytes bar; currently the CLI's local counter is sufficient signal.

🤖 Generated with Claude Code

claude · 2026-05-14T04:23:22Z

Reviewed; no blockers found.

When the CLI sends `Accept: text/event-stream` on `deploy_component`, the operations API now returns Server-Sent Events instead of a single buffered response. A ProgressEmitter is attached to the operation request and the handler emits `phase` events at the extract → install → load → replicate → restart boundaries; the stream terminates with a `done` event (carrying the operation result) or an `error` event. The CLI parses the stream live, rendering each phase as it happens so multi-minute deploys no longer look hung. Non-SSE callers see no behavior change — the emitter is undefined on that path and every emission is optional-chained. Builds on #530. First slice of #526. Follow-ups: streaming live npm install stdout/stderr as `install` events, and re-emitting per-peer SSE events once the direct-HTTPS replication relay lands in #524 follow-up. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…ress Pulls in slice 1's post-rebase state (the .js→.ts conversion of cliOperations and common_utils on main). Slice 2 conflicts: - bin/cliOperations.ts: kept slice 1's ESM imports + layered in the SSE consumer import (parseSSE, renderDeployProgress). - utility/common_utils.ts: kept slice 2's streamResponse short-circuit inside the new typed response handler. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

… into feat/deploy-component-progress

…eaders access Two bugs flagged by review: 1. Blocker — `req.progress` (a ProgressEmitter) leaked into `replicateOperation(req)`, which serialises the request and forwards it to peer nodes. Functions don't survive that serialisation, so peers received `{ progress: { listeners: [] } }` — a truthy plain object whose `emit()` method throws `TypeError`. Every SSE deploy on a cluster would fail on every peer node. Fix: `delete req.progress` before the replicateOperation call. The local alias `const progress = req.progress` keeps the origin's emitter intact for the post-replicate restart phase events. 2. Test break — the SSE branch in `handlePostRequest` read `req.headers.accept` unguarded. Existing unit tests dispatch through that path with synthetic req shapes that don't set headers; production Fastify requests always do. Optional-chained the access. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Two visibility improvements requested mid-review for #531: 1. **Upload progress bar.** The CLI now computes the uncompressed source- tree total (`getPackagedDirectorySize`, mirrors the same skip_node_modules/skip_symlinks predicates as `streamPackagedDirectory`) and wraps the multipart body in a counting Transform that drives a `cli-progress` SingleBar. In a non-TTY environment (CI logs, redirected output) the renderer falls back to periodic `Uploaded X / ~Y (Z%)` lines logged on each 10% step, so logs stay grep-able. Percentage is against uncompressed source bytes, not wire bytes — the actual upload is gzipped and finishes slightly before 100%, an acceptable trade-off versus walking the tree twice. 2. **Live `npm install` stdout/stderr.** `nonInteractiveSpawn` accepts an optional `onLine` callback; `installApplication` plumbs it through all three install code paths (custom command, devEngines packageManager, npm fallback). Lines are buffered until a newline so a chunk that splits mid-line doesn't fire a half-event; trailing partial lines are flushed on close. The deploy progress emitter fires `install { manager, stream, line }` SSE events; the CLI renderer prefixes each line with `|` (stdout) or `!` (stderr) and shows the manager name once. Install-done summarises with a log-line count so the user knows how much they just saw. Was deferred as a follow-up in the original #531 PR description; brought forward at reviewer request. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

When useSse=true, httpRequest uses streamResponse mode and returns a raw IncomingMessage. The previous fallback branch read response.body which is undefined in that mode (e.g. on a 401 auth failure). Drain the stream chunks manually so error bodies are captured correctly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

heskew

🤖 PR Reviewed by Antigravity AI Coding Assistant.

Overall, the implementation of Server-Sent Events for deploy progress is excellent. The ProgressEmitter is cleanly decoupled and resilient. We have left three inline suggestions regarding split-chunk UTF-8 decoding, resource cleanup on early disconnect, and error parsing robustness.

…, error stringify - parseSSE: use StringDecoder so multi-byte chars split across chunk boundaries decode correctly instead of corrupting (e.g. box-drawing chars, emojis) - createSSEResponseStream: gate writes behind an active flag; listen to stream close/end to unsubscribe the emitter immediately on client disconnect so the subscription doesn't linger for the full operation lifetime - cliOperations SSE error: JSON.stringify fallback when sseError has no .message so the error renders as JSON rather than [object Object] Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

heskew

🤖 Interactive Review Suggestions by Antigravity AI Coding Assistant.

I have converted my previous review findings into interactive GitHub Suggested Changes. You can now apply these fixes with a single click directly inside the PR file viewer!

Well, 3.5 flash is eager (and fast) but not really picking up on needed skills and memory yet...

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

heskew

🤖 PR Reviewed by Gemini CLI

I've identified a minor specification compliance issue regarding multi-line SSE data. See the suggestion below.

Per the SSE spec, if a data value contains newlines each line must be emitted as a separate data: field. A single data: field with embedded newlines is not valid. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

kriszyp · 2026-05-20T11:23:01Z

🤖 PR Reviewed by Gemini CLI

I've identified a minor specification compliance issue regarding multi-line SSE data. See the suggestion below.

Nice. Testing for potentially using Gemini for the automated workflow reviews?

kriszyp · 2026-05-20T16:11:58Z

Pausing this PR — converting to draft.

This work is being rebuilt on top of the design in #641: a replicated hdb_deployment system table that owns the deploy lifecycle, audit trail, and (because Harper's BLOB_CHUNK replication already supports chunked, back-pressured blob transfer) the payload delivery channel.

In the new design:

The ProgressEmitter from this PR becomes one of two subscribers — the second being a DeploymentRecorder that writes the persistent record. SSE remains the live channel for the CLI; get_deployment becomes content-negotiated to serve the same stream to Studio.
_stagedPayloadPath (feat(deploy): stage streamed payloads to a temp file for replication #536) and the direct-HTTPS peer relay (harper-pro#146) both go away — peers read the payload directly from the replicated blob attribute on the row.

This work resumes as part of Slice B in #641 once Slice A (table + blob-backed multipart receive) lands.

— Claude

Wires the ProgressEmitter (resurrected from the paused #531) into the new DeploymentRecorder so every deploy_component lifecycle phase is captured on the row's event_log AND streamable live via SSE. Same content-negotiated branch serves get_deployment, letting Studio (or any client) replay a deploy's history and tail in-flight events through a single endpoint. What's new - DeploymentRecorder subscribes to a ProgressEmitter and coalesces writes: every emit appends to a bounded event_log (200 cap, head+tail retention so the lifecycle spine survives a noisy install); chained puts collapse a burst into one round trip. Emits a `_recorder_finished` sentinel on finish() so SSE tailers terminate cleanly even on crash paths. - deployComponent emits prepare/load/replicate/restart/success phase events around their respective steps. Strips req.progress before replicateOperation so peers see a clean payload. Skips recording entirely on replicated (peer-side) executions — origin owns the canonical row. - An in-memory activeEmitters Map keyed by deployment_id lets get_deployment SSE locate the live emitter and tail it. - handlePostRequest gains a content-negotiated SSE branch (req.headers.accept includes text/event-stream + op in SSE_PROGRESS_OPERATIONS). Prime write on the PassThrough so Fastify starts piping immediately — empirically Fastify buffers a returned Readable until end-of-stream without it, collapsing all intermediate writes into a single flush. - get_deployment with SSE subscribes to the live emitter BEFORE reading the row, then replays the historical event_log and dedupes by timestamp so no event is lost in the stitching gap. A polling fallback resolves the SSE promise even if the deploy disappears without signaling a terminal event. - CLI sends Accept: text/event-stream for deploy_component; consumes the SSE response via parseSSE; renders phase/install/error events through DeployRenderer. - httpRequest gains a streamResponse option that yields the raw IncomingMessage as a Readable instead of buffering — what the SSE consumer needs. Ported from #531 (with the multi-line data spec fix, StringDecoder, and disconnect cleanup already applied earlier in the session): - server/serverHelpers/progressEmitter.ts (+ tests) - bin/sseConsumer.ts (+ tests) - bin/deployRenderer.ts (+ tests) Integration coverage: integrationTests/deploy/deploy-tracking-events.test.ts asserts event_log shape on success, SSE replay+done on get_deployment, and the failure path emits an error event into the log. Refs #641 (Slice B1). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

kriszyp · 2026-05-23T03:23:14Z

Closing in favor of the deployment-tracking redesign tracked in #641.

What landed from this PR via Slice B1 (#657, merged):

ProgressEmitter + createSSEResponseStream helper (server/serverHelpers/progressEmitter.ts)
SSE branch in handlePostRequest with SSE_PROGRESS_OPERATIONS allowlist
deployComponent + Application.prepareApplication phase event emissions
bin/sseConsumer.ts (SSE record parser)
bin/deployRenderer.ts (upload bar + live event rendering)
getPackagedDirectorySize in components/packageComponent.ts
CLI multipart upload + SSE consumption in bin/cliOperations.ts

What's deferred to Slice B2:

Live install stdout/stderr line forwarding from nonInteractiveSpawn to the emitter. The CLI's renderInstall already handles install events (and there's a passing unit test for the rendering side), but no server-side code currently emits them. B2 will touch installApplication for peer-side install support anyway, so the line forwarding rides along there.

The CLI-side bits, the SSE infrastructure, and the per-deploy event_log audit trail are all live on main. This PR is now empty of unmerged substance — closing.

🤖 Generated by Claude

kriszyp requested a review from a team as a code owner May 14, 2026 04:16

claude Bot reviewed May 14, 2026

View reviewed changes

Comment thread components/operations.js

kriszyp force-pushed the feat/deploy-component-progress branch from 676513c to bf19a2a Compare May 14, 2026 10:53

This was referenced May 14, 2026

feat(deploy): stage streamed payloads to a temp file for replication #536

Closed

feat(deploy): direct-HTTPS peer relay for streamed deploy_component HarperFast/harper-pro#146

Closed

Kris Zyp and others added 2 commits May 19, 2026 07:54

Merge remote-tracking branch 'origin/feat/deploy-component-multipart'…

127f14e

… into feat/deploy-component-progress

claude Bot reviewed May 19, 2026

View reviewed changes

Comment thread components/operations.js

Merge remote-tracking branch 'origin/feat/deploy-component-multipart'…

b5177e0

… into feat/deploy-component-progress

claude Bot reviewed May 19, 2026

View reviewed changes

Comment thread components/operations.js

claude Bot reviewed May 19, 2026

View reviewed changes

Comment thread components/operations.js

Base automatically changed from feat/deploy-component-multipart to main May 20, 2026 04:36

Kris Zyp and others added 2 commits May 19, 2026 22:40

claude Bot reviewed May 20, 2026

View reviewed changes

Comment thread bin/cliOperations.ts

heskew reviewed May 20, 2026

View reviewed changes

Comment thread bin/sseConsumer.ts Outdated

Comment thread server/serverHelpers/progressEmitter.ts

Comment thread bin/cliOperations.ts

heskew reviewed May 20, 2026

View reviewed changes

Comment thread bin/sseConsumer.ts

Comment thread bin/sseConsumer.ts Outdated

Comment thread bin/sseConsumer.ts

Comment thread server/serverHelpers/progressEmitter.ts Outdated

Comment thread bin/cliOperations.ts

style: prettier format cliOperations errMsg line

d9740e8

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

heskew reviewed May 20, 2026

View reviewed changes

Comment thread server/serverHelpers/progressEmitter.ts

heskew approved these changes May 20, 2026

View reviewed changes

kriszyp mentioned this pull request May 20, 2026

Deployment tracking, replicated payload delivery, and rollback #641

Open

kriszyp marked this pull request as draft May 20, 2026 16:11

kriszyp mentioned this pull request May 21, 2026

feat(deploy): Slice A — hdb_deployment system table + audit record per deploy #655

Merged

6 tasks

kriszyp mentioned this pull request May 21, 2026

feat(deploy): Slice B1 — ProgressEmitter + SSE on hdb_deployment lifecycle #657

Merged

7 tasks

kriszyp closed this May 23, 2026

This was referenced May 23, 2026

Deployment tracking — Slice B2: peer-side blob reads + install-line streaming #758

Open

feat(deploy): Slice B2 — peer-side blob reads + install-line streaming #760

Draft

Conversation

kriszyp commented May 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Where to look

Backward compatibility

Test plan

Follow-ups (intentionally out of scope)

Uh oh!

Uh oh!

claude Bot commented May 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

heskew left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

heskew left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

heskew left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kriszyp commented May 20, 2026

Uh oh!

kriszyp commented May 20, 2026

Uh oh!

kriszyp commented May 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

kriszyp commented May 14, 2026 •

edited

Loading

claude Bot commented May 14, 2026 •

edited

Loading

heskew left a comment •

edited

Loading