feat(js): streaming hydrate/parse pipeline, CLI, and self-contained core - #80
Closed
ktech643 wants to merge 4 commits into
Closed
feat(js): streaming hydrate/parse pipeline, CLI, and self-contained core#80ktech643 wants to merge 4 commits into
ktech643 wants to merge 4 commits into
Conversation
Node's fetch (undici) sorts header field-names alphabetically before sending, so the echo server observes a different order than the source IR. assert.deepEqual is order-sensitive, causing 016-empty-header to fail despite both headers being transmitted correctly. Sort both expected and server headers by name in the fetch branch of normalizeForEchoServer. Distinct field-name order is not semantically significant, and the change is gated to the fetch adapter so the order-preserving curl path is unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Implement the httpt hydrate/parse pipeline per incubation-js.md, replacing the
scaffold stubs. hydrate() returns synchronously the hybrid triple-stream
signature { resolvedStream, mapStream, bodyStream } (resolved/map are
AsyncIterables, body is a Web ReadableStream) and drives a detached, chunk-safe
single-pass state machine: tag substitution with filters (raw/url/json-*),
index-shift map generation, first-double-newline head/body boundary, the
:httpt-body-type pseudo-header, dynamic header/body injection, and
BodyConflictError. parse() consumes the resolved head into the IR, lifts the
authority (Host / :authority) into host, strips the pseudo-header, and hands off
or materializes the body per type. Wire facade build()/execute() and the CLI
(parseArgs + hydrate/parse/run/emit commands).
Rewrite pipeline.test.js to the new signature: IR for all 16 e2e fixtures,
BodyConflictError, and the byte-exact index-shift map for 003/004 (the only
byte-exact map oracles; 001/002 are hand-authored approximations).
Fixes from an adversarial review of the new code (paths the fixtures do not
exercise):
- facade.build(): await parse + map collection together via Promise.all so a
worker failure (BodyConflictError, unknown filter) can no longer orphan the
map-stream rejection into a process-aborting unhandledRejection.
- hydrate(): apply consumer backpressure when draining an out-of-band
(provided/dynamic) body so large/live payloads stay O(1); end the map channel
before the body drain so map consumers never block on body backpressure.
- parse(): handle leading-colon pseudo-headers and lift :authority to host
instead of mangling it into an empty-name header.
Add facade.test.js regression coverage for build/execute, the clean-rejection
path, provided-body delivery, and :authority lifting.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…fication Close out the remaining verification depth and the one unimplemented feature: - §1.4.4 stream-reference validation: implicit default index 0, AmbiguousStream- ReferenceError (>1 reference with any implicit index), and DuplicateStream- ReferenceError (repeated indices), enforced before hydration output. - Streaming filters (§1.3.2.3): stream-as-base64 / stream-as-utf8 / stream-as-is, with §1.4.5 materialization of streams used inline in the head/body. (No official fixtures exist for these; semantics follow the spec text.) - Expand pipeline.test.js to assert the resolved head for every fixture, the resolved body for inline bodies, and the full index-shift map for all fixtures, plus a CRLF (\r\n\r\n) boundary end-to-end test. Correct four fixture files that were hand-authored and inconsistent with the spec and the byte-exact oracles (003/004): - 001/002 .httpt-map: arithmetic errors in the offsets/lengths (e.g. 001 tag 1 is at index 21, not 20; tag lengths were off by 1-2). Recomputed and verified by hand against the map definition. - 011 .httpt-map: was [] despite the identity template having four tags; now the four correct entries. - 014 .httpt-r: dropped a spurious injected ":httpt-body-type: text" line. Per §1.4.2 text is the default (not injected), matching siblings 002 and 008. Full suite: 80 pass, 0 fail, 0 todo. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the multi-file core/src/hydrate.js (which imported streams/filters/
stream-refs) with one self-contained CommonJS module exporting both hydrate()
and parse(), implementing incubation-js.md §1 + §2 in a single file:
- hydrate() returns synchronously { resolvedStream, mapStream, bodyStream }
from a detached, never-awaited worker with fail-fast propagation to all three.
- resolvedStream / mapStream are async generators (async function*); bodyStream
is a BYOB (type: 'bytes') ReadableStream with backpressure.
- Single-pass, chunk-safe state machine; polymorphic template input; filters
(raw/url/json-*), stream-as-* with §1.4.5 materialization, §1.4.4 reference
validation, dynamic header/body injection, BodyConflictError, CRLF+LF, and the
index-shift map.
- parse() consumes the resolved head into the IR (lifting Host/:authority,
stripping the pseudo-header) and hands off / materializes the body.
Supersedes the standalone js/hydrate.mjs (removed). Full suite: 80 pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Collaborator
Author
|
Closing in favor of a different approach: small, independently reviewable pieces. First installment: #81 (three standalone streaming primitives under js/lib/, one commit per piece). The full pipeline work remains on the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the httpt JavaScript SDK
hydrate/parsepipeline perincubation-js.md(§1 + §2), wires thehttptCLI, and corrects several hand-authored test fixtures.What's included
hydrate()— returns synchronously{ resolvedStream, mapStream, bodyStream }from a detached, never-awaited worker with fail-fast error propagation to all three outputs.resolvedStream/mapStreamare async generators;bodyStreamis a BYOB (type: 'bytes')ReadableStreamwith backpressure. Single-pass, chunk-safe state machine (hold-back across chunk splits, streaming UTF-8), polymorphic template input (string / ReadableStream / AsyncIterable / bytes).raw/url/json-value/json-string/json-key, andstream-as-*with §1.4.5 materialization), §1.4.4 stream-reference validation (ambiguity/uniqueness), dynamic header/body injection,BodyConflictError, CRLF+LF boundaries, and the index-shift map.parse()— consumes the resolved head into the IR (liftsHost/:authority, strips the:httpt-body-typepseudo-header) and hands off / materializes the body per type.core/src/hydrate.jsis a single self-contained module (hydrate+parse); facadebuild/executeand thehttptCLI (hydrate/parse/run/emit) are wired on top.Fixture corrections
Four hand-authored fixtures contradicted the spec and the byte-exact oracles (003/004); corrected with values verified by hand:
001/002.httpt-map— off-by-1/2 offsets and lengths.011.httpt-map— was[]despite the identity template having four tags.014.httpt-r— dropped a spurious injected:httpt-body-type: text(per §1.4.2textis the default, matching 002/008).Testing
Full workspace suite: 80 pass, 0 fail, 0 todo.
🤖 Generated with Claude Code