Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/tricky-socks-join.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@webiny/stdlib": patch
---

Replace `await using` / `AsyncDisposable` on `ReadStreamFactory` with explicit `destroy()` method for bundler compatibility
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@ Current features (each has a `README.md` in its feature folder):
- `PinoLogger` — pino-based Logger implementation. Registered under the `Logger` abstraction from `@webiny/stdlib`. Optional `PinoLoggerConfig` dependency.
- `PinoLoggerFeature` — registers `PinoLogger` in singleton scope.
- `NdJsonReaderTool` — parses NDJSON from a file path, a `Readable` stream, or an in-memory line iterable. Handles multi-line JSON via a `LineAccumulator` that tries newline-join and concatenation before discarding. `parseFile` uses `ReadStreamFactory` for guaranteed stream cleanup. Every yielded row is `{ data, line }` where `line` is the 1-based physical line number; pass `{ fromLine }` to any parse method to skip lines and resume from a checkpoint.
- `ReadStreamFactory` — creates disposable `node:fs` read streams. `create(path, options?)` returns an `IReadStream` that implements `AsyncDisposable`. Use `await using` to guarantee the underlying file handle is released on scope exit (including early generator break or thrown errors). DI token: `"Node/ReadStreamFactory"`.
- `ReadStreamFactory` — creates `node:fs` read streams with explicit cleanup. `create(path, options?)` returns an `IReadStream` with `getStream()` and `destroy()`. Call `destroy()` in a finally block to release the file handle on scope exit (including early generator break or thrown errors). DI token: `"Node/ReadStreamFactory"`.
- `PackageJsonFileTool` — reads, validates, and writes `package.json` files. `read`/`readOrThrow` return a `PackageJsonFile` value object with `readonly path`, `readonly raw` (typed as `PackageJson` from type-fest), and mutation helpers for `dependencies`, `devDependencies`, `peerDependencies`, and `resolutions`. Write methods accept either `(path, data)` or `(file)` — the latter uses the file's own path. Root-level well-known fields are validated with Zod (`.passthrough()` lets unknown fields through).

Note: The `Logger` abstraction lives in `@webiny/stdlib`, not `@webiny/stdlib/node`. Both `ConsoleLogger` and `PinoLogger` register under the same `Logger` token.
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ The package is ESM-only and ships three subpath exports. Because each is a separ
| `PathTool` / `PathToolFeature` | `node:path` wrapper + `resolvePackageFile` for package-relative paths — [docs](src/node/features/PathTool/README.md) |
| `PinoLogger` / `PinoLoggerFeature` | pino-based `Logger` implementation — [docs](src/node/features/PinoLogger/README.md) |
| `NdJsonReaderTool` / `NdJsonReaderToolFeature` | Parse NDJSON from files, streams, or in-memory lines with checkpoint support — [docs](src/node/features/NdJsonReaderTool/README.md) |
| `ReadStreamFactory` / `ReadStreamFactoryFeature` | Disposable `node:fs` read streams via `AsyncDisposable` — [docs](src/node/features/ReadStreamFactory/README.md) |
| `ReadStreamFactory` / `ReadStreamFactoryFeature` | `node:fs` read streams with explicit `destroy()` cleanup — [docs](src/node/features/ReadStreamFactory/README.md) |
| `PackageJsonFileTool` / `PackageJsonFileToolFeature` | Read, validate, mutate, and write `package.json` files — [docs](src/node/features/PackageJsonFileTool/README.md) |
| `HashFolderTool` / `HashFolderToolFeature` | Deterministic SHA-256 hash of a folder's contents — [docs](src/node/features/HashFolderTool/README.md) |
| `ProcessEnvFeature` | `Env` implementation backed by `process.env` — [docs](src/node/features/ProcessEnv/README.md) |
Expand Down
59 changes: 34 additions & 25 deletions __tests__/node/ReadStreamFactory.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -33,41 +33,46 @@ describe("ReadStreamFactory", () => {
const filePath = join(tmpDir, "test.txt");
writeFileSync(filePath, "hello world");

await using rs = factory.create(filePath);
const chunks: Buffer[] = [];
for await (const chunk of rs.getStream()) {
chunks.push(chunk as Buffer);
const rs = factory.create(filePath);
try {
const chunks: Buffer[] = [];
for await (const chunk of rs.getStream()) {
chunks.push(chunk as Buffer);
}
expect(Buffer.concat(chunks).toString()).toBe("hello world");
} finally {
rs.destroy();
}
expect(Buffer.concat(chunks).toString()).toBe("hello world");
});

it("destroys the stream on dispose", async () => {
it("destroys the stream on destroy()", async () => {
const filePath = join(tmpDir, "test.txt");
writeFileSync(filePath, "hello");

let capturedStream;
{
await using rs = factory.create(filePath);
capturedStream = rs.getStream();
// drain it so the stream ends naturally before dispose
const chunks: Buffer[] = [];
for await (const chunk of capturedStream) {
chunks.push(chunk as Buffer);
}
const rs = factory.create(filePath);
const capturedStream = rs.getStream();
const chunks: Buffer[] = [];
for await (const chunk of capturedStream) {
chunks.push(chunk as Buffer);
}
rs.destroy();
expect(capturedStream.destroyed).toBe(true);
});

it("respects ReadStreamOptions (start/end byte range)", async () => {
const filePath = join(tmpDir, "test.txt");
writeFileSync(filePath, "hello world");

await using rs = factory.create(filePath, { start: 6, end: 10 });
const chunks: Buffer[] = [];
for await (const chunk of rs.getStream()) {
chunks.push(chunk as Buffer);
const rs = factory.create(filePath, { start: 6, end: 10 });
try {
const chunks: Buffer[] = [];
for await (const chunk of rs.getStream()) {
chunks.push(chunk as Buffer);
}
expect(Buffer.concat(chunks).toString()).toBe("world");
} finally {
rs.destroy();
}
expect(Buffer.concat(chunks).toString()).toBe("world");
});
});

Expand All @@ -79,12 +84,16 @@ describe("createReadStreamFactory", () => {
const filePath = join(dir, "direct.txt");
writeFileSync(filePath, "direct");
const f = createReadStreamFactory();
await using rs = f.create(filePath);
const chunks: Buffer[] = [];
for await (const chunk of rs.getStream()) {
chunks.push(chunk as Buffer);
const rs = f.create(filePath);
try {
const chunks: Buffer[] = [];
for await (const chunk of rs.getStream()) {
chunks.push(chunk as Buffer);
}
expect(Buffer.concat(chunks).toString()).toBe("direct");
} finally {
rs.destroy();
}
expect(Buffer.concat(chunks).toString()).toBe("direct");
} finally {
rmSync(dir, { recursive: true, force: true });
}
Expand Down
8 changes: 6 additions & 2 deletions src/node/features/NdJsonReaderTool/NdJsonReaderTool.ts
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,12 @@ class NdJsonReaderToolImpl implements NdJsonReaderToolAbstraction.Interface {
path: string,
options?: NdJsonReaderOptions
): AsyncGenerator<NdJsonRow> {
await using rs = this.readStreamFactory.create(path);
yield* this.parseStream(rs.getStream(), options);
const rs = this.readStreamFactory.create(path);
try {
yield* this.parseStream(rs.getStream(), options);
} finally {
rs.destroy();
}
}

public async *parseStream(
Expand Down
29 changes: 19 additions & 10 deletions src/node/features/ReadStreamFactory/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# ReadStreamFactory

Creates disposable `node:fs` read streams that guarantee cleanup via the `AsyncDisposable` protocol. Use `await using` to ensure the underlying file handle is released on scope exit — including early `break` from an async generator loop or thrown errors.
Creates `node:fs` read streams with explicit cleanup. Call `destroy()` to release the underlying file handle when done — including early `break` from an async generator loop or thrown errors (use try/finally).

Node.js only — depends on `node:fs` and `node:stream`.

Expand All @@ -9,16 +9,18 @@ Node.js only — depends on `node:fs` and `node:stream`.
```ts
interface IReadStreamFactory {
/**
* Creates a disposable read stream for the given path.
* Creates a read stream for the given path.
* Mirrors node:fs createReadStream exactly — all native options are supported.
* Use `await using` to guarantee the stream is destroyed on scope exit.
* Call `destroy()` when done to release the file handle.
*/
create(path: PathLike, options?: BufferEncoding | ReadStreamOptions): IReadStream;
}

interface IReadStream extends AsyncDisposable {
interface IReadStream {
/** Returns the underlying Node.js Readable stream. */
getStream(): Readable;
/** Destroys the underlying stream, releasing the file handle. */
destroy(): void;
}
```

Expand All @@ -37,8 +39,12 @@ ReadStreamFactoryFeature.register(container);

const factory = container.resolve(ReadStreamFactory);

await using rs = factory.create("/path/to/file.bin");
const stream = rs.getStream(); // node:stream Readable
const rs = factory.create("/path/to/file.bin");
try {
const stream = rs.getStream(); // node:stream Readable
} finally {
rs.destroy();
}
```

### Without DI
Expand All @@ -48,9 +54,12 @@ import { createReadStreamFactory } from "@webiny/stdlib/node";

const factory = createReadStreamFactory();

await using rs = factory.create("/path/to/file.bin", { start: 0, end: 1023 });
for await (const chunk of rs.getStream()) {
// process chunk
const rs = factory.create("/path/to/file.bin", { start: 0, end: 1023 });
try {
for await (const chunk of rs.getStream()) {
// process chunk
}
} finally {
rs.destroy();
}
// stream.destroy() called automatically here
```
2 changes: 1 addition & 1 deletion src/node/features/ReadStreamFactory/ReadStreamFactory.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ class ReadStreamImpl implements ReadStreamFactoryAbstraction.Stream {
return this.stream;
}

public async [Symbol.asyncDispose](): Promise<void> {
public destroy(): void {
this.stream.destroy();
}
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,18 @@ import { createAbstraction } from "~/common/index.js";
import type { Readable } from "node:stream";
import type { PathLike, ReadStreamOptions } from "node:fs";

export interface IReadStream extends AsyncDisposable {
export interface IReadStream {
/** Returns the underlying Node.js Readable stream. */
getStream(): Readable;
/** Destroys the underlying stream, releasing the file handle. */
destroy(): void;
}

export interface IReadStreamFactory {
/**
* Creates a disposable read stream for the given path.
* Creates a read stream for the given path.
* Mirrors node:fs createReadStream exactly — all native options are supported.
* Use `await using` to guarantee the stream is destroyed on scope exit.
* Call `destroy()` when done to release the file handle.
*/
create(path: PathLike, options?: BufferEncoding | ReadStreamOptions): IReadStream;
}
Expand Down
Loading