Skip to content

[roadmap] Streaming query for large JSON files #4

Description

@Cryptoteep

Goal

Currently query() loads the entire document into memory. For very large JSON files (logs, exports), we should support a streaming query mode that parses incrementally and yields matches as they are found.

Proposed API

import { queryStream } from 'jsoncraft';
import { createReadStream } from 'node:fs';

for await (const match of queryStream(createReadStream('big.json'), '$..events[?(@.level == "error")]')) {
  console.log(match);
}

Implementation notes

  • A streaming JSON parser (SAX-style) would be needed. To preserve the zero-dependency core, this could live in a separate jsoncraft/stream entry point.
  • The query engine already walks the tree recursively; adapting it to emit matches from a streaming parser is the main work.

Acceptance

  • queryStream() works on Node.js readable streams
  • Memory usage stays flat regardless of file size
  • Supports the same JSONPath subset as query()

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesthelp wantedExtra attention is needed

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions