Skip to content

feat(data): add streaming JSONL and loader-output quality scanner - #416

Open
Liu14159 wants to merge 1 commit into
inclusionAI:mainfrom
Liu14159:feat/streaming-jsonl-quality-scanner
Open

feat(data): add streaming JSONL and loader-output quality scanner#416
Liu14159 wants to merge 1 commit into
inclusionAI:mainfrom
Liu14159:feat/streaming-jsonl-quality-scanner

Conversation

@Liu14159

Copy link
Copy Markdown

Summary

Add a streaming JSONL and loader-output quality scanner that inspects dataset files line-by-line without loading the full dataset into memory. Reports blank lines, JSON parse errors, non-object records, and schema surprises (missing required keys) with bounded, redacted previews of bad entries. Enables users to validate data quality before starting expensive training runs.

Closes #213.

What changed

  • areno/api/data.pyscan_jsonl_stream (stream file/stdin line-by-line), scan_loader_output (scan custom loader records), ScanIssue/ScanReport dataclasses, format_scan_report (human-readable + JSON output), redaction (_redact_preview) and truncation (_PREVIEW_MAX_CHARS=200) helpers
  • areno/cli/diagnostics.pyareno scan-dataset CLI command with --required-keys, --max-issues, --json, --loader-fn options, supporting file and stdin input; exit code 1 when issues found
  • areno/cli/main.py — register scan-dataset in the command table
  • tests/test_scan_dataset_cpu.py — 30 tests covering core scan logic, malformed input, boundary values, redaction, bounded memory (5000-line simulation), CLI paths (file/stdin/json/loader-fn/missing-file), and --help listing

Design

  • Streaming, bounded memory: File is read line-by-line; memory does not scale with file size. max_issues caps stored ScanIssue entries; excess counted in truncated_issues.
  • Issue categories: blank (whitespace-only line), json_error (JSON parse failure), non_object (valid JSON but not a dict), schema (dict missing required keys). Each issue records 1-based line number, detail string, and truncated/redacted raw preview.
  • Redaction: _redact_preview masks values of common secret keys (api_key, token, secret, password, etc.) in previews so sensitive training data never appears in logs or CLI output.
  • Defaults are backward compatible: required_keys=() (no schema check), max_issues=50. The command is purely additive — no existing CLI command, trainer, or config is modified.
  • No new dependencies: Uses only stdlib (json, re). No external database, no sandbox, no model initialization.
  • Loader-fn path: --loader-fn imports a user .py file, calls load_training_dataset(path, default_loader=...), and scans the returned records with scan_loader_output. Receives a no-op default_loader so scanning doesn't pull in the HF datasets stack.
  • Output formats: format_scan_report(report, json_output=False) for human-readable summary + per-issue lines; json_output=True for structured JSON via ScanReport.to_dict().

Testing

30 CPU tests across 5 test classes:

  • ScanJsonlStreamTest (13) — clean file, blank lines, JSON errors, non-object, schema check, error recovery, preview truncation, secret redaction, max_issues bounds (including 0 and negative), stdin input, source tracking
  • ScanLoaderOutputTest (4) — clean records, non-dict, schema missing keys, source label
  • FormatScanReportTest (3) — JSON output validity, human-readable status, issue line display
  • ScanDatasetCliTest (9) — clean/bad file exit codes, required-keys, json output, stdin, max-issues, --help listing, loader-fn scan, loader-fn missing file
  • BoundedMemoryTest (1) — 5000-line file with mixed content, verifies counts and issues cap

@Liu14159
Liu14159 force-pushed the feat/streaming-jsonl-quality-scanner branch from ed2bba2 to 2f2555d Compare July 30, 2026 07:49
…clusionAI#213)

Add scan_jsonl_stream and scan_loader_output to areno/api/data.py for
streaming quality inspection of JSONL datasets without loading the full
file into memory. Reports blank lines, JSON parse failures, non-object
records, and missing required keys with bounded, redacted previews.

Register 'areno scan-dataset' CLI command in diagnostics with file/stdin
input, --required-keys, --max-issues, --json output, and --loader-fn
support. Exit code 1 when issues are found.

Add CPU tests covering success, malformed input, boundary values,
redaction, large-file bounded memory, and all CLI paths.
@Liu14159
Liu14159 force-pushed the feat/streaming-jsonl-quality-scanner branch from 30ecfb4 to f6666b5 Compare August 4, 2026 05:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a streaming JSONL and loader-output quality scanner

1 participant