feat(data): add streaming JSONL and loader-output quality scanner - #416
Open
Liu14159 wants to merge 1 commit into
Open
feat(data): add streaming JSONL and loader-output quality scanner#416Liu14159 wants to merge 1 commit into
Liu14159 wants to merge 1 commit into
Conversation
Liu14159
force-pushed
the
feat/streaming-jsonl-quality-scanner
branch
from
July 30, 2026 07:49
ed2bba2 to
2f2555d
Compare
…clusionAI#213) Add scan_jsonl_stream and scan_loader_output to areno/api/data.py for streaming quality inspection of JSONL datasets without loading the full file into memory. Reports blank lines, JSON parse failures, non-object records, and missing required keys with bounded, redacted previews. Register 'areno scan-dataset' CLI command in diagnostics with file/stdin input, --required-keys, --max-issues, --json output, and --loader-fn support. Exit code 1 when issues are found. Add CPU tests covering success, malformed input, boundary values, redaction, large-file bounded memory, and all CLI paths.
Liu14159
force-pushed
the
feat/streaming-jsonl-quality-scanner
branch
from
August 4, 2026 05:27
30ecfb4 to
f6666b5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add a streaming JSONL and loader-output quality scanner that inspects dataset files line-by-line without loading the full dataset into memory. Reports blank lines, JSON parse errors, non-object records, and schema surprises (missing required keys) with bounded, redacted previews of bad entries. Enables users to validate data quality before starting expensive training runs.
Closes #213.
What changed
scan_jsonl_stream(stream file/stdin line-by-line),scan_loader_output(scan custom loader records),ScanIssue/ScanReportdataclasses,format_scan_report(human-readable + JSON output), redaction (_redact_preview) and truncation (_PREVIEW_MAX_CHARS=200) helpersareno scan-datasetCLI command with--required-keys,--max-issues,--json,--loader-fnoptions, supporting file and stdin input; exit code 1 when issues foundscan-datasetin the command table--helplistingDesign
max_issuescaps storedScanIssueentries; excess counted intruncated_issues.blank(whitespace-only line),json_error(JSON parse failure),non_object(valid JSON but not a dict),schema(dict missing required keys). Each issue records 1-based line number, detail string, and truncated/redacted raw preview._redact_previewmasks values of common secret keys (api_key,token,secret,password, etc.) in previews so sensitive training data never appears in logs or CLI output.required_keys=()(no schema check),max_issues=50. The command is purely additive — no existing CLI command, trainer, or config is modified.json,re). No external database, no sandbox, no model initialization.--loader-fnimports a user.pyfile, callsload_training_dataset(path, default_loader=...), and scans the returned records withscan_loader_output. Receives a no-opdefault_loaderso scanning doesn't pull in the HF datasets stack.format_scan_report(report, json_output=False)for human-readable summary + per-issue lines;json_output=Truefor structured JSON viaScanReport.to_dict().Testing
30 CPU tests across 5 test classes:
ScanJsonlStreamTest(13) — clean file, blank lines, JSON errors, non-object, schema check, error recovery, preview truncation, secret redaction, max_issues bounds (including 0 and negative), stdin input, source trackingScanLoaderOutputTest(4) — clean records, non-dict, schema missing keys, source labelFormatScanReportTest(3) — JSON output validity, human-readable status, issue line displayScanDatasetCliTest(9) — clean/bad file exit codes, required-keys, json output, stdin, max-issues,--helplisting, loader-fn scan, loader-fn missing fileBoundedMemoryTest(1) — 5000-line file with mixed content, verifies counts and issues cap