Skip to content

fix(metadata,cli,frontend): drop unnamed columns in the shared builder (#109)#114

Merged
jodeleeuw merged 1 commit into
mainfrom
fix/unnamed-columns-shared-builder
Jun 18, 2026
Merged

fix(metadata,cli,frontend): drop unnamed columns in the shared builder (#109)#114
jodeleeuw merged 1 commit into
mainfrom
fix/unnamed-columns-shared-builder

Conversation

@Mandyx22

Copy link
Copy Markdown
Contributor

What

Re-cut of finding #2 of #109, layered on #103's shared buildPsychDSDataFiles. Supersedes #112 and #113.

R's write.csv (default row.names = TRUE) prepends an unnamed row-index column, so the CSV header starts with a bare comma — an empty-string column name. It can't appear in variableMeasured (Psych-DS requires a name), so generate() drops it from the metadata, but the on-disk CSV kept it → validation failed with CSV_COLUMN_MISSING_FROM_METADATA.

How

The strip lives in the shared data-file path so the CLI and frontend behave identically by construction (no per-caller helper):

  • New exported stripUnnamedColumns helper.
  • generate() strips empty/whitespace-only columns from the parsed data up front, with a single warning instead of per-row spam — keeps variableMeasured clean and standalone library use safe.
  • buildPsychDSDataFiles strips the main table before emitting it: a clean CSV keeps its exact bytes (verbatim mainContent); a file with an unnamed column is re-serialised from the cleaned rows. Sidecars are already clean (they come from generate()'s stripped extraction).
  • Both callers feed parsed mainRows: the CLI (rename-plan and non-plan write paths) and the frontend runGenerate now parseCSV the CSV input, so the written / zipped / in-browser-validated CSV always matches the metadata.

This replaces #112's CLI verbatim-write rewrite and #113's frontend-only normalizeData.ts with one shared-builder implementation.

Testing

  • Full suites green: metadata 148, CLI 127, frontend 37.
  • New coverage: stripUnnamedColumns unit; buildPsychDSDataFiles (dirty CSV re-serialised, clean CSV byte-for-byte, JSON rows); generate() strip (one warning, not per-row); CLI processDirectory on both write paths; frontend runGenerate parse→build path.
  • tsc --noEmit clean; no new eslint any introduced.
  • End-to-end (built CLI) on an R-style fixture ,subject_id,trial_type,trial_index,rt,response: written header drops the empty column and matches variableMeasured; no CSV_COLUMN_MISSING_FROM_METADATA. (The residual time_elapsed error is finding Incorporating Psych-DS validator with the metadata module #1 / fix(metadata): register jsPsych system variables lazily (#109) #110, not on this branch.)

⚠️ Stacked PR

Targets fix/frontend-missing-datafile (#103), not main, because it builds on buildPsychDSDataFiles. Merge #103 first, then re-target this to main.

🤖 Generated with Claude Code

@changeset-bot

changeset-bot Bot commented Jun 16, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 840dc2b

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages
Name Type
@jspsych/metadata Patch
@jspsych/metadata-cli Patch
frontend Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

#109)

R's write.csv(row.names=TRUE) prepends an unnamed row-index column (empty-string
header). It can't appear in variableMeasured (Psych-DS requires a name), so
generate() drops it from the metadata — but the on-disk CSV kept it, failing
validation with CSV_COLUMN_MISSING_FROM_METADATA.

Put the strip in the shared data-file path so the CLI and frontend behave
identically (layered on #103's buildPsychDSDataFiles):

- New exported stripUnnamedColumns helper; generate() strips parsed data up front
  with a single warning (keeps variableMeasured clean + standalone use safe).
- buildPsychDSDataFiles strips the main table: a clean CSV keeps its exact bytes
  (verbatim mainContent), a dirty one is re-serialised from cleaned rows.
- CLI (rename-plan + non-plan paths) and frontend now feed parsed mainRows, so the
  written/zipped/validated CSV always matches the metadata.

Supersedes #112 and #113 (the CLI verbatim-write rewrite and the frontend-only
normalizeData helper) with one shared-builder implementation. Fixes finding #2 of #109.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@jodeleeuw jodeleeuw force-pushed the fix/unnamed-columns-shared-builder branch from 9770ac0 to 840dc2b Compare June 18, 2026 12:37
@jodeleeuw jodeleeuw changed the base branch from fix/frontend-missing-datafile to main June 18, 2026 12:37
@jodeleeuw jodeleeuw merged commit 8edc7c2 into main Jun 18, 2026
2 checks passed
jodeleeuw added a commit that referenced this pull request Jun 18, 2026
…changes

The DataUpload component gained a parseJsonData-based join-key preflight (#115) and a
buildPsychDSDataFiles conversion step (#103/#114) since these tests were written. Expand
the @jspsych/metadata mock (parseJsonData, parseCSV, buildPsychDSDataFiles, deriveFallbackBase,
isValidPsychDSDataFilename, PSYCHDS_IGNORE_*) and the metadata stub (getExtractedArrays/Objects,
getArrayJoinKeys) so the join-key chooser and session-sync paths exercise correctly.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
jodeleeuw added a commit that referenced this pull request Jun 18, 2026
* test(frontend): add unit tests for remaining page/UI components

Covers DataUpload, Variables, Authors, JsonViewer, PageHeader,
PreviewDrawer, Sidebar, ProjectInfo, and AppShell (201 tests total).
Closes #9.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* test(frontend): update DataUpload tests for merged preflight/builder changes

The DataUpload component gained a parseJsonData-based join-key preflight (#115) and a
buildPsychDSDataFiles conversion step (#103/#114) since these tests were written. Expand
the @jspsych/metadata mock (parseJsonData, parseCSV, buildPsychDSDataFiles, deriveFallbackBase,
isValidPsychDSDataFilename, PSYCHDS_IGNORE_*) and the metadata stub (getExtractedArrays/Objects,
getArrayJoinKeys) so the join-key chooser and session-sync paths exercise correctly.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Josh de Leeuw <josh.deleeuw@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants