Fix Pyodide analysis pipeline (dataKey routing, MEMFS, MNE 1.x API)

.llms/pyodide-fix-plan.md

@@ -0,0 +1,213 @@
+# Pyodide Analysis Workflow — Fix Plan
+
+**Review date:** 2026-03-15
+**Reviewer:** Claude Code (plan-eng-review)
+**Base branch:** `main`
+**Coworker branch:** `teonbrooks/pyodide-upgrade` (fetch from `git@github.com:teonbrooks/BrainWaves.git`)
+
+---
+
+## Context
+
+Analysis workflow (EEG data → Pyodide → plots) was completely broken due to stacked bugs.
+`teonbrooks/pyodide-upgrade` has QA'd Pyodide loading in-browser and addressed infrastructure issues.
+This plan documents what that branch resolves and what still needs to be applied on top.
+
+---
+
+## What `teonbrooks/pyodide-upgrade` has already fixed
+
+| Bug | Status |
+|-----|--------|
+| Pyodide from npm (0.29.3), ESM worker, `import { loadPyodide }` | ✅ Done |
+| Local HTTP asset server (port 17173) to serve .whl files past Vite SPA fallback | ✅ Done |
+| Vite `serve-pyodide-assets` dev middleware | ✅ Done |
+| `patches.py` Pyolite relative import — removed `patch_matplotlib` entirely; MPLBACKEND set in worker init | ✅ Done |
+| `csvArray` scope — now passed in message context, not on `window` | ✅ Done |
+| Inverted error toast (`if (results && !error)`) — fixed | ✅ Done |
+| `plotKey` routing in worker + `pyodideMessageEpic` — plots routed directly to Redux slots | ✅ Done |
+| SVG output for all plots (PSD, Topo, ERP, test) | ✅ Done |
+| `plotTopoMap` restored in `index.ts` (SVG version) | ✅ Done |
+| Directory renamed: `utils/pyodide/` → `utils/webworker/` | ✅ Done |
+| New IPC handlers: `fs:storePyodideImageSvg`, `fs:storePyodideImagePng` | ✅ Done |
+
+---
+
+## What still needs to be fixed (apply on top of teonbrooks branch)
+
+### CRITICAL — analysis data pipeline still broken
+
+#### 1. `requestEpochsInfo` / `requestChannelInfo` still return `void`
+**File:** `src/renderer/utils/webworker/index.ts`
+**Epics affected:** `getEpochsInfoEpic`, `getChannelInfoEpic`
+
+`worker.postMessage()` returns `void`. Both functions `await worker.postMessage(...)`, getting `undefined`.
+The epic then calls `.map()` on `undefined` → throws.
+
+**Fix approach:** Extend the `plotKey` pattern with a `dataKey` for data-returning calls.
+Worker sends `{ results, dataKey }`. `pyodideMessageEpic` routes `dataKey: 'epochsInfo'` →
+`SetEpochInfo`, `dataKey: 'channelInfo'` → `SetChannelInfo`.
+Update `loadEpochsEpic` and `loadCleanedEpochsEpic` to use `dataKey` in the message.
+
+```
+loadEpochsEpic flow (fixed):
+  LoadEpochs action
+  → loadCSV (fire-and-forget, no return needed)
+  → filterIIR (fire-and-forget)
+  → epochEvents (fire-and-forget)
+  → worker.postMessage({ data: 'get_epochs_info(raw_epochs)', dataKey: 'epochsInfo' })
+     └─ worker responds { results: [...], dataKey: 'epochsInfo' }
+        └─ pyodideMessageEpic routes → SetEpochInfo
+  → loadEpochsEpic emits EMPTY (like plot epics)
+```
+
+#### 2. `launchEpic` race condition
+**File:** `src/renderer/epics/pyodideEpics.ts`
+
+`loadPatches`, `applyPatches`, `loadUtils` all fire simultaneously in `tap()`.
+Even with patches.py simplified, `applyPatches()` can still NameError if patches.py hasn't
+finished running.
+
+**Fix:** Make `launchEpic` dispatch `SetPyodideWorker` only after `loadUtils` signals ready via
+the existing `plotKey: 'ready'` mechanism. The epic should wait for `SetWorkerReady` before
+completing, not fire-and-forget in tap.
+
+OR: simpler — sequence the init via `mergeMap(async (worker) => { ... await run each step ... })`.
+The `plotKey: 'ready'` approach is also fine if the app gates analysis on `workerReady` state.
+
+#### 3. `loadTopoEpic` still uses `plotTestPlot`, not `plotTopoMap`
+**File:** `src/renderer/epics/pyodideEpics.ts:239`
+The `plotTopoMap` function exists in `index.ts` (SVG version). The epic still calls `plotTestPlot`.
+One-line fix: `tap(() => plotTopoMap(state$.value.pyodide.worker!))`.
+
+### SIGNIFICANT — bugs that cause silent failures
+
+#### 4. `saveEpochs` missing closing parenthesis
+**File:** `src/renderer/utils/webworker/index.ts` (saveEpochs function)
+
+Generated Python: `raw_epochs.save("/path/to/file"` — SyntaxError, no closing `)`.
+Fix: add `)` before the closing backtick of the template literal.
+
+#### 5. `loadCleanedEpochs` passes OS paths to WASM filesystem
+**File:** `src/renderer/utils/webworker/index.ts` (loadCleanedEpochs function)
+
+`read_epochs(file)` receives `/Users/dano/BrainWaves_Workspaces/.../subj-cleaned-epo.fif`.
+Pyodide's WASM filesystem has no access to host OS paths → `FileNotFoundError`.
+
+**Fix:** Read `.fif` file bytes via IPC (`window.electronAPI.readFile(path)` returning `Uint8Array`),
+write to Pyodide MEMFS via `pyodide.FS.writeFile('/tmp/subj.fif', bytes)`, pass `/tmp/subj.fif`
+to `read_epochs`. Requires:
+- New IPC handler: `fs:readFileAsBytes` in `src/main/index.ts`
+- New helper in `index.ts`: `writeEpochsToMemfs(worker, filePaths)`
+- Update `loadCleanedEpochs` to use MEMFS paths
+
+#### 6. Channel index 0 falsy bug
+**File:** `src/renderer/epics/pyodideEpics.ts:268`
+
+```typescript
+if (index) { return index; }  // 0 is falsy → first channel always wrong
+```
+Fix: `if (index !== null)`.
+
+#### 7. `renderAnalyzeButton` drop threshold inverted
+**File:** `src/renderer/components/CleanComponent/index.tsx:131`
+
+```typescript
+if (drop && typeof drop === 'number' && drop >= 2) {  // shows button when data is bad
+```
+Fix: show button when `epochsInfo !== null` (any loaded data). User sees the drop % and
+can judge themselves.
+
+### MINOR — API compatibility
+
+#### 8. `raw.plot_psd()` deprecated MNE API
+**File:** `src/renderer/utils/webworker/index.ts` (plotPSD function)
+
+`raw.plot_psd(fmin=1, fmax=30, show=False)` was deprecated in MNE 1.0 and removed in 1.2+.
+Pyodide 0.29.3 ships MNE 1.x. Fix: `raw.compute_psd(fmin=1, fmax=30).plot(show=False)`.
+
+---
+
+## `SetWorkerReady` action — verify it exists
+**File:** `src/renderer/actions/pyodideActions.ts`
+
+`pyodideMessageEpic` references `PyodideActions.SetWorkerReady()`. Verify this action is defined.
+If not, add it. Consider adding a `workerReady: boolean` field to `PyodideStateType` and gate
+analysis dispatch on it.
+
+---
+
+## Data flow diagram (target state)
+
+```
+User clicks "Load Dataset" (CleanComponent)
+  ↓
+LoadEpochs action
+  ↓
+loadEpochsEpic:
+  loadCSV({ data: 'raw = load_data()', csvArray }) → fire-and-forget
+  filterIIR({ data: 'raw.filter(...)' })            → fire-and-forget
+  epochEvents({ data: 'raw_epochs = Epochs(...)' })  → fire-and-forget
+  worker.postMessage({ data: 'get_epochs_info(raw_epochs)', dataKey: 'epochsInfo' })
+  mergeMap(() => EMPTY)
+  ↓
+pyodideMessageEpic receives { results: [...], dataKey: 'epochsInfo' }
+  → SetEpochInfo([...]) → Redux state → CleanComponent re-renders stats
+
+User clicks "Analyze Dataset" (always available when epochsInfo !== null)
+  ↓
+LoadCleanedEpochs action
+  ↓
+loadCleanedEpochsEpic:
+  writeEpochsToMemfs(worker, filePaths)  → IPC read bytes → pyodide.FS.writeFile
+  loadCleanedEpochs({ data: 'clean_epochs = concatenate_epochs(...)' })
+  dispatch GetEpochsInfo, GetChannelInfo, LoadTopo
+  ↓
+Parallel:
+  getEpochsInfoEpic → { dataKey: 'epochsInfo' } → SetEpochInfo
+  getChannelInfoEpic → { dataKey: 'channelInfo' } → SetChannelInfo
+  loadTopoEpic → plotTopoMap({ plotKey: 'topo' }) → worker → SetTopoPlot(svg)
+
+User selects channel → LoadERP(channelName)
+  ↓
+loadERPEpic: index lookup (if (index !== null)) → plotERP({ plotKey: 'erp' })
+  → worker → SetERPPlot(svg) → AnalyzeComponent renders via PyodidePlotWidget
+```
+
+---
+
+## Files to change
+
+| File | Changes |
+|------|---------|
+| `src/renderer/utils/webworker/index.ts` | saveEpochs closing paren; plotPSD API; loadCleanedEpochs MEMFS |
+| `src/renderer/epics/pyodideEpics.ts` | dataKey routing for epochs/channel info; launch sequencing; loadTopoEpic; channel index 0 |
+| `src/renderer/components/CleanComponent/index.tsx` | renderAnalyzeButton condition |
+| `src/renderer/actions/pyodideActions.ts` | Verify/add SetWorkerReady, ReceiveError(string) |
+| `src/main/index.ts` | Add `fs:readFileAsBytes` IPC handler |
+| `src/renderer/utils/webworker/webworker.js` | Add `dataKey` passthrough (similar to existing `plotKey`) |
+
+---
+
+## Tests to write
+
+- `__tests__/pyodideWorker.test.ts` — dataKey routing (mock worker, verify SetEpochInfo dispatched)
+- `__tests__/pyodideEpics.test.ts` — launch sequencing, loadEpochsEpic chain, channel index 0 fix
+- `__tests__/pyodideIndex.test.ts` — Python code string generation: saveEpochs has correct syntax, plotPSD uses correct MNE API
+
+---
+
+## NOT in scope
+
+- Worker health monitoring / auto-restart on crash
+- Cancellation of in-progress analysis
+- Progress callbacks
+- MNE/pandas version upgrades beyond what Pyodide 0.29.3 ships
+
+---
+
+## TODO (deferred)
+
+- Pyodide integration test suite: load real Pyodide + MNE in Node/jsdom, run known-good CSV
+  through the analysis pipeline, verify output shape. High value but non-trivial setup.
+  Blocked by: Jest/Vitest compatibility with WASM workers.

eslint.config.mjs

@@ -24,6 +24,7 @@ export default [
       'coverage/**',
       '.worktrees/**',
       'src/renderer/utils/webworker/src/**',
+      'src/renderer/utils/pyodide/**',
       '**/*.css.d.ts',
       '**/*.scss.d.ts',
     ],

src/main/index.ts

@@ -5,7 +5,7 @@
  * All Node.js / filesystem / shell operations the renderer needs
  * are handled here via ipcMain handlers and exposed via the preload.
  */
 import { app, BrowserWindow, ipcMain, dialog, shell, session, protocol, net } from 'electron';
 import path from 'path';
 import fs from 'fs';
 import { pathToFileURL } from 'url';
@@ -30,8 +30,8 @@
   {
     scheme: 'pyodide',
     privileges: {
       standard: true,    // treat like http for URL parsing / resolution
       secure: true,      // counts as a secure origin (needed for WASM, SAB)
       supportFetchAPI: true, // allow fetch() from renderer and worker contexts
       corsEnabled: true, // no CORS errors when Pyodide fetches its own assets
     },
@@ -113,7 +113,9 @@
   try {
     return fs
       .readdirSync(workspaces)
-      .filter((workspace) => workspace !== '.DS_Store' && workspace !== 'Test_Plot');
+      .filter(
+        (workspace) => workspace !== '.DS_Store' && workspace !== 'Test_Plot'
+      );
   } catch (e: unknown) {
     if ((e as NodeJS.ErrnoException).code === 'ENOENT') {
       mkdirPathSync(workspaces);
@@ -235,10 +237,15 @@
     const dir = path.join(getWorkspaceDir(title), 'Results', 'Images');
     mkdirPathSync(dir);
     return new Promise<void>((resolve, reject) => {
-      fs.writeFile(path.join(dir, `${imageTitle}.svg`), svgContent, 'utf8', (err) => {
-        if (err) reject(err);
-        else resolve();
-      });
+      fs.writeFile(
+        path.join(dir, `${imageTitle}.svg`),
+        svgContent,
+        'utf8',
+        (err) => {
+          if (err) reject(err);
+          else resolve();
+        }
+      );
     });
   }
 );
@@ -339,6 +346,12 @@
   });
 });
 
+ipcMain.handle('fs:readFileAsBytes', (_event, filePath: string) => {
+  // Returns a Uint8Array (Buffer extends Uint8Array) for binary files like .fif.
+  // Transferred via IPC structured clone — arrives as Uint8Array in the renderer.
+  return fs.readFileSync(filePath);
+});
+
 // EEG streaming — main process holds write streams for performance
 ipcMain.handle(
   'eeg:createWriteStream',

src/preload/index.ts

@@ -93,7 +93,12 @@ contextBridge.exposeInMainWorld('electronAPI', {
     imageTitle: string,
     svgContent: string
   ): Promise<void> =>
-    ipcRenderer.invoke('fs:storePyodideImageSvg', title, imageTitle, svgContent),
+    ipcRenderer.invoke(
+      'fs:storePyodideImageSvg',
+      title,
+      imageTitle,
+      svgContent
+    ),
 
   storePyodideImagePng: (
     title: string,
@@ -127,6 +132,9 @@ contextBridge.exposeInMainWorld('electronAPI', {
   readFiles: (filePathsArray: string[]): Promise<string[]> =>
     ipcRenderer.invoke('fs:readFiles', filePathsArray),
 
+  readFileAsBytes: (filePath: string): Promise<Uint8Array> =>
+    ipcRenderer.invoke('fs:readFileAsBytes', filePath),
+
   // ------------------------------------------------------------------
   // EEG streaming — main process holds the write stream for performance
   // ------------------------------------------------------------------

src/renderer/actions/deviceActions.ts

@@ -1,6 +1,10 @@
 import { createAction } from '@reduxjs/toolkit';
 import { ActionType } from 'typesafe-actions';
-import { DEVICES, DEVICE_AVAILABILITY, CONNECTION_STATUS } from '../constants/constants';
+import {
+  DEVICES,
+  DEVICE_AVAILABILITY,
+  CONNECTION_STATUS,
+} from '../constants/constants';
 import { Device, DeviceInfo } from '../constants/interfaces';
 
 // -------------------------------------------------------------------------

src/renderer/components/CleanComponent/index.tsx

@@ -99,7 +99,7 @@
   }
 
   handleSidebarToggle() {
     this.setState({ isSidebarVisible: !this.state.isSidebarVisible });
   }
 
   renderEpochLabels() {
@@ -123,18 +123,12 @@
 
   renderAnalyzeButton() {
     const { epochsInfo } = this.props;
-    if (!isNil(epochsInfo)) {
-      const drop = epochsInfo.find(
-        (infoObj) => infoObj.name === 'Drop Percentage'
-      )?.value;
-
-      if (drop && typeof drop === 'number' && drop >= 2) {
-        return (
-          <Link to="/analyze">
-            <Button variant="default">Analyze Dataset</Button>
-          </Link>
-        );
-      }
+    if (!isNil(epochsInfo) && epochsInfo.length > 0) {
+      return (
+        <Link to="/analyze">
+          <Button variant="default">Analyze Dataset</Button>
+        </Link>
+      );
     }
     return null;
   }

src/renderer/components/CollectComponent/ConnectModal.tsx

@@ -105,7 +105,9 @@ export default class ConnectModal extends Component<Props, State> {
             tabIndex={0}
             className="flex items-center gap-2 py-2 cursor-pointer text-lg"
             onClick={() => this.setState({ selectedDevice: device })}
-            onKeyDown={(e) => e.key === 'Enter' && this.setState({ selectedDevice: device })}
+            onKeyDown={(e) =>
+              e.key === 'Enter' && this.setState({ selectedDevice: device })
+            }
           >
             <span>{this.state.selectedDevice === device ? '✓' : '○'}</span>
             <span>{ConnectModal.getDeviceName(device)}</span>

src/renderer/components/CollectComponent/index.tsx

@@ -102,7 +102,9 @@ export default class Collect extends Component<Props, State> {
           open={this.state.isConnectModalOpen}
           onClose={this.handleConnectModalClose}
           connectedDevice={this.props.connectedDevice}
-          signalQualityObservable={this.props.signalQualityObservable ?? undefined}
+          signalQualityObservable={
+            this.props.signalQualityObservable ?? undefined
+          }
           deviceType={this.props.deviceType}
           deviceAvailability={this.props.deviceAvailability}
           connectionStatus={this.props.connectionStatus}

src/renderer/components/DesignComponent/CustomDesignComponent.tsx

@@ -231,7 +231,9 @@ export default class CustomDesign extends Component<DesignProps, State> {
               </div>
               <div className="grid grid-cols-3 gap-2.5 self-end justify-self-end">
                 <div>
-                  <label htmlFor="trial-order" className="block text-sm mb-1">Order</label>
+                  <label htmlFor="trial-order" className="block text-sm mb-1">
+                    Order
+                  </label>
                   <select
                     id="trial-order"
                     className="border border-gray-300 rounded px-2 py-1"
@@ -271,7 +273,10 @@ export default class CustomDesign extends Component<DesignProps, State> {
                   />
                 </div>
                 <div>
-                  <label htmlFor="nb-practice-trials" className="block text-sm mb-1">
+                  <label
+                    htmlFor="nb-practice-trials"
+                    className="block text-sm mb-1"
+                  >
                     Total practice trials
                   </label>
                   <input

src/renderer/components/HomeComponent/OverviewComponent.tsx

@@ -16,7 +16,8 @@ interface Props {
 
 // Generic curried enum type guard
 function isEnum<T extends object>(en: T) {
-  return (val: unknown): val is T[keyof T] => Object.values(en).includes(val as T[keyof T]);
+  return (val: unknown): val is T[keyof T] =>
+    Object.values(en).includes(val as T[keyof T]);
 }
 
 const OverviewComponent: React.FC<Props> = ({

src/renderer/components/HomeComponent/index.tsx

@@ -325,7 +325,9 @@ export default class Home extends Component<Props, State> {
                 disabled={!this.props.isWorkerReady}
                 onClick={() => this.props.PyodideActions.LoadTopo()}
               >
-                {this.props.isWorkerReady ? 'Generate Plot' : 'Loading libraries…'}
+                {this.props.isWorkerReady
+                  ? 'Generate Plot'
+                  : 'Loading libraries…'}
               </Button>
             </div>
             <div>