Absorb MUIO v5.5 changes

[SeaCelo]

Closes #388, closes #389, closes #390, closes #391, closes #392

Summary

• Syncs MUIOGO with upstream MUIO v5.5, following upstream by default and patching only where MUIOGO needs to preserve runtime safety, security, or cross-platform correctness.
• Adds a smoke test harness and sync checklist (UPSTREAM_SYNC.md) so future upstream pulls have repeatable guardrails.
• Adds runtime logging infrastructure with file rotation and a fallback to temp dir when .runtime/ isn't writable.
• Ports the v5.5 diagnostics UI: DataFile model/log viewer, standalone ModelFile page with MathJax equation rendering, scenario toggle-all.
• Takes upstream v5.5 result metadata, unit rules, and pivot updates (Variables.json, Parameters.json, DataModelResult.Class.js, Const.Class.js, Pivot.js).
• Backend files that carry MUIOGO-specific fixes (path validation, session guards, solver resolution, subprocess safety) are kept on the MUIOGO version — not replaced with upstream.

What changed

• Guardrails ([Task] Add guardrails and a smoke harness for the MUIO v5.5 pull #389): UPSTREAM_SYNC.md, smoke.sh, smoke.bat, test_app_smoke.py
• Runtime/backend ([Task] Pull the safe backend/runtime changes from MUIO v5.5 #390): Config.py logging paths, app.py structured logging, DataFileClass.py cleanup (resData consistency, file handle fix, cleanUp robustness), DataFileRoute.py readModelFile/readLogFile routes, model.v.5.4.txt whitespace cleanup
• Diagnostics UI ([Task] Port the DataFile model/log diagnostics from MUIO v5.5 #391): DataFile.js, DataFile.html, ModelFile.js, ModelFile.html, ModelFile.Model.js, DataFile.Model.js, Osemosys.Class.js, Html.Class.js, Routes.Class.js, index.html (MathJax), Config.Model.js
• Results metadata ([Task] Fix result metadata and shadow-price rendering from MUIO v5.5 #392): DataModelResult.Class.js, Const.Class.js, Pivot.js, Variables.json, Parameters.json
• Version: Navbar.html → ver.5.5, Versions.html with MUIOGO-specific changelog

What was intentionally not taken from upstream v5.5

• Home.js, Base.Class.js, CaseRoute.py — upstream didn't change these; MUIOGO's fixes stay as-is
• UploadRoute.py — upstream change is trivial (unused import); MUIOGO's path traversal protection stays
• FileClass.py, OsemosysClass.py — MUIOGO's refactored versions kept; upstream changes are subsumed
• app.config.js — upstream sets SA_IGNORE_CSS = true which is a broad layout toggle; not taken
• osy.pro.css, osy.prov2.css, osemosys.gif, .vscode/launch.json, FileClassCompressed.py, WebAPP/app.log — passive upstream assets not wired into the app

Known residuals

• RYC unit display is blank in results view — upstream v5.5 metadata gap, intentionally left unpatched
• pandas FutureWarning during CBC postprocessing — pre-existing, not from this work

Test plan

• ./scripts/setup.sh --check
• ./scripts/smoke.sh — 5 tests pass
• git ls-files -u — no unmerged paths
• git diff --check — no whitespace errors
• DataFile page loads, model/log buttons present and working
• /#/ModelFile renders equations (MathJax typeset when available, readable fallback when not)
• Scenario toggle-all works, SC_0 stays locked
• CBC demo run end-to-end
• Results/pivot screens load with correct data
• Manual browser validation pass

[github-advanced-security]

CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

[parthdagia05]

Hi @SeaCelo ,

I noticed the CodeQL check is flagging 72 high-severity alerts for "Uncontrolled data used in path expression" in API/Classes/Base/FileClass.py and API/Classes/Case/DataFileClass.py. These are path traversal warnings file paths built from user input without sanitization.

Let me know if you'd like me to open a PR against your branch with the fixes, or if you'd prefer to handle it yourself.

[SeaCelo]

This PR syncs MUIOGO with upstream MUIO v5.5 and adds path traversal protection.

What changed upstream in v5.5

MUIO v5.5 reworked the diagnostics UI (the "DataFile" page), added a new "ModelFile" page for viewing/editing the model text file, simplified how results metadata works (fewer hardcoded parameters/variables), and cleaned up the backend routing and app startup.

We took all of it. The only places we held back were where upstream's changes would undo MUIOGO-specific fixes — cross-platform path handling, the Python version guard, and _muiogo_handler (our custom error handler that replaces MUIO's inline path logic with Config.DATA_STORAGE).

Path traversal guards (new)

CodeQL flagged 72 py/path-injection alerts. The core problem: several routes take a case name or filename from the user and plug it straight into a file path. A malicious user could send ../../etc/passwd instead of a real case name and read or delete files outside the data folder.

The fix has two parts:

1. OsemosysClass constructor — one validate_path call at the top. Every case operation flows through Osemosys(case) or its child DataFile(case), so this single check covers all ~55 alerts in DataFileClass.py.

2. DataFileRoute download/delete routes — five routes build paths from multiple user-supplied pieces (case + run + filename) without going through the constructor. Each gets its own validate_path call before any file access.

validate_path (in Config.py) resolves the full path and checks it stays inside DATA_STORAGE. If not, it raises PermissionError and the request gets a 400.

The 72 CodeQL alerts are false positives. CodeQL tracks user input from source to sink and flags any path that uses it in a file operation. It can't recognise custom sanitizer functions — only built-in ones. We explored adding a CodeQL model file to declare validate_path as a sanitizer barrier, but the YAML-based model approach doesn't resolve local project modules, only installed packages. The alerts remain open, but the runtime protection is real and tested.

To guard against future changes to validate_path weakening the protection without anyone noticing, we added 8 unit tests that verify it blocks the main attack patterns: ../ traversal, absolute paths, encoded traversal, null bytes, and None input.

Also included

• UPSTREAM_SYNC.md documenting what we changed and why
• Smoke tests (tests/test_app_smoke.py) covering the path traversal guards and basic route health
• Shell/batch scripts for running smoke tests on any platform

[SeaCelo]

Hi @SeaCelo ,

I noticed the CodeQL check is flagging 72 high-severity alerts for "Uncontrolled data used in path expression" in API/Classes/Base/FileClass.py and API/Classes/Case/DataFileClass.py. These are path traversal warnings file paths built from user input without sanitization.

Let me know if you'd like me to open a PR against your branch with the fixes, or if you'd prefer to handle it yourself.

@parthdagia05 yes, I'm fixed the traversal by sanitizing, but CodeQL doesn't recognize it for other reasons. I documented and we will dismiss the alert as a false positive.

[brightyorcerf]

This is a massive lift, @SeaCelo!

Stabilizing the v5.5 baseline with a formal UPSTREAM_SYNC playbook is a huge win for the repo’s long-term health.

I'll take a look at the RYC metadata gap and those CBC FutureWarnings. I've already prototyped a lightweight way to fill those specific blanks without diverging from the new upstream baseline, so I'll follow up with a targeted patch to clear the residuals.

Update: #433