Address #15; Lay Validation Foundation; Light Cleanup Work by olivia-banks · Pull Request #17 · EpiForeSITE/epiworldPythonStreamlit

olivia-banks · 2026-03-18T20:53:21Z

Address #15 and lay foundation for validation; light cleanup work. Running list of changes:

We're introducing some dependencies here, but they are already baked into pyodide, which is what stlite uses under the hood, so there's no extra overhead!

Discussion for the FRW may be found here.

olivia-banks · 2026-03-21T03:57:29Z

There's still some more cleanup I have to do on the YAML side.

Copilot

Pull request overview

This PR restructures the Streamlit app into a src/epicc/ package layout and introduces a format read/write + validation foundation (Pydantic-backed) for loading parameters and model/config YAML/XLSX in a uniform way, with accompanying unit tests.

Changes:

Migrates the app from top-level app.py + utils/ + models/ into src/epicc/ (new __main__.py, config/, formats/, model/, models/, utils/).
Adds a new FRW (format read/write) layer for YAML/XLSX plus Pydantic validation and template generation.
Adds pytest-based tests for the formats and model loader, and updates dependencies (PyYAML → ruamel-yaml, adds Pydantic).

Reviewed changes

Copilot reviewed 35 out of 43 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
uv.lock	Updates locked dependencies (adds Pydantic + ruamel-yaml; removes PyYAML).
pyproject.toml	Updates runtime deps (adds `pydantic[email]`, `ruamel-yaml`; removes PyYAML).
.gitignore	Adds ignores for common dev/build artifacts.
app.py	Removes old top-level Streamlit entrypoint.
config/app.yaml	Removes old app config file (migrated).
config/global_defaults.yaml	Removes old global defaults file (migrated).
config/paths.yaml	Removes old model path config (no longer used).
models/tb_isolation.py	Removes old dynamic python-file model module.
models/measles_outbreak.py	Removes old dynamic python-file model module.
utils/model_loader.py	Removes old model discovery/dynamic import loader.
utils/parameter_loader.py	Removes old YAML/XLSX parameter loader utilities.
utils/parameter_ui.py	Removes old Streamlit parameter UI renderer.
utils/section_renderer.py	Removes old section renderer.
src/epicc/main.py	New Streamlit app entrypoint using the new package structure.
src/epicc/web/sidebar.css	Adds CSS for sidebar styling.
src/epicc/config/init.py	Adds config loader (resource-based) and exports `CONFIG`.
src/epicc/config/schema.py	Adds Pydantic config schema models.
src/epicc/config/default.yaml	Adds default app config YAML.
src/epicc/formats/init.py	Adds format registry, validation wrapper, and `read_from_format`.
src/epicc/formats/base.py	Introduces the `BaseFormat` FRW interface.
src/epicc/formats/yaml.py	Adds YAML FRW backend (ruamel-yaml).
src/epicc/formats/xlsx.py	Adds XLSX FRW backend (openpyxl).
src/epicc/formats/template.py	Adds template generation from Pydantic models.
src/epicc/model/init.py	Adds model loader for schema-based YAML models.
src/epicc/model/base.py	Adds `BaseSimulationModel` ABC for Python-based models.
src/epicc/model/schema.py	Adds Pydantic schema for YAML-defined models.
src/epicc/models/tb_isolation.py	Ports TB Isolation model to `BaseSimulationModel`.
src/epicc/models/tb_isolation.yaml	Adds default parameters for TB Isolation model.
src/epicc/models/measles_outbreak.py	Ports Measles Outbreak model to `BaseSimulationModel`.
src/epicc/models/measles_outbreak.yaml	Adds default parameters for Measles Outbreak model.
src/epicc/utils/model_loader.py	Defines built-in model registry (returns model instances).
src/epicc/utils/parameter_loader.py	Adds typed parameter loading + flattening via FRW.
src/epicc/utils/parameter_ui.py	Adds updated Streamlit parameter UI rendering helpers.
src/epicc/utils/section_renderer.py	Adds updated section rendering utilities.
src/epicc/utils/excel_model_runner.py	Refactors existing Excel runner (style cleanup; reuses shared `flatten_dict`).
tests/epicc/test_formats.py	Adds tests for format selection + `read_from_format`.
tests/epicc/test_formats_yaml.py	Adds YAMLFormat tests (read/write, error cases).
tests/epicc/test_formats_xlsx.py	Adds XLSXFormat tests (read/write, error cases).
tests/epicc/test_formats_template.py	Adds template generation integration tests (YAML + XLSX).
tests/epicc/test_model_loader.py	Adds tests for schema-based model loader.
sample/TB Isolation.xlsx	Adds sample Excel file for TB Isolation.
sample/Measles Outbreak.xlsx	Adds sample Excel file for Measles Outbreak.
sample/~$TB Isolation.xlsx	Adds an Excel temp/lock file (artifact).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-24T05:04:40Z

src/epicc/model/__init__.py

+    # We're okay to cast here because we know the file exists and Traversable is a supertype of
+    # Path, but mypy doesn't know that.
+    config_path = cast(
+        Path, importlib.resources.files("epicc").joinpath(f"models/{name}.yaml")
+    )
+
+    return read_from_format(config_path, config_path.open("rb"), Model)


importlib.resources.files(...) returns a Traversable, not a pathlib.Path. Casting it to Path doesn’t change the runtime type, and read_from_format() ultimately calls Path(path).suffix, which will raise TypeError for a Traversable in zipped/pyodide-style deployments. Prefer passing a real str/Path for suffix detection (e.g., f"{name}.yaml") while using the Traversable only to open the stream, or extend read_from_format/get_format to accept Traversable safely.

Suggested change

# We're okay to cast here because we know the file exists and Traversable is a supertype of

# Path, but mypy doesn't know that.

config_path = cast(

Path, importlib.resources.files("epicc").joinpath(f"models/{name}.yaml")

)

return read_from_format(config_path, config_path.open("rb"), Model)

# Use a Traversable for opening the resource, and a real Path/str for suffix detection.

config_resource = importlib.resources.files("epicc").joinpath(f"models/{name}.yaml")

config_name = Path(f"{name}.yaml")

return read_from_format(config_name, config_resource.open("rb"), Model)

@olivia-banks fyi

Copilot · 2026-03-24T05:04:40Z

src/epicc/config/__init__.py

+    # We're okay to cast here because we know the file exists and Traversable is a supertype of
+    # Path, but mypy doesn't know that.
+    config_path = cast(
+        Path, importlib.resources.files("epicc").joinpath(f"config/{name}.yaml")
+    )
+    return read_from_format(config_path, config_path.open("rb"), Config)


Same issue as epicc.model.load_model: importlib.resources.files(...) returns a Traversable, and casting to Path can lead to runtime errors when read_from_format() tries to wrap it with Path(...). To keep this compatible with packaging/pyodide, pass a str/Path filename for format detection and use the Traversable only for .open() (or update read_from_format/get_format to support Traversable).

Suggested change

# We're okay to cast here because we know the file exists and Traversable is a supertype of

# Path, but mypy doesn't know that.

config_path = cast(

Path, importlib.resources.files("epicc").joinpath(f"config/{name}.yaml")

)

return read_from_format(config_path, config_path.open("rb"), Config)

resource = importlib.resources.files("epicc").joinpath(f"config/{name}.yaml")

filename = f"config/{name}.yaml"

return read_from_format(filename, resource.open("rb"), Config)

@olivia-banks fyi

Copilot · 2026-03-24T05:04:40Z

src/epicc/formats/xlsx.py

+    def write(self, data: dict[str, Any], template: Workbook | None = None) -> bytes:
+        """
+        Write a dictionary to an XLSX file.
+
+        Args:
+            data: Dictionary to write.
+            template: Optional Workbook object to use as a template for writing. If provided, the
+                      structure and formatting of the template will be preserved as much as possible.
+
+        Returns:
+            Byte array containing the XLSX data.
+        """
+
+        wb = template or Workbook()
+        ws = wb.active
+        assert ws is not None, "Workbook must have an active worksheet (bug)."
+
+        # Populate with the provided data piecewise.
+        for row in ws.iter_rows():
+            key_cell = row[_COL_PARAMETER]
+            val_cell = row[_COL_VALUE]
+            if key_cell.value in data:
+                val_cell.value = data[key_cell.value]  # type: ignore
+


XLSXFormat.read() builds a nested dict from dot-notation keys, but write() only updates cells when key_cell.value in data, which only works for a flat dict keyed by the worksheet strings. As-is, a read→modify→write flow will silently fail for nested values (e.g., updating {"costs": {"latent": 1}} won’t match a costs.latent row). Consider flattening nested input data to dot-notation before the loop (or change read() to keep flat keys).

Copilot · 2026-03-24T05:04:41Z

src/epicc/formats/yaml.py

+        yaml = YAML(typ="safe")
+
+        try:
+            data = yaml.load(data)
+        except Exception as e:
+            raise ValueError(f"Failed to parse YAML data at {self.path}") from e
+
+        if not isinstance(data, dict):
+            raise ValueError(
+                f"Expected a YAML mapping at the top level in {self.path}, got {type(data).__name__}"
+            )
+
+        return data, yaml


The module docstring says this reader preserves YAML structure/trivia, but the implementation uses YAML(typ="safe") and returns a plain dict, which discards comments/formatting. If trivia preservation is a goal, use ruamel’s round-trip loader (typ="rt") and keep the loaded CommentedMap (or similar) as the template for write().

Copilot · 2026-03-24T05:04:41Z

src/epicc/formats/yaml.py

@@ -0,0 +1,76 @@
+"""
+Generic reader for YAML parameter files. Excepts a YAML file with a mapping at the top level, which


Spelling in the module docstring: “Excepts a YAML file …” should be “Expects a YAML file …”.

Suggested change

Generic reader for YAML parameter files. Excepts a YAML file with a mapping at the top level, which

Generic reader for YAML parameter files. Expects a YAML file with a mapping at the top level, which

Copilot · 2026-03-24T05:04:41Z

tests/epicc/test_model_loader.py

+import importlib.resources
+import tempfile
+import textwrap
+from pathlib import Path
+from unittest.mock import MagicMock, patch
+


These imports appear unused and will be flagged by Ruff’s default rules (F401): importlib.resources, MagicMock. Remove them or use them in the tests.

Copilot · 2026-03-24T05:04:41Z

pyproject.toml

 [dependency-groups]
 dev = [
    "ruff>=0.15.6",
+    "pytest>=8.0.0",


Pytest is used by the newly added test suite (tests/epicc/*) but isn’t declared in any dependency group, and it also doesn’t appear in uv.lock. Add pytest (and any other test runner deps) to an appropriate dependency group (e.g. dev or a new test group) and regenerate the lockfile so a fresh env/container can run the tests.

Copilot · 2026-03-24T05:04:42Z

src/epicc/__main__.py

+import importlib.resources
+from typing import Any
+
+import streamlit as st
+
+from epicc.config import CONFIG
+from epicc.formats import VALID_PARAMETER_SUFFIXES
+from epicc.model.base import BaseSimulationModel
+from epicc.utils.excel_model_runner import (


This module uses absolute imports like from epicc.config import CONFIG, which requires src/ to be on PYTHONPATH or the project to be installed as a package. If users run streamlit run src/epicc/__main__.py (a common pattern after moving to src/ layout), sys.path will include src/epicc (not src) and these imports will fail. Consider adding a documented/automated entrypoint (e.g., a small top-level app.py shim, or ensuring uv sync installs the project and updating run instructions) so the app is runnable without manual PYTHONPATH setup.

[stash] draft

c2493c7

olivia-banks self-assigned this Mar 18, 2026

olivia-banks mentioned this pull request Mar 18, 2026

Define YAML schema and write a validation tool validate_yaml #15

Open

olivia-banks added this to the MVP milestone Mar 18, 2026

olivia-banks linked an issue Mar 18, 2026 that may be closed by this pull request

Define YAML schema and write a validation tool validate_yaml #15

Open

olivia-banks added the enhancement New feature or request label Mar 18, 2026

[stash] draft

d5125b9

olivia-banks added 2 commits March 23, 2026 22:44

[test] test!

f4d25e8

[format] templating

37d6e1e

olivia-banks marked this pull request as ready for review March 24, 2026 04:57

Copilot AI review requested due to automatic review settings March 24, 2026 04:57

Copilot started reviewing on behalf of olivia-banks March 24, 2026 04:58 View session

[dep] dep

3050891

Copilot AI reviewed Mar 24, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Address #15; Lay Validation Foundation; Light Cleanup Work#17

Address #15; Lay Validation Foundation; Light Cleanup Work#17
olivia-banks wants to merge 5 commits intoEpiForeSITE:mainfrom
olivia-banks:ob-organize

olivia-banks commented Mar 18, 2026 •

edited

Loading

Uh oh!

olivia-banks commented Mar 21, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Mar 24, 2026

Uh oh!

olivia-banks Mar 24, 2026

Uh oh!

Copilot AI Mar 24, 2026

Uh oh!

olivia-banks Mar 24, 2026

Uh oh!

Copilot AI Mar 24, 2026

Uh oh!

Copilot AI Mar 24, 2026

Uh oh!

Copilot AI Mar 24, 2026

Uh oh!

Copilot AI Mar 24, 2026

Uh oh!

Copilot AI Mar 24, 2026

Uh oh!

Copilot AI Mar 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		@@ -0,0 +1,76 @@
		"""
		Generic reader for YAML parameter files. Excepts a YAML file with a mapping at the top level, which

Conversation

olivia-banks commented Mar 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

olivia-banks commented Mar 21, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

olivia-banks Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

olivia-banks Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

olivia-banks commented Mar 18, 2026 •

edited

Loading