diff --git a/AGENTS.md b/AGENTS.md index bfee034c..71501e75 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -135,6 +135,7 @@ docker compose -f docker-compose.local.yml up --build | Topic | File | |---|---| | Project overview | `README.md` | +| Developer onboarding | `docs/developer/README.md` | | Backend details | `backend/README.md` | | Frontend architecture | `frontend/README.md` | | CI/CD & deployment | `docs/cicd/DEPLOYMENT.md` | diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 00000000..d0cbfd19 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,61 @@ +# Contributing to SimBoard + +Use this file for contribution workflow. For local setup, architecture, and developer onboarding, see [docs/developer/README.md](docs/developer/README.md). + +## Start With an Issue + +Start from an issue when the work is not trivial. + +- bug reports: `.github/ISSUE_TEMPLATE/bug_report.yml` +- enhancements: `.github/ISSUE_TEMPLATE/enhancement_request.yml` +- planning work: `.github/ISSUE_TEMPLATE/planning_task.yml` +- DevOps work: `.github/ISSUE_TEMPLATE/devops.yml` + +If no issue exists, open one first or document the reason the change is intentionally small and self-contained. + +## Branches and Commits + +- branch from `main` +- use short-lived, descriptive branch names +- keep commits focused and reviewable +- avoid mixing behavior changes, refactors, and unrelated cleanup in one commit + +No repository branch naming convention is documented, so prefer clarity over personal shorthand. + +## Pull Requests + +Use the PR template in `.github/pull_request_template.md`. + +Each PR should: + +- link the relevant issue when applicable +- explain the change and motivation +- note any required documentation updates +- note any deployment or migration steps if applicable + +## Required Checks Before Opening a PR + +Run the relevant checks from the repository root: + +```bash +make backend-test +make frontend-lint +make pre-commit-run +pnpm --dir frontend run type-check +``` + +If your change only touches one subsystem, still prefer running the nearest relevant checks rather than skipping validation entirely. + +## Review Expectations + +- keep diffs small enough to review safely +- add or update tests when behavior changes +- include schema or migration notes when persistence changes +- update documentation when workflows or capabilities change + +## Where to Go Next + +- developer guide: [docs/developer/README.md](docs/developer/README.md) +- backend details: [backend/README.md](backend/README.md) +- frontend details: [frontend/README.md](frontend/README.md) +- operations and deployment docs: [docs/README.md](docs/README.md) diff --git a/LICENSE b/LICENSE new file mode 100644 index 00000000..261eeb9e --- /dev/null +++ b/LICENSE @@ -0,0 +1,201 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/Makefile b/Makefile index 02fb8870..eee12f1e 100644 --- a/Makefile +++ b/Makefile @@ -55,7 +55,7 @@ help: @echo " make backend-seed # Run DB seed script" @echo " make backend-rollback-seed # Rollback seeded data" @echo " make backend-create-admin # Create admin user (interactive)" - @echo " make backend-provision-service # Provision service account (interactive)" + @echo " make backend-provision-service service_name= # Provision service account" @echo "" @echo "$(BLUE)Frontend:$(NC)" @@ -253,9 +253,6 @@ frontend-install: frontend-clean: cd $(FRONTEND_DIR) && rm -rf node_modules dist .turbo -frontend-local: - cd $(FRONTEND_DIR) && pnpm dev - frontend-build: cd $(FRONTEND_DIR) && pnpm build diff --git a/README.md b/README.md index edb50d81..ba77af83 100644 --- a/README.md +++ b/README.md @@ -4,317 +4,58 @@

-SimBoard is a platform for managing and comparing Earth system simulation metadata, with a focus on **E3SM** (Energy Exascale Earth System Model) reference simulations. - -SimBoard helps researchers: - -- Store and organize simulation metadata -- Browse and visualize simulation details -- Compare runs side-by-side -- Surface diagnostics and metadata-driven insights - -AI-assisted capabilities are being explored to supplement these features, including automated metadata summarization, simulation comparison analysis, and intelligent discovery of patterns across runs. - ---- - -## Table of Contents - -- [Prerequisites](#prerequisites) -- [Developer Quickstart](#developer-quickstart) - - [Commands](#1-commands) - - [Setup GitHub OAuth Authentication](#2-setup-github-oauth-authentication) - - [Local HTTPS](#3-local-https) -- [Repository Structure](#repository-structure) -- [Development Notes](#development-notes) - - [Pre-commit Hooks](#pre-commit-hooks) -- [Staging and Production Environments](#staging-and-production-environments) - - [Deployment Guide (CI/CD)](#deployment-guide-cicd) - - [NERSC Spin Runbook (Rancher UI)](#nersc-spin-runbook-rancher-ui) -- [License](#license) - ---- - -## Prerequisites - -1. **Install Docker Desktop** - - Download and install: [https://www.docker.com/products/docker-desktop](https://www.docker.com/products/docker-desktop) - - Ensure Docker Desktop is running before using any Docker-based commands. - -2. **Install uv (Python dependency manager)** - - macOS/Linux: - - ```bash - curl -LsSf https://astral.sh/uv/install.sh | sh - ``` - - - Windows: - - ```bash - powershell -c "irm https://astral.sh/uv/install.ps1 | iex" - ``` - - - Verify installation: - - ```bash - uv --version - ``` - -3. **Install Node.js, npm, and pnpm** - - Install Node.js (LTS recommended): [https://nodejs.org](https://nodejs.org) - - Verify Node and npm: - - ```bash - node --version - npm --version - ``` - - - Install pnpm: - - ```bash - npm install -g pnpm - pnpm --version - ``` - -4. **Clone the repository** - - ```bash - git clone https://github.com/E3SM-Project/simboard.git - cd simboard - ``` - -## Developer Quickstart - -> ⚠️ This is a bare-metal development environment and is _not production-accurate_ — it is optimized for speed. - -This is the **recommended daily workflow**: - -- Fastest hot reloads -- Best debugging experience -- No Docker overhead - -### 1. Commands - -```bash -cd simboard - -# 1. Setup development assets (env files + certs + DB + deps) -make setup-local - -# 2. (First time only) Create an admin account (interactive) -make backend-create-admin - -# 3. Start backend (terminal 1) -make backend-run - -# 4. Start frontend (terminal 2) -make frontend-run - -# 5. Open API and UI -open https://127.0.0.1:8000/docs -open https://127.0.0.1:5173 -``` - -For a list of all `make` commands, run: - -```bash -make help -``` - -### 2. Setup GitHub OAuth Authentication - -#### 1. Create a GitHub OAuth App - -1. Go to: https://github.com/settings/developers -2. Click **“New OAuth App”**. -3. Fill in: - - **Application name:** SimBoard (local) - - **Homepage URL:** https://127.0.0.1:5173 - - **Authorization callback URL:** - - ```bash - https://127.0.0.1:8000/api/v1/auth/github/callback - ``` - -4. Click **Register application**. -5. Copy the generated: - - **Client ID** - - **Client Secret** - -#### 2. Configure local environment variables - -Add the credentials to: - -```bash -.envs/local/backend.env -``` - -Example: - -```env -GITHUB_CLIENT_ID=your_client_id -GITHUB_CLIENT_SECRET=your_client_secret - -# Must match GitHub OAuth callback URL exactly -GITHUB_REDIRECT_URL=https://127.0.0.1:8000/auth/github/callback - -# Generate securely: -# python -c "import secrets; print(secrets.token_urlsafe(64))" -GITHUB_STATE_SECRET_KEY=your_secret -``` - -Restart the backend after updating environment variables. - -#### 3. Local HTTPS - -For local development, SimBoard uses **local HTTPS** with development certificates: - -```bash -certs/local.crt -certs/local.key -``` - -These files are generated automatically with `make install`. To re-generate, run: - -```bash -make gen-certs -``` - -Used automatically by: - -- FastAPI (Uvicorn SSL) -- Vite (via `VITE_SSL_CERT`, `VITE_SSL_KEY`) - -## Development Notes - -- Backend dependencies managed using **uv** -- Frontend dependencies managed using **pnpm** - -Use [GitHub Issues](https://github.com/E3SM-Project/simboard/issues/new/choose) to report bugs or propose features. -Pull requests should include tests + documentation updates. - -### Repository Structure - -```bash -simboard/ -├── backend/ # FastAPI, SQLAlchemy, Alembic, OAuth, metadata ingestion -├── frontend/ # Vite + React + Tailwind + shadcn -├── .envs/ # Env configs: example/ (templates, committed) + local/ (developer values, ignored) -├── docker-compose.local.yml -├── docker-compose.yml -├── Makefile # unified monorepo automation -└── certs/ # dev HTTPS certificates -``` - -### Pre-commit Hooks - -This repository uses **pre-commit** to enforce consistent checks for both the backend (Python) and frontend (TypeScript). - -Pre-commit runs automatically on `git commit` and will block commits if checks fail. - -#### 1. Where to Run It - -Always run pre-commit from the **repository root**. - -Some hooks (e.g., `mypy`) rely on config paths like `backend/pyproject.toml`. Running from a subdirectory can cause incorrect or inconsistent results. - -✅ Correct: - -```bash -pre-commit run --all-files -``` - -❌ Incorrect: - -```bash -cd backend -pre-commit run --all-files -``` - -CI also runs pre-commit from the repo root. - -#### 2. What It Checks - -Backend: - -- Ruff (lint + format) -- mypy (type checking) - -Frontend: - -- ESLint -- Prettier - -All hooks are configured in the root `.pre-commit-config.yaml`. - -#### 3. Installation - -After cloning: - -```bash -make install -``` - -This will: - -- Create the backend `uv` environment (if missing) -- Install dependencies -- Install git hooks - -To reinstall hooks only: - -```bash -make pre-commit-install -``` - -#### 4. Run Manually - -```bash -make pre-commit-run -``` - -Or: - -```bash -uv run pre-commit run --all-files -``` +

+ Backend CI + Frontend CI + License: Apache 2.0 +

-#### 5. Notes +## Overview -- Python hooks run via `uv` -- Frontend hooks run via `pnpm` -- Required tools (`uv`, `pnpm`, `node`) must be available in your system `PATH` -- Hooks auto-fix most formatting issues; re-stage files and re-commit if needed +SimBoard is an E3SM-focused catalog for simulation metadata. It turns simulation archives and run metadata into a browsable, comparable record of cases, executions, provenance, and related artifacts. -#### 6. Skipping Hooks (Not Recommended) +## Why SimBoard Exists -```bash -git commit --no-verify -``` +SimBoard exists so researchers and maintainers do not have to reconstruct simulation context from raw files, ad hoc notes, and scattered links. The repository is centered on simulation metadata, comparison, and provenance rather than raw model output serving. -Use only when absolutely necessary. +## Current Capabilities -## Staging and Production Environments +- Ingest packaged simulation archives into normalized case and simulation records +- Browse runs and cases from the UI +- View case details, simulation details, artifacts, and external links +- Compare selected simulations side by side +- Preserve provenance such as machine, Git metadata, HPC username, artifacts, and external links +- Resolve PACE experiment links from execution IDs +- Support browser-based GitHub auth and service-account token auth for privileged automation -### Deployment Guide (CI/CD) +## System Summary -Complete CI/CD pipelines, release process, rollback guidance, troubleshooting, and manual image build instructions are documented in: +SimBoard is organized as a React frontend, a FastAPI backend, and PostgreSQL-backed persistence. Together they handle metadata ingestion, normalization, browsing, comparison, provenance, and authenticated upload workflows. -- [docs/cicd/DEPLOYMENT.md](docs/cicd/DEPLOYMENT.md) +For architecture diagrams, API/data-flow detail, and contributor-oriented system context, see [docs/developer/README.md](docs/developer/README.md). -### NERSC Spin Runbook (Rancher UI) +## Technology At A Glance -All NERSC Spin-specific workload configuration and ingestion service-account setup are documented in: +- Frontend: React, TypeScript, Vite, React Router, TanStack Query, Tailwind CSS, shadcn/ui +- Backend: FastAPI, Pydantic, SQLAlchemy, Alembic +- Database: PostgreSQL +- Auth: GitHub OAuth for browser flows, API tokens for service accounts +- Tooling: `uv`, `pnpm`, `ruff`, `mypy`, `eslint`, `prettier`, `pre-commit` +- CI/CD: GitHub Actions plus NERSC-focused deployment/build docs under `docs/` -- [docs/deploy/spin.md](docs/deploy/spin.md) +## Documentation Map -Use this runbook for: +- Contributor guide: [docs/developer/README.md](docs/developer/README.md) +- Contribution workflow: [CONTRIBUTING.md](CONTRIBUTING.md) +- Backend details: [backend/README.md](backend/README.md) +- Frontend details: [frontend/README.md](frontend/README.md) +- Docs index: [docs/README.md](docs/README.md) +- CI/CD and deployment docs: [docs/cicd/README.md](docs/cicd/README.md) -- Rancher workload setup (`db`, `backend`, `frontend`, ingress, TLS, CronJob) -- Ingestion service-account provisioning and `simboard-ingestion-env` secret setup -- NERSC archive mount and ingestion PVC configuration +## How to Contribute ---- +Start with [CONTRIBUTING.md](CONTRIBUTING.md) for issue, branch, commit, PR, and validation expectations. If you are new to the repo, use the contributor guide at [docs/developer/README.md](docs/developer/README.md) for local setup, architecture, and development workflow. ## License -TBD +Apache License 2.0 — see [LICENSE](LICENSE). diff --git a/backend/README.md b/backend/README.md index b288b960..8f1c2887 100644 --- a/backend/README.md +++ b/backend/README.md @@ -1,18 +1,37 @@ # SimBoard Backend -This backend uses [UV](https://uv-py.github.io/) for dependency management and [FastAPI](https://fastapi.tiangolo.com/) as the web framework. +The backend is a FastAPI service that ingests simulation archives, stores normalized metadata in PostgreSQL, and exposes the `/api/v1` REST API used by the frontend and service-account tooling. -It provides a REST API for managing and querying simulation metadata, including endpoints for browsing, comparing, and analyzing **E3SM** (Energy Exascale Earth System Model) simulation data. +## Responsibilities -## Tech Stack +- archive ingestion and validation +- case, simulation, machine, and ingestion persistence +- GitHub OAuth and API-token authentication +- PACE execution lookup +- API schemas and routing -> ℹ️ **Note:** The backend runs as a Docker container. +## Important Locations -- **FastAPI** — Web framework for building APIs -- **UV** — Python dependency and environment management -- **SQLAlchemy** — ORM and database toolkit, with **Alembic** for database migrations -- **PostgreSQL** — Primary relational database +```text +backend/app/main.py FastAPI app and router registration +backend/app/features/ingestion/ ingestion endpoints and parser integration +backend/app/features/simulation/ cases, simulations, schemas, delta logic +backend/app/features/machine/ machine models and API +backend/app/features/user/ auth, tokens, user models +backend/app/core/ config, DB setup, exceptions, logging +backend/migrations/ Alembic migrations +backend/tests/ pytest coverage +``` -## License +## Developer Commands -For license information, see the [root LICENSE file](../LICENSE). +Run these from the repo root: + +```bash +make backend-run +make backend-test +make backend-migrate m='message' +make backend-upgrade +``` + +For repo-wide setup and contributor workflow, see [docs/developer/README.md](../docs/developer/README.md). diff --git a/docs/README.md b/docs/README.md index 6a624654..a5d81bd8 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,48 +1,23 @@ -# SimBoard Documentation +# SimBoard Docs -Documentation for the SimBoard project. +Use this directory as the documentation router by audience. ---- +## For Contributors -## 📁 Documentation Structure +- Developer guide: [developer/README.md](developer/README.md) +- Contribution workflow: [../CONTRIBUTING.md](../CONTRIBUTING.md) -```bash -docs/ -├── README.md # This file -├── cicd/ # CI/CD and deployment -│ ├── README.md # Quick start and overview -│ └── DEPLOYMENT.md # Complete reference guide -└── deploy/ # Environment-specific deployment runbooks - └── spin.md # Spin backend migration rollout + frontend/db/ingress config -``` +## For Backend Developers ---- +- Backend guide: [../backend/README.md](../backend/README.md) -## 🚀 CI/CD Quick Links +## For Frontend Developers -**New to CI/CD setup?** +- Frontend guide: [../frontend/README.md](../frontend/README.md) -- [cicd/README.md](cicd/README.md) - Quick start guide +## For Operations and Deployment -**Need deployment details?** - -- [cicd/DEPLOYMENT.md](cicd/DEPLOYMENT.md) - Complete reference -- [deploy/spin.md](deploy/spin.md) - Spin backend/frontend/db/ingress workload runbook - ---- - -## 📚 CI/CD Documentation - -All CI/CD and deployment documentation is in the [`cicd/`](cicd/) directory: - -- **[cicd/README.md](cicd/README.md)** - Quick start, overview, and common operations -- **[cicd/DEPLOYMENT.md](cicd/DEPLOYMENT.md)** - Complete deployment guide with workflows, Kubernetes examples, and troubleshooting -- **[deploy/spin.md](deploy/spin.md)** - Spin-specific backend migration-first plus frontend/db/ingress runbook - ---- - -## 🔗 External Links - -- [NERSC Registry](https://registry.nersc.gov/harbor/projects) -- [NERSC Spin Dashboard](https://rancher2.spin.nersc.gov/) -- [GitHub Actions](https://github.com/E3SM-Project/simboard/actions) +- CI/CD overview: [cicd/README.md](cicd/README.md) +- Deployment reference: [cicd/DEPLOYMENT.md](cicd/DEPLOYMENT.md) +- NERSC Spin runbook: [deploy/spin.md](deploy/spin.md) +- HPC token and service-account auth: [hpc_api_token_authentication.md](hpc_api_token_authentication.md) diff --git a/docs/cicd/DEPLOYMENT.md b/docs/cicd/DEPLOYMENT.md index d4f14151..dc1bcaa7 100644 --- a/docs/cicd/DEPLOYMENT.md +++ b/docs/cicd/DEPLOYMENT.md @@ -1,5 +1,7 @@ # Deployment Guide +Audience: maintainers operating deployments and release rollouts. + Complete reference for CI/CD pipelines and NERSC Spin deployments. ## Table of Contents @@ -34,7 +36,7 @@ SimBoard uses **GitHub Actions** to automatically build and publish container im ## Quick Links - **Harbor Registry:** -- **Rancher Dashboard:** +- **Rancher Dashboard:** - **GitHub Actions:** - **NERSC Spin Runbook (Rancher UI):** [docs/deploy/spin.md](../deploy/spin.md) @@ -53,10 +55,10 @@ SimBoard uses **GitHub Actions** to automatically build and publish container im | Component | Hosting | Image | Pull Policy | | --------- | ----------------- | ---------------- | ------------ | -| Backend | NERSC Spin (prod) | `backend:1.0.0` | IfNotPresent | -| Frontend | NERSC Spin (prod) | `frontend:2.1.0` | IfNotPresent | +| Backend | NERSC Spin (prod) | `backend:X.Y.Z` | IfNotPresent | +| Frontend | NERSC Spin (prod) | `frontend:X.Y.Z` | IfNotPresent | -**Trigger:** Component-scoped GitHub Release tag (e.g., `backend-v1.0.0`, `frontend-v2.1.0`) +**Trigger:** Component-scoped GitHub Release tag (for example, `backend-vX.Y.Z`, `frontend-vX.Y.Z`) > **Note:** Frontend and backend are versioned independently. Each component can be released on its own schedule without affecting the other. @@ -66,7 +68,7 @@ SimBoard uses **GitHub Actions** to automatically build and publish container im Dev workflows build and push images tagged with `:dev` and `:sha-` whenever changes are pushed to `main`. These do **not** affect production images. -#### Backend Dev (`build-backend-dev.yml`) +#### Backend Dev Workflow **Triggers:** Push to `main` (backend changes) or manual dispatch @@ -74,7 +76,7 @@ Dev workflows build and push images tagged with `:dev` and `:sha-` whene **Registry:** `registry.nersc.gov/e3sm/simboard/backend` -#### Frontend Dev (`build-frontend-dev.yml`) +#### Frontend Dev Workflow **Triggers:** Push to `main` (frontend changes) or manual dispatch @@ -90,7 +92,7 @@ Dev workflows build and push images tagged with `:dev` and `:sha-` whene Release workflows are triggered by component-scoped Git tags created through GitHub Releases. Each component has its own workflow and tag namespace. Release builds do **not** modify the `:dev` image. -#### Backend Prod (`build-backend-prod.yml`) +#### Backend Prod Workflow **Triggers:** Tag push matching `backend-v*` @@ -98,7 +100,7 @@ Release workflows are triggered by component-scoped Git tags created through Git **Registry:** `registry.nersc.gov/e3sm/simboard/backend` -#### Frontend Prod (`build-frontend-prod.yml`) +#### Frontend Prod Workflow **Triggers:** Tag push matching `frontend-v*` @@ -112,7 +114,7 @@ Release workflows are triggered by component-scoped Git tags created through Git ### Build Flow Summary -``` +```text Dev builds: push to main → :dev, :sha- Release builds: component tag → :X.Y.Z, :sha-, :latest ``` @@ -139,7 +141,7 @@ docker login registry.nersc.gov **Security:** - Use service account tokens when available -- Rotate credentials every 90 days +- Rotate credentials on a schedule that matches current NERSC and project policy - Never commit credentials to source code ## Image Tagging Strategy @@ -155,7 +157,7 @@ docker login registry.nersc.gov | Tag | Description | Use Case | | --------- | -------------- | ------------------------ | -| `:1.2.0` | Full version | Production (recommended) | +| `:X.Y.Z` | Full version | Production (recommended) | | `:latest` | Latest release | Reference only | **Best practice:** Use full semantic versions (`:X.Y.Z`) in production for reproducibility. @@ -164,8 +166,8 @@ docker login registry.nersc.gov | Git Tag | Component | Docker Image Tag | | ----------------- | --------- | ------------------------------------------------- | -| `backend-v1.0.0` | Backend | `registry.nersc.gov/e3sm/simboard/backend:1.0.0` | -| `frontend-v2.1.0` | Frontend | `registry.nersc.gov/e3sm/simboard/frontend:2.1.0` | +| `backend-vX.Y.Z` | Backend | `registry.nersc.gov/e3sm/simboard/backend:X.Y.Z` | +| `frontend-vX.Y.Z` | Frontend | `registry.nersc.gov/e3sm/simboard/frontend:X.Y.Z` | ## Development Deployment @@ -212,34 +214,36 @@ make frontend-lint 1. Navigate to [Releases](https://github.com/E3SM-Project/simboard/releases/new) 2. Click **Draft a new release** 3. In **Choose a tag**, enter a new tag following the convention: + + ```text + frontend-vX.Y.Z ``` - frontend-v1.2.0 - ``` + 4. Ensure the **Target** branch is `main` -5. Set the release title (e.g., `Frontend v1.2.0`) +5. Set the release title (for example, `Frontend vX.Y.Z`) 6. Add release notes summarizing the changes 7. Click **Publish release** Publishing the release creates the Git tag, which: -- Triggers the `frontend-v*` workflow (`build-frontend-prod.yml`) +- Triggers the frontend release workflow for `frontend-v*` tags - Builds the Docker image -- Pushes versioned tags (`:1.2.0`, `:sha-`, `:latest`) to the registry +- Pushes versioned tags (`:X.Y.Z`, `:sha-`, `:latest`) to the registry - Does **not** modify the `:dev` image ### Step 2b: Create GitHub Release (Backend) Follow the same steps as above, but use a backend-scoped tag: -``` -backend-v1.0.0 +```text +backend-vX.Y.Z ``` -This triggers `build-backend-prod.yml` and pushes backend-specific versioned tags. +This triggers the backend release workflow and pushes backend-specific versioned tags. ### Step 3: Monitor Builds -Check the [Actions tab](https://github.com/E3SM-Project/simboard/actions) — only the workflow matching the component tag will trigger. Build typically completes in ~10-15 minutes. +Check the [Actions tab](https://github.com/E3SM-Project/simboard/actions) and confirm that only the workflow matching the component tag triggers. ### Step 4: Deploy to Production @@ -248,8 +252,8 @@ Update the image tags in the [Rancher UI](https://rancher2.spin.nersc.gov/dashbo 1. Navigate to **Workloads → Deployments** in the prod namespace 2. Click the target deployment → **⋮ → Edit Config** 3. Update the **Image** field to the new versioned image, e.g.: - - Backend: `registry.nersc.gov/e3sm/simboard/backend:1.0.0` - - Frontend: `registry.nersc.gov/e3sm/simboard/frontend:1.2.0` + - Backend: `registry.nersc.gov/e3sm/simboard/backend:X.Y.Z` + - Frontend: `registry.nersc.gov/e3sm/simboard/frontend:X.Y.Z` 4. Set **Pull Policy** to `IfNotPresent` 5. Click **Save** — Rancher will roll out the new version @@ -325,7 +329,7 @@ Rolling back the backend container image does not roll back database schema auto ## Rollback Procedure -Version-tagged images are **immutable** — once published, a version tag (e.g., `:1.0.0`) always refers to the same image. This makes rollbacks safe and predictable. +Version-tagged images are **immutable** — once published, a version tag (for example, `:X.Y.Z`) always refers to the same image. This makes rollbacks safe and predictable. ### Rolling Back via Rancher @@ -333,8 +337,8 @@ Version-tagged images are **immutable** — once published, a version tag (e.g., 2. Navigate to **Workloads → Deployments** in the prod namespace 3. Click the deployment to roll back → **⋮ → Edit Config** 4. Change the **Image** tag to the previous known-good version, e.g.: - - `registry.nersc.gov/e3sm/simboard/backend:0.9.0` - - `registry.nersc.gov/e3sm/simboard/frontend:1.1.0` + - `registry.nersc.gov/e3sm/simboard/backend:X.Y.Z` + - `registry.nersc.gov/e3sm/simboard/frontend:X.Y.Z` 5. Click **Save** to trigger the rollout Alternatively, use the built-in Rancher rollback: @@ -344,7 +348,7 @@ Alternatively, use the built-in Rancher rollback: ### Key Rollback Principles -- **Version tags are immutable:** `:1.0.0` always points to the same image digest. You can safely redeploy any previously released version. +- **Version tags are immutable:** A published `:X.Y.Z` tag always points to the same image digest. You can safely redeploy any previously released version. - **Components are independent:** Rolling back the frontend does not require rolling back the backend, and vice versa. - **`:dev` is unaffected:** Release rollbacks have no impact on the dev environment. - **Use commit-based tags for precision:** If you need to deploy a specific build, use the `:sha-` tag from the GitHub Actions build log. @@ -416,10 +420,12 @@ docker buildx build \ 1. Check workflow logs in Actions tab 2. Test Dockerfile locally: + ```bash cd backend && docker build . cd frontend && docker build --build-arg VITE_API_BASE_URL=https://example.com . ``` + 3. Verify all dependencies are pinned ### Dev Image Not Updating @@ -461,8 +467,8 @@ docker buildx build \ **Solutions:** 1. Verify the tag follows the component convention: - - Backend: `backend-vX.Y.Z` (e.g., `backend-v1.0.0`) - - Frontend: `frontend-vX.Y.Z` (e.g., `frontend-v1.2.0`) + - Backend: `backend-vX.Y.Z` + - Frontend: `frontend-vX.Y.Z` 2. Ensure the tag was created via a published GitHub Release (draft releases do not create tags) 3. Check the [Actions tab](https://github.com/E3SM-Project/simboard/actions) for the corresponding workflow diff --git a/docs/cicd/README.md b/docs/cicd/README.md index 05fc1c24..e3189c2f 100644 --- a/docs/cicd/README.md +++ b/docs/cicd/README.md @@ -1,5 +1,7 @@ # CI/CD Automation for NERSC Container Builds +Audience: maintainers operating CI/CD and release image builds. + Automated multi-arch container builds to NERSC Registry with dev/prod separation. --- @@ -36,12 +38,14 @@ docker pull registry.nersc.gov/e3sm/simboard/frontend:dev ## 📋 Workflows -| Workflow | Trigger | Image Tags | -| ------------------------- | --------------------------------- | ----------------------------------- | -| `build-backend-dev.yml` | Push to `main` (backend changes) | `:dev`, `:sha-` | -| `build-frontend-dev.yml` | Push to `main` (frontend changes) | `:dev`, `:sha-` | -| `build-backend-prod.yml` | Tag `backend-v*` | `:X.Y.Z`, `:sha-`, `:latest` | -| `build-frontend-prod.yml` | Tag `frontend-v*` | `:X.Y.Z`, `:sha-`, `:latest` | +Current workflow files and exact trigger filters live under [`../../.github/workflows/`](../../.github/workflows/). + +| Workflow | Trigger | Image Tags | +| ---------------------- | --------------------------------- | ------------------------------------ | +| Backend dev build | Push to `main` (backend changes) | `:dev`, `:sha-` | +| Frontend dev build | Push to `main` (frontend changes) | `:dev`, `:sha-` | +| Backend release build | Tag `backend-v*` | `:X.Y.Z`, `:sha-`, `:latest` | +| Frontend release build | Tag `frontend-v*` | `:X.Y.Z`, `:sha-`, `:latest` | **Registry:** `registry.nersc.gov/e3sm/simboard/{backend,frontend}` @@ -56,7 +60,7 @@ docker pull registry.nersc.gov/e3sm/simboard/frontend:dev ## 🏗️ Architecture -``` +```text Development: main branch → :dev tag → NERSC Spin dev namespace @@ -73,8 +77,8 @@ Production: **Prod Environment:** -- Backend: `registry.nersc.gov/e3sm/simboard/backend:1.0.0` -- Frontend: `registry.nersc.gov/e3sm/simboard/frontend:2.1.0` +- Backend: `registry.nersc.gov/e3sm/simboard/backend:X.Y.Z` +- Frontend: `registry.nersc.gov/e3sm/simboard/frontend:X.Y.Z` - Independently versioned via component-scoped GitHub Releases --- @@ -93,8 +97,8 @@ Frontend and backend are released independently using component-scoped tags. 2. **Create release on GitHub:** - Go to [Releases](https://github.com/E3SM-Project/simboard/releases/new) - Create tag using component convention: - - Frontend: `frontend-v1.2.0` - - Backend: `backend-v1.0.0` + - Frontend: `frontend-vX.Y.Z` + - Backend: `backend-vX.Y.Z` - Target: `main` - Write release notes - Publish release @@ -107,7 +111,7 @@ Frontend and backend are released independently using component-scoped tags. - Backend migrations run automatically in the backend initContainer via `/app/migrate.sh` during rollout - Open the [Rancher UI](https://rancher2.spin.nersc.gov/dashboard/home) - Navigate to **Workloads → Deployments** in the prod namespace - - Edit the deployment's image tag to the new version (e.g., `1.0.0`) + - Edit the deployment's image tag to the new release version (for example, `X.Y.Z`) - Set **Pull Policy** to `IfNotPresent` - Save to trigger the rollout @@ -130,7 +134,7 @@ To redeploy with the latest image, use the [Rancher UI](https://rancher2.spin.ne ```yaml # Prod deployments use explicit versions with IfNotPresent -image: registry.nersc.gov/e3sm/simboard/backend:1.0.0 +image: registry.nersc.gov/e3sm/simboard/backend:X.Y.Z imagePullPolicy: IfNotPresent ``` diff --git a/docs/deploy/spin.md b/docs/deploy/spin.md index fc93d554..c4989aa3 100644 --- a/docs/deploy/spin.md +++ b/docs/deploy/spin.md @@ -1,5 +1,7 @@ # NERSC Spin Workloads (Backend InitContainer Migrations) +Audience: operators deploying SimBoard on NERSC Spin. + This runbook defines the NERSC Spin workload baseline and backend rollout flow using an initContainer for automatic Alembic migrations. This runbook uses the Rancher UI as the primary deployment workflow. diff --git a/docs/developer/README.md b/docs/developer/README.md new file mode 100644 index 00000000..4e6332a0 --- /dev/null +++ b/docs/developer/README.md @@ -0,0 +1,294 @@ +# Developer Guide + +Use this guide for local setup, repo-wide development workflow, and contributor-oriented architecture. For service-specific detail, see [backend/README.md](../../backend/README.md) and [frontend/README.md](../../frontend/README.md). + +## Local Setup + +Prerequisites: + +- [Docker Desktop](https://www.docker.com/products/docker-desktop/) or compatible local Docker runtime +- [`uv`](https://docs.astral.sh/uv/getting-started/installation/) — fast Python package manager (replaces pip/venv) +- [Node.js](https://nodejs.org/) and [`pnpm`](https://pnpm.io/installation) — JavaScript runtime and package manager + +Recommended first-run flow from the repository root: + +```bash +make setup-local +make backend-run +make frontend-run +``` + +Open: + +- API docs: `https://127.0.0.1:8000/docs` +- UI: `https://127.0.0.1:5173` + +What `make setup-local` does: + +- copies `.envs/example/*` into `.envs/local/` if missing +- generates local TLS certs in `certs/` +- starts PostgreSQL from `docker-compose.local.yml` +- installs backend and frontend dependencies +- runs Alembic migrations +- seeds development data + +Useful commands: + +```bash +make backend-test # run backend pytest suite +make frontend-lint # lint frontend with ESLint +make pre-commit-run # run all pre-commit hooks (formatting, linting, etc.) +pnpm --dir frontend run type-check # TypeScript type checking (no Makefile wrapper yet) +make help # list all available Makefile targets +``` + +## GitHub Auth Setup + +If you need authenticated browser flows such as upload: + +1. [Create a GitHub OAuth app](https://github.com/settings/developers) with homepage `https://127.0.0.1:5173`. +2. Set the callback URL to `https://127.0.0.1:8000/api/v1/auth/github/callback`. +3. Put the GitHub credentials in `.envs/local/backend.env`. +4. Restart `make backend-run`. + +If you need admin-only local flows such as service-account or token provisioning: + +```bash +make backend-create-admin +``` + +For token-based ingestion and service-account details, see [docs/hpc_api_token_authentication.md](../hpc_api_token_authentication.md). + +## Architecture + +SimBoard is a web application for cataloging and comparing E3SM simulation metadata. The full application (frontend, backend, and database) is hosted on NERSC Spin. Automated ingestion jobs running on HPC sites collect metadata from an E3SM performance archive and push it to SimBoard, where the backend normalizes it and the frontend lets researchers browse, compare, and analyze results. + +```mermaid +flowchart LR + user[Browser User] + ingest([Automated Ingestion]) + + subgraph mono[SimBoard — hosted on NERSC Spin] + direction LR + fe[Frontend\nReact + Vite SPA] + be[Backend\nFastAPI /api/v1] + db[(PostgreSQL)] + end + + gh[GitHub OAuth] + pace[PACE Lookup] + + user --> fe + fe -- HTTPS + cookie auth --> be + ingest --> be + be --> db + be --> gh + be --> pace +``` + +- **Frontend** — browse, detail, compare, auth, and upload views. Calls the backend over HTTPS via `frontend/src/api/api.ts` with credentials enabled for cookie auth. +- **Backend** — parses ingested archives, applies validation and reference-simulation rules, persists normalized records, and exposes `/api/v1` endpoints. +- **PostgreSQL** — stores cases, simulations, machines, users, tokens, artifacts, links, and ingestion records. +- **External services** — GitHub OAuth (user login) and PACE (performance lookup). + +## Automated HPC Metadata Ingestion + +HPC sites automatically produce `performance_archive` metadata. Automated ingestion jobs running on those sites collect the metadata and push it to SimBoard through one of two ingestion modes: + +- **Path ingestion** — an ingestion job sends a path reference to SimBoard, and the backend reads the archive directly from a mounted filesystem (used when the site's storage is accessible to NERSC Spin, e.g., NERSC / Perlmutter). +- **Archive upload** — an ingestion job packages the archive and uploads it to SimBoard over HTTPS (used when the filesystem is not accessible from NERSC Spin, e.g., LCRC / Chrysalis). + +```mermaid +flowchart TD + subgraph SOURCES["Source Archives"] + NERSC_SRC["NERSC / Perlmutter"] + LCRC_SRC["LCRC / Chrysalis"] + ADDL_SRC["Additional HPC Sites"] + end + + subgraph AUTOMATION["Site-Side Automation"] + NERSC_WRAP["Ingestion Job\npushes path reference"] + UPLOAD_WRAP["Ingestion Job\npackages and uploads archive"] + end + + subgraph BACKEND["SimBoard Backend"] + PATH["Path Ingestion\nvalidate token, parse in place"] + UPLOAD["Archive Upload Ingestion\nvalidate token, stage and parse"] + NORMALIZE["Normalize and Validate"] + AUDIT["Ingestion Audit Record"] + DB[("PostgreSQL")] + end + + NERSC_SRC --> NERSC_WRAP -->|"path reference"| PATH + LCRC_SRC --> UPLOAD_WRAP + ADDL_SRC -.-> UPLOAD_WRAP + UPLOAD_WRAP -->|"archive upload"| UPLOAD + + PATH --> NORMALIZE + UPLOAD --> NORMALIZE + NORMALIZE --> AUDIT --> DB +``` + +All ingestion requests require a bearer API token. Site-side ingestion jobs are configured with machine name, source path, API URL, state path, dry-run flag, and the token. + +After ingestion completes, the backend stores normalized cases, simulations, machines, artifacts, links, and audit records in PostgreSQL. The frontend reads the resulting catalog data through `/api/v1` endpoints. + +| Site | Ingestion mode | Source archive location | +| -------------------- | ------------------------- | ---------------------------------------------------------------------- | +| NERSC / Perlmutter | Path reference | `/global/cfs/projectdirs/e3sm/performance_archive` | +| LCRC / Chrysalis | Archive upload | `/lcrc/group/e3sm/PERF_Chrysalis/performance_archive` | +| Additional HPC sites | Archive upload by default | Site-specific `performance_archive` path, packaged by the ingestion job | + +## Daily Workflow + +Common tasks beyond the initial setup: + +```bash +make backend-run # start backend with hot reload +make frontend-run # start frontend with hot reload +make backend-test # run full backend test suite +make backend-seed # seed the database with sample data +make backend-rollback-seed # remove seeded data +make backend-upgrade # apply pending Alembic migrations +make backend-downgrade rev= # roll back to a specific Alembic revision +make backend-reset # recreate the backend venv and reinstall deps +make frontend-lint # lint frontend +make frontend-fix # lint frontend with auto-fix +make pre-commit-run # run all pre-commit hooks +``` + +To reset the database completely, stop the backend, bring down the Docker container, remove the volume, then re-run setup: + +```bash +docker compose -f docker-compose.local.yml down -v +make setup-local +``` + +To run a single backend test file or test function: + +```bash +cd backend +uv run pytest tests/path/to/test_file.py +uv run pytest tests/path/to/test_file.py::test_function_name +``` + +## Making a Change — Walkthrough + +### Backend example: add a new API field + +1. **Read** the relevant feature code under `backend/app/features/` and the corresponding test file under `backend/tests/`. +2. **Edit** the schema, model, or endpoint as needed. +3. **Migrate** if the change touches the database schema: + + ```bash + make backend-migrate m='add field_name to table_name' + make backend-upgrade + ``` + +4. **Test**: + + ```bash + make backend-test + ``` + +5. **Validate** with pre-commit before committing: + + ```bash + make pre-commit-run + ``` + +6. **Commit and push**, then open a PR per [CONTRIBUTING.md](../../CONTRIBUTING.md). + +### Frontend example: update a feature component + +1. **Read** the feature module under `frontend/src/features/` and its API/hooks directories. +2. **Edit** the component, hook, or API call. +3. **Lint and type-check**: + + ```bash + make frontend-lint + pnpm --dir frontend run type-check + ``` + +4. **Validate** with pre-commit: + + ```bash + make pre-commit-run + ``` + +5. **Commit and push**, then open a PR. + +Key rule: feature modules must not import from other feature modules. If you need to share code between features, move it to `frontend/src/components/shared/` or `frontend/src/lib/`. + +## Troubleshooting + +**Docker not running or port conflict** +`make setup-local` starts PostgreSQL via Docker Compose. If Docker Desktop is not running, or port 5432 is already in use, the setup will fail. Start Docker Desktop and stop any conflicting services first. + +**Missing environment variables** +If the backend fails to start with config or env errors, regenerate the env files: + +```bash +make setup-local-assets +``` + +This copies `.envs/example/*` into `.envs/local/` without overwriting existing files. + +**SSL / certificate errors** +The backend and frontend use local TLS certs from `certs/`. If they are missing or expired, regenerate them: + +```bash +make gen-certs +``` + +Your browser will show a self-signed certificate warning — this is expected for local development. + +**`uv` or `pnpm` not found** +The backend uses `uv` (not pip) and the frontend uses `pnpm` (not npm/yarn). Both must be on your `PATH`. See the [Prerequisites](#local-setup) section for install links. + +**Pre-commit fails or gives inconsistent results** +Always run pre-commit from the repository root, not from `backend/` or `frontend/`. Some hooks (e.g., `mypy`) depend on root-relative config paths. + +```bash +# correct +make pre-commit-run + +# incorrect — may produce wrong results +cd backend && uv run pre-commit run --all-files +``` + +**Frontend ESLint error about cross-feature imports** +Feature modules under `frontend/src/features/*/` must not import from other features. This is enforced by `eslint-plugin-boundaries`. Move shared code to `frontend/src/components/shared/` or `frontend/src/lib/`. + +**Database out of sync after pulling new changes** +If a teammate added a migration, apply it: + +```bash +make backend-upgrade +``` + +If the schema diverged significantly, reset the database entirely: + +```bash +docker compose -f docker-compose.local.yml down -v +make setup-local +``` + +## Contributing + +See [CONTRIBUTING.md](../../CONTRIBUTING.md) for issue, branch, commit, and PR expectations. + +Key habits for safe changes: + +- read the touched feature before editing it +- keep frontend feature boundaries intact (`eslint-plugin-boundaries` enforces this) +- update backend tests when behavior changes +- add Alembic migrations when schema changes +- run `make pre-commit-run` from the repository root, not from subdirectories + +## Where Important Details Live + +- backend service detail: [backend/README.md](../../backend/README.md) +- frontend service detail: [frontend/README.md](../../frontend/README.md) +- docs index: [docs/README.md](../README.md) +- CI/CD and deployment docs: [docs/cicd/README.md](../cicd/README.md) diff --git a/docs/hpc_api_token_authentication.md b/docs/hpc_api_token_authentication.md index ef3b5e97..b3e0fb4e 100644 --- a/docs/hpc_api_token_authentication.md +++ b/docs/hpc_api_token_authentication.md @@ -1,5 +1,7 @@ # API Token Authentication Guide +Audience: maintainers configuring service-account and token-based ingestion for HPC or privileged automation. + ## Overview API token authentication enables secure programmatic access to ingestion endpoints from external HPC systems without requiring browser-based GitHub OAuth flows. diff --git a/frontend/README.md b/frontend/README.md index 61f8ab3e..85688cfd 100644 --- a/frontend/README.md +++ b/frontend/README.md @@ -1,68 +1,50 @@ # SimBoard Frontend -The **SimBoard Frontend** is a modern web application built with **React**, **TypeScript**, and **Vite**. +The frontend is a React single-page application for browsing, comparing, and uploading E3SM simulation metadata. -It provides the user interface for browsing, comparing, and analyzing **E3SM** (Energy Exascale -Earth System Model) simulation metadata. +## Responsibilities -## Tech Stack +- route composition and page-level UI +- case and simulation browsing +- side-by-side comparison +- authenticated upload flow +- cookie-based browser auth integration -- **React 19** — Core UI library -- **TypeScript** — Type-safe development -- **Vite 6** — Lightning-fast build and dev environment -- **Tailwind CSS** + **shadcn** — Styling and components -- **ESLint + Prettier** — Code linting, formatting, and architectural enforcement -- **pnpm** — Dependency management +## Architecture Notes -## Development Guide +The frontend uses feature-based organization and enforces import boundaries with ESLint. -For the development guide, see the [root README.md file](../README.md). -It includes information on how to get the frontend service started via bare-metal or Docker. +- `src/routes/` composes top-level routes. +- `src/features/*/` contains feature modules such as `browse`, `simulations`, `compare`, and `upload`. +- `src/features/*/api/` contains feature-specific API calls. +- `src/features/*/hooks/` contains feature-specific hooks. +- `src/components/shared/` is for reusable shared components. +- `src/components/ui/` is for lower-level UI primitives. -## Architecture +Feature modules should not import directly from other feature modules. This is enforced at lint time by `eslint-plugin-boundaries` — if you see an ESLint error about an invalid cross-feature import, move the shared code to `src/components/shared/` or `src/lib/`. -This frontend follows a **feature-based architecture** enforced by **ESLint architectural boundaries**. +## Important Locations -### Feature Organization +```text +frontend/src/routes/ top-level route composition +frontend/src/features/browse/ run browser and filters +frontend/src/features/simulations/ + cases, runs, and detail pages +frontend/src/features/compare/ side-by-side comparison UI +frontend/src/features/upload/ authenticated archive upload flow +frontend/src/auth/ auth provider, callback, protected routes +frontend/src/components/ layout, shared, and UI components +frontend/src/api/ shared Axios client and auth-state glue +``` -- **Features are the primary unit of organization** (e.g. `features/browse`, `features/upload`) -- Domain features such as **`simulations`** and **`machines`** represent core application data -- UI-oriented features (browse, compare, home) may depend on domain features -- **Features must not depend on other features directly** -- API logic lives under `features/*/api` -- Feature-specific hooks live under `features/*/hooks` -- Shared components must be genuinely reusable and belong under `components/shared` +## Developer Commands -### Architectural Boundaries (ESLint) +Run these from the repo root: -The project uses **`eslint-plugin-boundaries`** to enforce architectural rules at import time. +```bash +make frontend-run +make frontend-lint +pnpm --dir frontend run type-check +``` -Each file is classified into a single architectural layer based on its path: - -- **`routes`** — Application routing and top-level composition -- **`feature`** — Feature modules (browse, upload, compare, etc.) -- **`ui`** — Design-system primitives and low-level UI components -- **`shared`** — Reusable composite components -- **`lib`** — Generic utilities and helpers -- **`types`** — Domain and API contract types -- **`api`** — API clients and adapters - -#### Dependency Rules - -- **Features are isolated** - Features may not import or depend on other features directly. - -- **Routes compose the application** - Routes may import features, shared/UI components, and domain types. - -- **UI remains presentation-only** - UI components may depend only on utilities and types. - -- **Types are globally safe** - Type definitions may be imported from any layer. - -Any import that violates these rules is reported as an ESLint error, preventing invalid architectural dependencies from being introduced. - -## License - -For license information, see the [root LICENSE file](../LICENSE). +For repo-wide setup and contributor workflow, see [docs/developer/README.md](../docs/developer/README.md).