GPU-native computational fluid dynamics, OpenFOAM-compatible, fully resident on the GPU.

Brae keeps the whole CFD solve on one GPU. The mesh, the fields, and every linear solve stay on the device from the first iteration to the last, with no per-iteration copies to the CPU. Point it at an existing OpenFOAM case and it writes standard OpenFOAM results, validated cell-by-cell, so it drops into your workflow unchanged.

Up to 30x faster than OpenFOAM's own GPU acceleration (AMGX, PETSc) on an H100, because brae keeps the whole SIMPLE loop on the device instead of offloading only the linear solve.

---

⚡ How fast

On one H100, at matched accuracy (under 1% on the fields):

• 26-30x faster than OpenFOAM's own GPU offloads (AMGX, PETSc)
• 3.9x faster than the SPUMA OpenFOAM-GPU port
• 2.5x faster than a 24-core CPU node

The chart is GB10, the conservative baseline: with no HBM its GPU shares the CPU's memory, so there it only reaches about 5x over the offloads and parity with the CPU. On the H100's HBM the same code widens to 26-30x. Not bit-identical to OpenFOAM (the GPU reorders the floating-point sums), and not meant to be. Full method and numbers: docs/performance.md and the H100 report.

---

🎯 Same case, same answer

motorBike, 2.9M cells, k-omega SST. brae, OpenFOAM on the CPU, its AMGX and PETSc GPU offloads, and the SPUMA port produce a visually identical surface-pressure field and agree to ~1.6% on drag.

brae (Blackwell GPU)	OpenFOAM (Grace CPU)
OpenFOAM + AMGX (GPU)	OpenFOAM + PETSc (GPU)
SPUMA (OpenFOAM-GPU port)

See the full five-way comparison for the drag and lift numbers.

---

💡 Why brae

Most "OpenFOAM on GPU" approaches offload only the linear solver: the matrix is rebuilt on the CPU and copied to the GPU every iteration, and assembly, momentum, and turbulence still run on one CPU core. Brae is device-resident, the entire loop lives on the GPU, so there is no migration tax and no serial-CPU ceiling.

• Drop-in, reads your existing 0/ constant/ system/ case (ASCII or binary mesh) and writes standard time directories for ParaView / postProcess.
• Faithful, a clean-room reimplementation of OpenFOAM v2412, validated cell-by-cell (sub-1% on the fields).
• Fast where it counts, it pulls ahead as the mesh grows, and the lead scales with the GPU's memory bandwidth.

See docs/memory-model.md for the data-layout rationale (device pool over pinned/thrust, LDU-gather over CSR/cuSPARSE).

---

📦 Install

Needs an NVIDIA GPU (Ampere or newer, including H100 / GH200 / B200), CUDA 12.4+ (13.x recommended), and a C++17 toolchain. Brae is standard CUDA, so a newer architecture is just a recompile.

# deps: cmake >= 3.24, CUDA toolkit, an MPI (OpenMPI), SCOTCH, zlib
git clone https://github.com/simd-ai/brae.git
cd brae
cmake -B build -DCMAKE_CUDA_ARCHITECTURES=<your_arch>
cmake --build build -j --target brae

Set <your_arch> to your GPU's compute capability

GPU	<your_arch>
GB10	121
RTX 50-series	120
GB300 / B300	103
B200 / GB200	100
H100 / GH200	90
RTX 40-series / L40	89
RTX 30-series	86
A100	80

---

🚀 Get started

Run brae from inside any OpenFOAM simpleFoam case, exactly as you would run simpleFoam itself:

cd yourCase                       # your OpenFOAM case (0/  constant/  system/)
brae                              # solve in the current directory
brae -case /path/to/yourCase      # or run it from anywhere
brae -partition -case yourCase    # optional: cache the mesh + AMG once, then later runs start warm
brae --help                       # all options

No decomposePar, brae auto-partitions for the GPU. The fast path (device-resident solver + mixed-precision multigrid) is on by default; opt out with BRAE_PCG_DEVICE=0 BRAE_AMG_FP32=0.

Run several cases across GPUs

Run a mesh-independence study or a parameter sweep, one case per GPU (extras queue as GPUs free up):

brae -cases mesh_coarse mesh_medium mesh_fine   # one case per GPU
BRAE_JOBS=2 brae -cases caseA caseB caseC       # cap how many run at once

Each case's residual output is tagged [GPUn case], and it ends with a per-case summary. On a single GPU the cases run back to back. Override the detected GPU count with BRAE_GPUS.

---

🌊 Solvers

Brae implements OpenFOAM's incompressible solvers one at a time, each fully device-resident and validated cell-by-cell. Today it runs simpleFoam (steady incompressible); more to follow.

---

🚧 Roadmap

See the roadmap for the current scope and what is coming next.

---

📚 Documentation

• Getting started, install, first run, verifying against OpenFOAM
• Performance & tuning, the BRAE_* knobs and benchmarks
• Memory model, device pool vs pinned, LDU-gather vs CSR
• Roadmap, scope and what is coming next