Tutorial#61
Closed
mawolf2023 wants to merge 58 commits into
Closed
Conversation
Source code, pre-trained model, configs, and tests for the quantum pre-decoder (surface-code memory circuits). Migration from GitLab: - Removed GitLab CI (.gitlab-ci.yml) and personal .zshrc - Removed empty code/qec/color_code/ directory (no source files) - Removed verify_he_from_experimental.sh (referenced GitLab branch) - Cleaned all __pycache__/ directories GitHub platform setup: - Added GitHub Actions CI (.github/workflows/ci.yml) with CPU and GPU jobs - Added SPDX header to .github/copy-pr-bot.yaml - Model file (models/*.pt) tracked via Git LFS (.gitattributes) - Updated .gitignore for Python/GitHub conventions README updates: - Fixed repo root reference - Removed references to non-existent PYTHON_COMPATIBILITY.md, Makefile, and scripts/test.sh - Replaced outdated "phase 1" CI section with GitHub Actions table Verified locally: - SPDX header check passes - 195 unit tests pass (CPU and GPU) Made-with: Cursor
- Triggers: add pull-request/[0-9]+ (copy-pr-bot), merge_group, workflow_dispatch; add concurrency group - GPU jobs: run inside ubuntu:22.04 container on linux-amd64-gpu-h100-latest-1 runners with --shm-size 16g, NVIDIA_VISIBLE_DEVICES passthrough, nv-gha-runners/setup-proxy-cache, and manual git/git-lfs install (matching NVIDIA/warp pattern) - Guard GPU jobs with repository check - Add timeout-minutes to GPU jobs Made-with: Cursor
Self-hosted GPU runners are not yet registered for this repo, so GPU jobs were blocking CI indefinitely. Split into two workflows: - ci.yml: CPU-only jobs that gate every push/PR (fast, always available) - gpu.yml: GPU jobs via workflow_dispatch with configurable runner label; push trigger is commented out and ready to enable once runners are set up The gpu.yml runner label defaults to linux-amd64-gpu-h100-latest-1 (matching NVIDIA/warp) and can be overridden from the dispatch UI. Made-with: Cursor
Revert the GPU/CPU split. All jobs (CPU + GPU) live in ci.yml and trigger on main, pull-request/[0-9]+ (copy-pr-bot), merge_group, and workflow_dispatch. GPU jobs follow the NVIDIA/warp pattern: run inside ubuntu:22.04 container on linux-amd64-gpu-h100-latest-1 runners with shm-size 16g, NVIDIA_VISIBLE_DEVICES passthrough, and nv-gha-runners/setup-proxy-cache. Made-with: Cursor
Use the actual runner available to this repo (RTX PRO 6000, runner group nv-gpu-amd64-rtxpro6000-1gpu) instead of the H100 label which is not shared with this repository. Made-with: Cursor
At d=13 with 262k Monte Carlo shots and LER ~2e-4, the per-basis standard error is ~2.8e-5. Comparing two independent estimates (pre-decoder vs baseline) gives a combined SE of ~4e-5, making the old 5e-5 tolerance a ~1.2-sigma bound that flakes regularly in CI. 1e-4 provides a ~2.5-sigma guard: stable in CI while still catching any real regression where the pre-decoder worsens LER. Made-with: Cursor
Per-basis delta: 1e-4 -> 2e-4 (range [0, 4e-4] around expected 2e-4). Average delta: 1e-4 -> 1.5e-4. With 262k shots and LER ~2e-4, per-basis SE is ~2.8e-5. The old 1e-4 delta put the upper bound at 3e-4, which is only ~3.6 sigma from the mean — too tight for stable CI. The new bounds give ~7-sigma headroom while still catching genuine regressions (e.g. LER jumping to 4e-4+). Made-with: Cursor
The d=9 code distance has ~5-10x higher LER than d=13, so Monte Carlo variance is proportionally larger. The global tolerance of 1e-4 is too tight for d=9 — observed flake: LER_after 0.001328 vs baseline 0.001190 (difference 1.38e-4, just above 1e-4). Setting d=9-specific tolerance to 5e-4, which provides comfortable headroom while still catching real regressions (a genuine degradation would push LER well above baseline). Made-with: Cursor
* Add pre-trained model checkpoints for r=13 and r=9 decoders Add LFS-tracked model files: - PreDecoderModelMemory_r13_v1.0.86.pt (code rate r=13) - PreDecoderModelMemory_r9_v1.0.77.pt (code rate r=9) Made-with: Cursor * chore: retrigger CI (empty commit) Made-with: Cursor
* fix(test): relax flaky LER boundary-detector assertion to assertLessEqual With only 2000 CI samples at p=0.002, discrete error counts often coincide for "with BD" and "without BD" circuits, causing assertLess to fail spuriously. Use assertLessEqual, consistent with the sibling test_ler_improves_with_bd_all_orientations. Made-with: Cursor * fix(test): increase CI sample count from 2k to 20k for LER comparison 2000 samples at p=0.002 produced only ~5 logical errors, making ties and reversals likely. 20000 samples yields ~50 errors — enough for reliable statistical separation while adding negligible CI time. Made-with: Cursor * fix(test): restore original assertLess assertion Keep the strict assertLess check — the increased sample count (20k) provides enough statistical power to reliably distinguish LER values. Made-with: Cursor * fix(test): widen d=13 LER improvement tolerance from 1e-4 to 2e-4 The previous 1e-4 tolerance gave only ~2.5-sigma headroom with combined SE ~4e-5, resulting in ~1.2% per-run flake probability. Widen to 2e-4 (~5 sigma) to eliminate CI flakes while still catching real regressions. Made-with: Cursor * ci: trigger required status checks Made-with: Cursor
… matrix (#8) * Consolidate CI test jobs: merge GPU smoke test and add Python version matrix - Remove separate smoke-test-gpu job (was serial after gpu-tests, increasing pipeline time). Smoke training+inference now runs in the same gpu-tests job. - Replace python-compat matrix (6 jobs, SKIP_TESTS=1) with two focused job groups that actually run tests: * gpu-tests: matrix over Python 3.11/3.12/3.13 on GPU runners — installs train deps, runs full test suite (CPU+GPU), then smoke training+inference. * inference-tests: matrix over Python 3.11/3.12/3.13 on CPU — installs inference deps, runs tests with pre-trained models (GPU tests auto-skip). Reduces total jobs from 11 to 9 while increasing actual test coverage. Made-with: Cursor * Fix GPU CI: set DEBIAN_FRONTEND=noninteractive to prevent tzdata hang The deadsnakes PPA pulls in tzdata as a dependency, which triggers an interactive timezone configuration prompt in the container. This caused all 3 GPU matrix jobs to hang for 45 minutes until timeout. Made-with: Cursor * Add pull_request trigger and gate GPU jobs to push/merge_group only Without the pull_request trigger, CI never fires on PRs — checks aren't even planned (e.g. PR #9 shows zero checks). GPU jobs are gated to push/merge_group events to avoid consuming self-hosted GPU runners on every PR update. Made-with: Cursor * Remove event gate on GPU jobs so they run on PRs too GPU jobs complete in ~5-10 minutes and serve as a useful pre-merge check. Made-with: Cursor * Remove pull-request/[0-9]+ from push trigger to fix duplicate CI runs The copy-pr-bot creates pull-request/N branches for each PR, which matched the push trigger and caused every CI job to run twice (once from pull_request, once from push). The pull_request trigger already covers PRs targeting main, so the push pattern is redundant. Made-with: Cursor * Fix GPU CI: gate on event type, restore push trigger for copy-pr-bot NVIDIA self-hosted runners block pull_request events outright. GPU CI must run via push events — either to main or to pull-request/[0-9]+ branches created by copy-pr-bot for PR testing. - Restore "pull-request/[0-9]+" in push trigger - Gate gpu-tests with if: github.event_name != 'pull_request' - CPU jobs (inference-tests, unit-tests, etc.) still run on pull_request Made-with: Cursor * Remove pull-request/[0-9]+ push pattern and pull_request gate on GPU jobs Simplify triggers: all jobs (including GPU) run on pull_request, push to main, and merge_group. The pull-request/[0-9]+ branch convention is not used by contributors. Made-with: Cursor * Merge unit-tests + inference-tests, gate GPU jobs from pull_request - Combine unit-tests (py3.12) and inference-tests (py3.11/3.12/3.13) into a single unit-tests matrix job across all three Python versions. Both ran identical test suites with inference requirements. - Re-add if: github.event_name != 'pull_request' on gpu-tests since NVIDIA self-hosted runners block pull_request events entirely. GPU CI runs on push to main and merge_group. Made-with: Cursor * Split GPU tests into separate workflow to avoid skipped PR noise NVIDIA self-hosted runners block pull_request events, so GPU jobs in the main CI workflow always showed as a single "Skipped" entry with unresolved matrix names on every PR. Move GPU jobs to ci-gpu.yml (triggers: push to main, merge_group, workflow_dispatch). The main ci.yml keeps CPU jobs only (triggers: pull_request, push to main, merge_group, workflow_dispatch). Made-with: Cursor * Enable GPU CI on PRs via copy-pr-bot push trigger Add pull-request/[0-9]+ to ci-gpu.yml push trigger so GPU tests run when copy-pr-bot creates the corresponding branch for a PR. Made-with: Cursor * Fix smoke test step: use bash shell for source command The container default shell is sh, which doesn't have the source builtin. Explicitly set shell: bash for the venv activation step. Made-with: Cursor * Install gcc in GPU container for torch.compile/inductor The smoke training step uses torch.compile which invokes the inductor backend, requiring a C compiler. The ubuntu:22.04 container doesn't ship with gcc. Made-with: Cursor * Switch CPU jobs to NVIDIA self-hosted linux-amd64-cpu4 runners Use nv-cpu-general runner group instead of GitHub-hosted ubuntu-latest. Also restore pull-request/[0-9]+ push trigger in case self-hosted CPU runners block pull_request events (same as GPU runners). Made-with: Cursor * Remove pull_request trigger since all runners are NVIDIA self-hosted NVIDIA self-hosted runners block pull_request events. All CI (CPU and GPU) now runs via copy-pr-bot push to pull-request/[0-9]+ branches. Made-with: Cursor
…els (#5) * Remove legacy PreDecoderModelMemory_v1.0.94.pt, migrate to r9/r13 models Remove the old single-model file and update all code and tests to use the two receptive-field-specific pre-trained models: - PreDecoderModelMemory_r9_v1.0.77.pt (R=9, model_id=1) - PreDecoderModelMemory_r13_v1.0.86.pt (R=13, model_id=4) Changes: - git rm models/PreDecoderModelMemory_v1.0.94.pt - run.py: broaden find_best_model prefix to PreDecoderModelMemory_*, scan directory for checkpoint files instead of fixed filename pattern - README.md: update pre-trained model docs for both shipped models - test_inference_public_model.py: rewrite to cover both r9 and r13 models with d=9/d=13 evaluation combinations - test_ler_pretrained_models.py: iterate over both models by default, infer model_id from receptive field in filename - measure_d9_ler.py: switch to r9 model (checkpoint 77) Made-with: Cursor * Remove package-users section from README Users will get repo access directly in the next release, so the separate distribution-channel instructions are no longer needed. Made-with: Cursor
Configure .style.yapf (Google base + dedent_closing_brackets, split_before_closing_bracket, column_limit=100), apply formatting to all 64 Python files, and add a yapf-check CI job so future PRs are automatically validated. Made-with: Cursor
* Add CI test tiers: mid-running and long-running GPU tests Add a three-tier test model (short / mid / long) adapted from GitLab MR !19: Mid-tier (ci.yml, ~5-10 min, every push): - New mid-gpu-tests job: extended smoke with 32k train samples, 2 epochs, validates training convergence beyond the minimal 4k-sample smoke test. Long-tier (long-running-tests.yml, daily schedule + manual dispatch): - statistical-noise-model: RUN_SLOW=1, 100k+ shot noise model tests - orientation-inference: inference over all 4 orientations (O1-O4) - ler-regression: LER quality at d=9 and d=13 with pre-trained models - full-epoch-training: 1 epoch with 2M samples + LER validation Helper scripts: - run_tests_tier.sh: local convenience runner for short/mid/long tiers - run_orientations_long.sh: runs train or inference for all 4 orientations Training sweep scripts (Slurm automation) deferred to a separate PR. Made-with: Cursor * Add GPU unit tests and refine CI test tiers - New test_gpu.py: 19 GPU-gated unit tests covering DEM sampling, MemoryCircuitTorch, QCDataGeneratorTorch, PreDecoder v1/v2 forward pass (incl. gradient flow and mixed-precision), HE kernel, oracle residuals, memory-leak detection, and CPU/GPU equivalence on CUDA. All tests skip cleanly on CPU-only runners. - ci-gpu.yml: add mid-gpu-tests job (32k train samples, 2 epochs, Python 3.13) gated to main-only; add gpu-coverage job that uploads GPU-specific coverage artifacts. - long-running-tests.yml: add check-for-changes gate so the daily schedule skips when main had no commits in the last 24h; pin all four long jobs to Python 3.13 via deadsnakes venvs. Coverage impact (with GPU): generator_torch 85%→90%, memory_circuit_torch 72%→75%, +1 line in HE torch. Made-with: Cursor * ci: use ubuntu:24.04 in GPU and long-running workflow containers Made-with: Cursor * ci: LER checks for training jobs, drop smoke naming, fix gpu-coverage venv - Add check_ler_from_log.py to assert validation LER from training logs - Short/mid/full-epoch training steps: tee log + LER threshold (0.25/0.15/0.1) - Rename CI steps (Training + inference with LER check, etc.); update README - smoke_run.sh: echo Short training/inference; keep script name for compatibility - gpu-coverage: use venv to avoid PEP 668 externally-managed-environment on Ubuntu 24.04 Made-with: Cursor * ci/docs: orientation-inference LER output check; drop smoke wording - orientation-inference: assert log has 4 'LER - Avg' blocks (one per orientation) - Rename ci_smoke -> ci_short, default EXPERIMENT_NAME short; reword smoke in tests/docs - README: quick short runs, gpu-tests row; inference.py comment; run_tests_tier help - test_noise_model: Fast/Slow tier; test_oracle: stability check; test_homological: quick/basic check Made-with: Cursor * ci: relax LER thresholds to reduce flakiness (short 0.35, mid 0.2, long 0.15) Made-with: Cursor * style: apply yapf to check_ler_from_log.py and test_gpu.py Made-with: Cursor * ci(gpu): more samples and 2 epochs for short tier to stabilize LER (e.g. py3.11) Use 8192 train / 1024 val, test and 2 epochs instead of 4096/512 and 1 epoch. Keeps max-ler 0.35; avoids flaky high LER on some platforms. Made-with: Cursor * ci(gpu): short tier 16k train / 2k val for more stable LER Made-with: Cursor
…et) (#2) * feat(qec): surface-code noise model upscaling for training (6e-3 target) Replace the old 1e-3 "sparsity guard" in train.py with a proper noise-model upscaling module in noise_model.py that: - Scales all 25 noise-model parameters so max(grouped totals) = 6e-3 (just below surface-code threshold ~7.5e-3) for training data only. - Skips upscaling for non-surface-code types. - Provides skip_noise_upscaling config flag and PREDECODER_SKIP_NOISE_UPSCALING=1 env var to bypass upscaling. - Emits clear warnings when noise is above target or downscale is skipped. Adds 10 unit tests covering upscale, downscale, skip, non-surface-code, zero-totals, and reference-preservation scenarios. Updates README with detailed documentation of the upscaling rules and how to disable them. Ported from gitlab MR !33 (feature/surface-code-noise-upscale-6e3). Made-with: Cursor * Apply yapf style to rebased branch Made-with: Cursor
PIPESTATUS is bash-specific; ubuntu:24.04 containers default to sh, causing "Bad substitution" on the mid-gpu-tests job. Add shell: bash to all affected steps across ci-gpu.yml and long-running-tests.yml.
* Add fp16/fp32 SafeTensors export and inference loading support Signed-off-by: Igor Baratta <ialmeidabara@nvidia.com> * Fix SafeTensors model_id auto-detection from metadata * SafeTensors PR: fix CI tests, docs, and checkpoint resolution - Restore test_inference_public_model to use r9/r13 models (v1.0.77, v1.0.86) instead of retired v1.0.94; fix test_ler_pretrained_models and measure_d9_ler - Restore run.py checkpoint resolution to accept PreDecoderModelMemory_r*_v1.0.*.pt - Add README section for .pt to .safetensors export and PREDECODER_SAFETENSORS_CHECKPOINT - Update export script docstring and local_run.sh with SafeTensors examples - Document export package in __init__.py; revert test_boundary_detectors LER samples Made-with: Cursor * Review fixes: weights_only, fp16 load order, README, tests - Add weights_only=False to torch.load in checkpoint_to_safetensors.py to silence PyTorch future warning and make intent explicit - Move model.half() before load_state_dict in safetensors_utils.py to avoid unnecessary fp32 round-trip when loading fp16 weights - Fix README bash example: replace invalid [--fp16] syntax with two separate fp16/fp32 examples - Add code/tests/test_safetensors_export.py: 5 unit tests covering fp32/fp16 round-trips, auto model_id detection, and error cases * docs: replace hardcoded safetensors filenames with generic training output paths SafeTensors export is optional and post-training; point examples at outputs/<EXPERIMENT_NAME>/models/ instead of hardcoded public model names. Remove misleading hardcoded example from local_run.sh, add a generic note. * style: apply yapf to checkpoint_to_safetensors.py * test: add _load_model integration tests for PREDECODER_SAFETENSORS_CHECKPOINT Adds TestSafeTensorsRunPyIntegration to test_safetensors_export.py covering the PREDECODER_SAFETENSORS_CHECKPOINT env var code path in workflows/run.py: - fp32 .safetensors loads correctly and weights match - fp16 .safetensors loads correctly and cfg.enable_fp16 is set - empty env var falls through to the normal .pt checkpoint path --------- Signed-off-by: Igor Baratta <ialmeidabara@nvidia.com> Co-authored-by: Igor Baratta <ialmeidabara@nvidia.com>
* Add Docker-based training infrastructure and cluster scripts
Dockerfile, SLURM sbatch template, and supporting shell scripts for
running pre-decoder training on remote GPU nodes (Docker, bare-metal,
or SLURM). Includes two production training configs (R=9, R=13),
PREDECODER_LR_MILESTONES env override in train.py, and comprehensive
TRAINING.md documentation.
Made-with: Cursor
* Fix training script portability and documentation issues
- sbatch_train.sh: resolve REPO_ROOT from script location, not $(pwd)
- sbatch_train.sh: consolidate PREDECODER_DISABLE_SDR/TORCH_COMPILE defaults
so Docker and bare-metal paths behave identically
- sbatch_train.sh: log message before chmod 1777; add --nodes=1 to multi-GPU examples
- cluster_install_deps.sh: arch-aware Miniconda URL (supports aarch64/ARM)
- cluster_install_deps.sh: single TORCH_CUDA default (remove redundant fallback)
- TRAINING.md: document SHARED_LOG_DIR; correct cluster defaults for SDR/compile vars
- conf/config_qec_decoder_r{9,13}_fp8.yaml: note that training hyperparams come
from internal defaults, point to config_public.yaml
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
ler-regression: switch from dotted module path to discover PYTHONPATH=code python -m unittest code.tests.test_inference_public_model fails on Python 3.13 because the local `code/` directory has no `__init__.py`, so Python resolves `code` to the stdlib `code` module (which has no `tests` attribute) rather than the local package. Switch to the same `discover -s code/tests` form already used by the passing `statistical-noise-model` job, which avoids the ambiguous dotted import path entirely. orientation-inference: add training step before inference The job ran `ORIENTATIONS_LONG_TASK=inference` on a fresh runner with no pre-existing checkpoints. `local_run.sh` defaults FRESH_START=0, which sets `++load_checkpoint=True`, causing a FileNotFoundError. Add a training step first (FRESH_START=1, small sample counts to fit within the 90-minute timeout) so that checkpoints exist when inference runs. Fixes: https://github.com/NVIDIA/quantum-predecoder/actions/runs/22884983829 Signed-off-by: Ivan Basov <ibasov@nvidia.com>
* [ci] Check for large files Signed-off-by: Ben Howe <bhowe@nvidia.com> * Add copyright Signed-off-by: Ben Howe <bhowe@nvidia.com> --------- Signed-off-by: Ben Howe <bhowe@nvidia.com>
…ding (#14) * feat(onnx): add QUANT_FORMAT int8/fp8 PTQ via modelopt.onnx - Add _collect_calibration_dets module-level helper that samples detector inputs from the inference dataloader for ONNX calibration - Parse QUANT_FORMAT env var (int8, fp8) in OnnxWorkflow export path; invalid values are ignored with a warning - Two-step export: always write FP32 ONNX first, then optionally apply modelopt.onnx.quantization.quantize() for the requested format - fp8 is fail-fast on error; int8 silently falls back to FP32 ONNX - Add QUANT_CALIB_SAMPLES env var (default 256) to control calibration sample count - Add test_onnx_quant_workflow.py: 13 CPU-only unit tests covering the calibration helper and QUANT_FORMAT routing logic * fix(onnx): re-derive engine_path from final onnx_path after quant fallback * review: fix run.py, temp file cleanup, YAPF, README ONNX section - run.py: remove emoji from print statements (style inconsistency) - run.py: remove no-op torch.compile(disable=True) calls - run.py: extract _resolve_dir() helper to replace 4 copies of the current_file/project_root path resolution pattern - run.py: replace bare torch.load/load_state_dict with _load_state_dict_from_pt() which handles model_state_dict/state_dict/bare-dict formats and strips the DDP "module." prefix — consistent with checkpoint_to_safetensors.py - tests: add addCleanup(os.unlink) for all NamedTemporaryFile paths - YAPF: reformat logical_error_rate.py and test_onnx_quant_workflow.py - README: add ONNX export and quantization section documenting ONNX_WORKFLOW modes, QUANT_FORMAT, QUANT_CALIB_SAMPLES Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * quantize only CNN layers * fix(ci): YAPF, move nvidia-modelopt to train reqs, add prerequisite tests - YAPF: reformat 3 long lines in logical_error_rate.py introduced by the "quantize only CNN layers" commit (d7b8217) - Move nvidia-modelopt[onnx] from requirements_public_inference.txt to requirements_public_train.txt; it is only needed for ONNX PTQ export (QUANT_FORMAT env var), not for pure inference, and has no Python 3.13 build — keeping it in inference reqs broke unit-tests/py3.13 in CI - Add python_version<"3.13" marker so the CI train matrix installs it on supported Python versions without failing on 3.13 - Add TestModeloptPrerequisite in test_onnx_quant_workflow.py: - asserts nvidia-modelopt is declared in requirements_public_train.txt - asserts it is absent from requirements_public_inference.txt - conditionally checks the import is resolvable when the package is present (skipped on Python 3.13+ and when not installed) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(onnx): add onnxruntime INT8 fallback for Python 3.13+ nvidia-modelopt does not support Python 3.13+. Add a conditional backend dispatch so QUANT_FORMAT=int8 works on all supported Python versions: - Add _ort_quantize_int8() module-level helper that uses onnxruntime.quantization.quantize_static() with QDQ/QInt8 format and a CalibrationDataReader wrapping the pre-collected calib_dets array - In the quantization block, branch on sys.version_info >= (3, 13): - Python 3.13+: call _ort_quantize_int8(); raise immediately for FP8 (no viable 3.13-compatible FP8 PTQ library available) - Python <3.13: keep existing modelopt path unchanged - Add onnxruntime (python_version >= "3.13") to requirements_public_train.txt - Expand TestOrtQuantizeInt8 tests: - round-trip test (build tiny Gemm ONNX, quantize, validate) on 3.13+ - dispatch test verifying _ort_quantize_int8 is called on 3.13+ - FP8-on-3.13 raises RuntimeError - Expand TestModeloptPrerequisite: assert onnxruntime appears in train requirements and both quant packages are absent from inference requirements Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(prereqs): document tensorrt as optional GPU dep, add fallback tests tensorrt is a heavy CUDA-only SDK (~500 MB) that cannot be pip-installed in CPU-only CI, so it is not added as an active pip requirement. Instead: - Add a comment block in requirements_public_inference.txt documenting tensorrt as an optional prerequisite for ONNX_WORKFLOW=2/3 paths, with the install command and a note about graceful fallback - Add test_tensorrt_fallback.py with three test classes: - TestTensorrtDocumented: asserts the requirements comment exists and tensorrt is NOT an active pip requirement - TestTensorrtFallback: verifies both TRT import sites (USE_ENGINE_ONLY and EXPORT_AND_USE_TRT) set trt_context=None on ImportError and do not propagate the exception to the caller - TestTensorrtImportable: checks key TRT symbols (Logger, Runtime, Builder, BuilderFlag, LayerInformationFormat) when tensorrt is installed; skipped silently on CPU-only environments Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(onnx): use import-based dispatch for modelopt/ort; install modelopt on py3.13 nvidia-modelopt works on Python 3.13 when installed with --ignore-requires-python (confirmed by modelopt maintainers). - logical_error_rate.py: replace sys.version_info dispatch with an ImportError-based dispatch — try modelopt first (INT8+FP8), fall back to _ort_quantize_int8 only when modelopt is not importable; FP8 raises RuntimeError with the --ignore-requires-python install hint - check_python_compat.sh: after the main requirements install, re-install nvidia-modelopt[onnx] with --ignore-requires-python when MODE=train and Python >= 3.13, so GPU CI on 3.13 uses the full modelopt path - requirements_public_train.txt: add comment documenting the 3.13 install approach for manual setups - test_onnx_quant_workflow.py: - remove py3.13-specific skip from test_ort_quantize_int8_produces_output_file (now skips when onnxruntime is not installed, regardless of version) - replace test_ort_quantize_int8_dispatch_on_py313 with test_ort_quantize_int8_called_on_modelopt_import_error - replace test_fp8_raises_on_py313 with test_fp8_raises_on_modelopt_import_error - remove py3.13 version guard from test_modelopt_importable_when_installed - remove py3.13 version guard from test_ort_importable_when_installed Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(test): pin ONNX IR version 8 in ort quantize test modelopt[onnx] pulls in onnxruntime-gpu~=1.22.0 as a dependency on all Python versions. Newer ONNX packages (1.19+) default model.ir_version to 12, but onnxruntime-gpu 1.22.0 only supports up to IR version 10, causing test_ort_quantize_int8_produces_output_file to fail on the GPU CI for py3.11, py3.12, and py3.13. Pin model.ir_version = 8 (the minimum required for opset 17) before saving the test model so the calibration InferenceSession succeeds with any onnxruntime version that supports IR ≤ 10. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test(onnx): add end-to-end mq.quantize() tests for modelopt Previous coverage only verified that modelopt.onnx.quantization was importable. Add TestModeloptQuantize with two tests that actually call mq.quantize() on a real ONNX model: - test_mq_quantize_int8_produces_valid_onnx: verifies the output file is created and passes onnx.checker (confirms modelopt works at runtime, not just at import time — this is the key Python 3.13 regression check) - test_mq_quantize_int8_output_differs_from_fp32: verifies QDQ nodes were inserted (output graph has more nodes than the FP32 source) Both tests share a _build_tiny_model() helper that creates a minimal Gemm ONNX model with input "dets" and 16 calibration rows, matching the production calibration_data={"dets": calib_dets} call convention. model.ir_version is pinned to 8 for onnxruntime-gpu 1.22.0 compatibility. Tests are skipped when nvidia-modelopt is not installed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(test): use float32 calibration data in TestModeloptQuantize mq.quantize() runs an internal ONNX inference session to profile MatMul nodes; feeding uint8 calibration data to a float-input model caused InvalidArgument. Switch to np.random.randn(...).astype(float32). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(trt): raise RuntimeError when tensorrt missing for ONNX_WORKFLOW=2/3 Previously both TRT import sites caught ImportError inside a broad `except Exception` block and silently fell back to PyTorch with a print. This masked misconfiguration: the user explicitly selected ONNX_WORKFLOW=2 or 3, so a missing tensorrt install is always a hard error. Changes: - USE_ENGINE_ONLY (workflow=3): ImportError now raises RuntimeError with install hint; other TRT errors (bad engine file) still fall back gracefully. - EXPORT_AND_USE_TRT (workflow=2): same split. - test_tensorrt_fallback.py: replace the old "falls back on ImportError" tests with "raises RuntimeError on ImportError" tests; add chained cause check and non-import fallback tests. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(test): fix ORT calibration for ort quantize test ORT's MinMaxCalibrater augments the model to expose intermediate tensors for calibration, but graph *inputs* are not included in the augmented outputs. When the test model had dets->Gemm directly, ORT never collected calibration stats for 'dets', causing: ValueError: Quantization parameters are not specified for param dets. Fix: insert a Relu node (dets -> Relu -> dets_relu -> Gemm) so the Gemm input is an intermediate tensor that gets calibrated. Also switch the calibration array to float32 (consistent with model dtype) and add rewind() to _DetCalibReader in production code for calibration methods that make multiple passes. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(test): skip ort quantize output test when modelopt is installed _ort_quantize_int8 is only invoked when modelopt is absent. When modelopt IS installed its mq.quantize() call leaves ORT's execution- provider state dirty (failed TRT EP init), causing the calibration InferenceSession to run silently without producing stats, which makes quantize_static raise: ValueError: Quantization parameters are not specified for param dets. The test is meaningless in that environment anyway — if modelopt is present the ort path is never taken. Skip when modelopt is importable. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * review: address PR #14 review comments - README: ONNX_WORKFLOW=1 runs PyTorch inference after export (not stop) (bmhowe23 suggestion) - LER: cast calib_dets to float32 before passing to mq.quantize(); _collect_calibration_dets returns uint8 but modelopt expects float (sacpis: bug report on line 1077) - LER: use Path.with_suffix('.engine') instead of str.replace (sacpis nit on line 1104) - LER: add pathlib.Path import - test: remove spurious @skipUnless from _build_tiny_model helper; it is not a test method and the decorator has no effect (sacpis nit on line 299) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * refactor: extract _parse_quant_format() helper from LER Move the QUANT_FORMAT env-var read/validate/warn block into a module-level helper so the test can call the real production logic instead of re-implementing it. - Add _parse_quant_format(rank=0) -> str in logical_error_rate.py - Replace inline parsing block in run_inference_and_decode with a single _parse_quant_format(rank=dist.rank) call - Import _parse_quant_format in test_onnx_quant_workflow.py and simplify _run_quant_block to delegate to it Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: guard against num_obs < 1 in _collect_calibration_dets Python's [:, :-0] is equivalent to [:, :0] and silently returns an empty tensor rather than the full row. Add an explicit check so the caller gets a clear ValueError instead of a confusing width-mismatch error downstream. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * docs: fix tensorrt comment — missing TRT now raises RuntimeError The comment said "Absent at runtime causes graceful fallback to the PyTorch path", but since the TRT ImportError fix (ae0f3b1) both ONNX_WORKFLOW=2 and =3 raise RuntimeError instead of falling back. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Igor Baratta <ialmeidabara@nvidia.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
V3 Torch HE optimizations, eval/train integration, and cleanup - Implement Torch homological equivalence (HE) with spacelike/timelike weight support - Integrate evaluation and training with composable features - Remove all residual JAX references - Miscellaneous cleanup - Revert dual-path inline inference; retain PreDecoderMemoryEvalModule for ONNX/TRT compatibility - Restore legacy-only noise scaling and remove V3 sparsity guard from main branch - Fix PREDECODER_TORCH_COMPILE handling in inference - Adjust CI: remove compilation skips, move slow HE tests to mid-tier Signed-off-by: kvmto <kmato@nvidia.com> Co-authored-by: Ivan Basov <5455484+ivanbasov@users.noreply.github.com> Co-authored-by: Ivan Basov <ibasov@nvidia.com>
* adding dependencies for decoder_ablation workflow Signed-off-by: Sachin Pisal <spisal@nvidia.com> * adding failure_analysis containing the decoder helpers, decoder ablation, and plotting helpers Signed-off-by: Sachin Pisal <spisal@nvidia.com> * adding modified code/evaluation/failure_analysis.py Signed-off-by: Sachin Pisal <spisal@nvidia.com> * adding tests for failure analysis Signed-off-by: Sachin Pisal <spisal@nvidia.com> * adding decoder_ablation as a workflow task Signed-off-by: Sachin Pisal <spisal@nvidia.com> * overriding the config copy with the resolved single basis Signed-off-by: Sachin Pisal <spisal@nvidia.com> * formatting Signed-off-by: Sachin Pisal <spisal@nvidia.com> * formatting Signed-off-by: Sachin Pisal <spisal@nvidia.com> * adding CUDA-Q nv-qldpc-decoder from internal repo and tests Signed-off-by: Sachin Pisal <spisal@nvidia.com> * removing _PYMATCHING_SUPPORTS_CORRELATIONS and inspect Signed-off-by: Sachin Pisal <spisal@nvidia.com> * formatting Signed-off-by: Sachin Pisal <spisal@nvidia.com> * removing unconditional imports Signed-off-by: Sachin Pisal <spisal@nvidia.com> * adding a test to check predecoder actually modifies residual syndromes Signed-off-by: Sachin Pisal <spisal@nvidia.com> * adding comments for _decode_ldpc_batch Signed-off-by: Sachin Pisal <spisal@nvidia.com> * refactoring decoder_ablation_study Signed-off-by: Sachin Pisal <spisal@nvidia.com> * adding unittests for refactored functions Signed-off-by: Sachin Pisal <spisal@nvidia.com> * tracking unavailable decoders Signed-off-by: Sachin Pisal <spisal@nvidia.com> * adding BP variants in try/except block Signed-off-by: Sachin Pisal <spisal@nvidia.com> * removing redundant check Signed-off-by: Sachin Pisal <spisal@nvidia.com> * adding modules to install to requirements Signed-off-by: Sachin Pisal <spisal@nvidia.com> --------- Signed-off-by: Sachin Pisal <spisal@nvidia.com>
* Remove unused model architecture Signed-off-by: Ben Howe <bhowe@nvidia.com> * Update config file, too Signed-off-by: Ben Howe <bhowe@nvidia.com> --------- Signed-off-by: Ben Howe <bhowe@nvidia.com>
* Replace proprietary license headers with Apache-2.0 Update all SPDX headers from LicenseRef-NvidiaProprietary to Apache-2.0 across all 70 tracked source files. Also updates spdx_headers.py to generate Apache-2.0 headers and replace old proprietary headers in-place. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(headers): apply Apache-2.0 headers to files added after branch cut Files added by PRs #13, #14, and #17 still carried the proprietary LicenseRef-NvidiaProprietary header. Replace with Apache-2.0 to match the rest of the codebase after the header migration. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * style: apply YAPF formatting after header replacement Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(headers): restore full file content truncated during rebase The first rebase used --theirs to resolve header conflicts, which took the old PR branch content instead of main's newer content for 5 files. Restore from upstream/main and apply Apache-2.0 header correctly. Affected files: - code/qec/noise_model.py - code/qec/surface_code/homological_equivalence_torch.py - code/tests/mid/test_homological_equivalence.py - code/tests/test_noise_model.py - code/training/train.py Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(test): remove PreDecoderModelMemory_v2 test removed by PR #18 PR #18 removed the unused v2 model architecture. Drop the corresponding test class and import to fix the ImportError in CI. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Added the Apache License 2.0 to the project.
Added a Contributor License Agreement to clarify terms for contributions.
- NOTICE: lists third-party OSS dependencies (Stim, PyMatching, PyTorch, NumPy, Hydra, OmegaConf, SafeTensors, ONNX Runtime, NVIDIA ModelOpt, TensorBoard, torchinfo, Matplotlib) with copyright notices and license texts - README: adds a License section linking LICENSE and NOTICE, and documenting the per-file SPDX headers enforced by the spdx-header-check CI job Covers three OSS distribution compliance requirements: 1. Link to project license file in repo 2. Third-party OSS notice files 3. Link to source files containing required notices/attribution Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds a "Commit Sign-off" section at the top of CONTRIBUTING.md explaining the --signoff requirement, how to use it, and what it appends to the commit message. The DCO full text was already present; this adds the missing how-to context for new contributors. Signed-off-by: Ivan Basov <ibasov@nvidia.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* Add cuStabilizer BitMatrixSampler integration to DEM sampling
Replace the pure-torch dem_sampling with a version that transparently
uses cuQuantum's BitMatrixSampler when available, falling back to the
original torch path when cuST is not installed or USE_CUSTAB=0.
- custab_matrix_sampling() with sampler caching and max_shots tracking
- CuPy zero-copy DLPack GPU pipeline (torch -> cupy -> cuST -> torch)
- Timing instrumentation (get_dem_sampling_avg_ms) for training logs
- Input validation on H/p shapes
- USE_CUSTAB env var toggle with reset helpers for testing
- Vectorized measure_from_stacked_frames (kept from main)
- New tests: test_dem_sampling_custab.py, test_dem_sampling_integration.py
Signed-off-by: kvmto <kmato@nvidia.com>
* feat: add CuPy dependency, tests, and NOTICE entry
requirements_public_inference.txt:
- Document cupy-cudaXXX as an optional GPU-only prerequisite alongside
the existing tensorrt comment; explains the DLPack fallback behaviour.
tests/test_dem_sampling_custab.py:
- Add TestDEMSamplingCupyGPUPath (skipped unless custab + CuPy + CUDA
are all present) covering:
- _CUPY_AVAILABLE flag is set
- correct shape and uint8 dtype from the GPU-native path
- deterministic syndrome matches expected checks
- GPU/CuPy result matches torch CPU fallback on deterministic input
NOTICE:
- Add CuPy (MIT, Preferred Networks) entry
- Add TensorRT (Apache 2.0, NVIDIA) entry — was missing
- Add onnxscript (MIT, Microsoft) entry — was missing
- Add OmegaConf (BSD-3-Clause, Omry Yadan) entry — was missing
- Include full license text or reference for all new entries
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Ivan Basov <ibasov@nvidia.com>
---------
Signed-off-by: kvmto <kmato@nvidia.com>
Signed-off-by: Ivan Basov <ibasov@nvidia.com>
Co-authored-by: Ivan Basov <ibasov@nvidia.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
) These packages are used by the project (listed in requirements_public_inference.txt) but were missing from the NOTICE file: - SciPy (BSD-3-Clause, Enthought / SciPy Developers) - ldpc (MIT, Joschka Roffe) - BeliefMatching (Apache 2.0, Oscar Higgott) Signed-off-by: Ivan Basov <ibasov@nvidia.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
…#30) compute_syndrome_density() does not accept sdr_as_percent — the flag is only used downstream in train.py for display formatting (the SDR unit shown as "%" vs "x"). Passing it caused a TypeError that aborted the orientation-inference long-running CI job after a full training epoch. Fixes: TypeError: compute_syndrome_density() got an unexpected keyword argument 'sdr_as_percent' Signed-off-by: Ivan Basov <ibasov@nvidia.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Production defaults use num_workers=4 with spawn multiprocessing, but all CI jobs and tests forced num_workers=0. Add coverage for both layers: - New test (test_dataloader_multiprocessing.py): verifies the Stim inference datapipe is pickle-safe and produces correct results with num_workers=2 across X, Z, and mixed bases. Runs on CPU in a dedicated ci.yml job. - New ci-gpu.yml step: re-runs inference with PREDECODER_INFERENCE_NUM_WORKERS=2 after the existing smoke run, exercising the full logical_error_rate.py pipeline (multi-worker DataLoader → model forward → PyMatching → LER check). Signed-off-by: kvmto <kmato@nvidia.com>
) * Make cuStabilizer the sole DEM sampling backend and consolidate tests Remove the torch fallback path from dem_sampling.py — cuquantum's BitMatrixSampler is now the only sampling backend, simplifying the module and eliminating the USE_CUSTAB toggle. The sampler cache uses identity-based comparison with a pre-cached transpose to avoid redundant reallocation. Merge test_dem_sampling_custab.py and test_dem_sampling_integration.py into test_dem_sampling.py for a single, comprehensive test suite. Also: - Add cuquantum>=26.3.0 to requirements_public_train.txt - Fix CI to install train (not inference) requirements for GPU tests - Apply yapf formatting (Google style, 100-col limit) Signed-off-by: kvmto <kmato@nvidia.com> * fixed license Signed-off-by: kvmto <kmato@nvidia.com> * fix: use cuquantum-python-cu12 wheel to avoid pkg_resources build failure The cuquantum meta-package fails to build in environments where pkg_resources is unavailable. Pin the CUDA-12 specific wheel directly to bypass the broken auto-detection setup.py. Signed-off-by: kvmto <kmato@nvidia.com> * lazy imports for safe separation between training and inference Signed-off-by: kvmto <kmato@nvidia.com> * quick fix to CI Signed-off-by: kvmto <kmato@nvidia.com> * route cuQuantum dem_sampling tests to GPU CI Signed-off-by: kvmto <kmato@nvidia.com> * left behind change Signed-off-by: kvmto <kmato@nvidia.com> * missing bash session Signed-off-by: kvmto <kmato@nvidia.com> * Make CUDA major version specific requirements files and use custabilizer-cuXY * Revert some changes to test files that are hopefully no longer needed * Revert REQUIRE_CUQUANTUM changes * Change custabilizer version to 0.3.0 * Change custabilizer back to cuquantum-python * Skip test_dem_sampling.py if required deps are not present * Try again * Skip a few more tests if cuquantum-python not installed * Revert CUDA major version specific requirements files Since custabilizer-cuXY is not a viable standalone package, there is no need to try to make CUDA major version specific files. Rather, we just rely on the auto detection logic in cuquantum-python. * Revert "Revert CUDA major version specific requirements files" This reverts commit ce055a2. * small torch device object bug fix for nccl Signed-off-by: kvmto <kmato@nvidia.com> * overcome custab device id limitation Signed-off-by: kvmto <kmato@nvidia.com> * added tiny logging Signed-off-by: kvmto <kmato@nvidia.com> * linted Signed-off-by: kvmto <kmato@nvidia.com> * Revert "Revert "Revert CUDA major version specific requirements files"" This reverts commit 05e92f8. * Revert "Revert "Revert "Revert CUDA major version specific requirements files""" This reverts commit 84c814b. --------- Signed-off-by: kvmto <kmato@nvidia.com> Co-authored-by: Ben Howe <bhowe@nvidia.com>
* fix(ci): disable torch.compile in orientation training to prevent segfault torch.compile=on combined with DataLoader spawn workers during LER validation causes a segfault (20 leaked semaphores, core dumped). Set PREDECODER_TORCH_COMPILE=0 for the Train all orientations step. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Revert "fix(ci): disable torch.compile in orientation training to prevent segfault" This reverts commit 7f0f6c8. * fix(ci): bypass commit-age gate when retrying long-running tests `github.run_attempt > 1` short-circuits the 24 h commit check so that "Re-run all jobs" from the UI always executes the test jobs, even on quiet days with no recent pushes to main. Signed-off-by: Ivan Basov <ibasov@nvidia.com> Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Signed-off-by: Ivan Basov <ibasov@nvidia.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(ci): disable torch.compile in orientation training to prevent segfault torch.compile=on combined with DataLoader spawn workers during LER validation causes a segfault (20 leaked semaphores, core dumped). Set PREDECODER_TORCH_COMPILE=0 for the Train all orientations step. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Revert "fix(ci): disable torch.compile in orientation training to prevent segfault" This reverts commit 7f0f6c8. * feat(ci): add multi-GPU tests and CI job for DDP validation - Add code/tests/test_multi_gpu.py with three test classes (skipped unless torch.cuda.device_count() >= 2): - TestNCCLCommunication: verifies NCCL all_reduce sum across 2 ranks - TestDDPForwardBackward: DDP forward+backward with PreDecoder, checks finite gradients - TestMultiGPUDataGenerator: QCDataGeneratorTorch places output on the correct cuda:{rank} device per rank Uses mp.spawn with file:// rendezvous to avoid port conflicts. - Add multi-gpu-tests job to ci-gpu.yml: - Runs on linux-amd64-gpu-rtxpro6000-latest-2 (2-GPU runner) - Post-merge only (if: main + needs: gpu-tests), 20 min timeout - Verifies >=2 GPUs are visible before proceeding - Runs test_multi_gpu.py then a 2-GPU DDP smoke train+inference via local_run.sh with GPUS=2 (smoke_run.sh hardcodes GPUS=1) - LER check <= 0.35 matches the existing gpu-tests threshold Signed-off-by: Ivan Basov <ibasov@nvidia.com> Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(ci): fix YAPF formatting and show multi-gpu-tests on PRs - Apply yapf formatting (blank lines before top-level functions, assert formatting) to test_multi_gpu.py - Remove `if: github.ref == 'refs/heads/main'` from multi-gpu-tests so the job appears in PR CI checks (was invisible on pull-request branches) Signed-off-by: Ivan Basov <ibasov@nvidia.com> Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(ci): check training log for LER in multi-gpu-tests check_ler_from_log.py looks for [LER Validation] lines which are emitted during training, not inference. Was incorrectly pointing at the inference log. Signed-off-by: Ivan Basov <ibasov@nvidia.com> Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Signed-off-by: Ivan Basov <ibasov@nvidia.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
requirements_public_train.txt was replaced by cu12/cu13 variants; multi-gpu-tests install step still referenced the old filename, causing the job to fail immediately. Switch to cu12 to match the runner's CUDA stack (consistent with mid-gpu-tests and gpu-coverage). Signed-off-by: Ivan Basov <ibasov@nvidia.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
…#38) * fix: steps_per_epoch should count forward passes, not optimizer steps steps_per_epoch was computed by dividing num_samples by (batch_size * accumulate_steps * world_size), which yields the number of optimizer updates. However train_epoch() runs one generate_batch() per loop iteration, so the loop count must be num_samples divided by (batch_size * world_size) only. With the default accumulate_steps=2, every epoch was silently processing only 50% of the intended data. Signed-off-by: huaweil <huaweil@nvidia.com> * ci: trigger CI validation Signed-off-by: Ivan Basov <ibasov@nvidia.com> --------- Signed-off-by: huaweil <huaweil@nvidia.com> Signed-off-by: Ivan Basov <ibasov@nvidia.com> Co-authored-by: Ivan Basov <ibasov@nvidia.com>
When SDR runs before LER, the same model object gets torch.compile'd twice, producing a nested OptimizedModule that segfaults during the first forward pass. Skip compilation when the model is already compiled. Also eagerly tear down SDR's DataLoader workers before LER starts to prevent leaked /dev/shm semaphores. Signed-off-by: kvmto <kmato@nvidia.com>
* fix(ci): disable torch.compile in orientation training to prevent segfault torch.compile=on combined with DataLoader spawn workers during LER validation causes a segfault (20 leaked semaphores, core dumped). Set PREDECODER_TORCH_COMPILE=0 for the Train all orientations step. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Revert "fix(ci): disable torch.compile in orientation training to prevent segfault" This reverts commit 7f0f6c8. * ci: add cu12/cu13 matrix to GPU unit tests Expand the gpu-tests job to a 3×2 matrix (Python 3.11/3.12/3.13 × cu124/cu130) so both CUDA 12.x and CUDA 13.x PyTorch wheels are exercised on every push. TORCH_CUDA and VENV_DIR are namespaced per matrix cell to prevent venv collisions. REQ_FILE selects the new cu-specific requirements files that add the matching cupy wheel (cupy-cuda12x / cupy-cuda13x) for zero-copy DLPack GPU transfers. CPU unit tests are unchanged — the cpu wheel is CUDA-version-agnostic. Signed-off-by: Ivan Basov <ibasov@nvidia.com> Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(ci): correct venv path in multi-worker DataLoader test step The source command was missing _${{ matrix.torch-cuda }} suffix, so the multi-worker step would fail to activate the correct venv created by check_python_compat.sh. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * ci: bump representative CUDA 12.x wheel from cu124 to cu126 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * ci: use cu128 instead of cu126 for CUDA 12.x wheel cu126 is not published by PyTorch; pip fell back to a CPU build causing cudaErrorNoKernelImageForDevice on all cu126 matrix cells. cu128 is a valid published wheel tag. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Signed-off-by: Ivan Basov <ibasov@nvidia.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
…tests (#34) * fix(ci): disable torch.compile in orientation training to prevent segfault torch.compile=on combined with DataLoader spawn workers during LER validation causes a segfault (20 leaked semaphores, core dumped). Set PREDECODER_TORCH_COMPILE=0 for the Train all orientations step. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Revert "fix(ci): disable torch.compile in orientation training to prevent segfault" This reverts commit 7f0f6c8. * feat(decoder_ablation): add TRT pre-decoder backend and cudaq-qec documentation Extends decoder_ablation_study to support the same ONNX_WORKFLOW env-var used by the inference workflow, enabling a full GPU pipeline where the neural pre-decoder runs via TensorRT (FP16/INT8/FP8) while cudaq-qec decoders handle the residual syndromes. - failure_analysis.py: honour ONNX_WORKFLOW=1/2/3 in decoder_ablation_study; add PreDecoderMemoryEvalModule wrapping, TRT engine export/load, and a direct TRT batch execution path that feeds raw stim_dets into the engine and reads L_and_residual_dets without calling _model_forward_and_residual - test_failure_analysis.py: 16 new tests across 4 classes covering env-var parsing, graceful fallback when the engine file is missing, ONNX export path (workflow=1), and full mock-TRT execution path (CPU-safe via patched tensorrt module and _MockCUDADevice) - local_run.sh: document TRT + decoder_ablation command examples - README.md: new "Decoder ablation study with cudaq-qec" section with TRT + cudaq-qec full GPU pipeline examples and decoder variant table Signed-off-by: Ivan Basov <ibasov@nvidia.com> Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(ci): add missing os import in tests; fix yapf formatting in failure_analysis Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(tests): patch torch.zeros and torch.empty for _MockCUDADevice in TRT tests torch.zeros/empty called with device=_MockCUDADevice raised TypeError; extend _patch_tensor_to_for_mock_cuda to redirect mock device to CPU for all tensor creation calls in addition to Tensor.to. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(tests): patch all torch factory fns for _MockCUDADevice in TRT tests arange/full/ones etc. also receive device=_MockCUDADevice from the call chain; replace per-function patches with an ExitStack loop over all common torch factory function names. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(tests): make _MockCUDADevice inherit torch.device("cpu") for full compat Instead of patching every torch factory function, make _MockCUDADevice a real torch.device subclass backed by cpu so all C-level tensor ops work natively. Override the type property to return "cuda" for branch coverage. Only torch.cuda.synchronize needs stubbing. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(tests): revert to plain _MockCUDADevice; patch all torch C-entry-points torch.device cannot be subclassed (TypeError at import). Revert to a plain Python class for _MockCUDADevice and restore comprehensive patching via ExitStack: Tensor.to, nn.Module.to, all factory fns (zeros/arange/full/ as_tensor/…), and torch.cuda.synchronize (no-op stub). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(review): remove os import from tests; fix weight accumulation order - test_failure_analysis.py: remove bare os.environ/os.path usages (original file never imported os; use Path and simplified assertions) - failure_analysis.py: move all_baseline_weights.extend() to after the T < 2 guard so skipped batches (PyTorch path, T < 2) do not inflate baseline weight counts — restores behaviour of the original code Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * refactor: address PR review comments from sacpis - Extract TRT setup to _setup_trt_for_ablation() helper function - Move T < 2 guard outside if/else so both TRT and PyTorch paths skip short rounds consistently - Cache _trt_out_ncols before batch loop to avoid per-batch engine query - Use pinned-memory H2D transfer (torch.as_tensor + pin_memory + non_blocking=True) instead of from_numpy().to() - Single D2H transfer: L_and_residual_out.cpu().numpy() then slice, avoiding two separate round trips - Add note about execute_v2 deprecation in TRT >= 10 - Use setUpClass in TRTFallback and TRTExecution test classes to run the ablation study once per class instead of once per test method Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: shape comments, distributed sync bug, add ONNX_WORKFLOW=2 test - Fix shape comments: stim_dets is (N, (2*T+1)*half) not (2*T*half) — boundary detectors add one extra half-width round - Fix distributed sync: replace barrier() with broadcast_object_list() so non-zero ranks learn about rank-0 ONNX export failures and skip the TRT build instead of hitting FileNotFoundError silently - Add TestDecoderAblationStudyExportAndBuildTRT covering ONNX_WORKFLOW=2 (export + engine build + TRT inference) with mocked onnx.export and tensorrt, using setUpClass for a single shared run Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Signed-off-by: Ivan Basov <ibasov@nvidia.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
… segfault (#41) * fix(ci): disable torch.compile in orientation training to prevent segfault torch.compile=on combined with DataLoader spawn workers during LER validation causes a segfault (20 leaked semaphores, core dumped). Set PREDECODER_TORCH_COMPILE=0 for the Train all orientations step. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Revert "fix(ci): disable torch.compile in orientation training to prevent segfault" This reverts commit 7f0f6c8. * fix(ler): force num_workers=0 when torch.compile is active to prevent segfault Spawning DataLoader workers (multiprocessing_context="spawn") after torch.compile has been applied causes a CUDA context conflict in the spawned subprocesses, resulting in a segfault and ~20 leaked semaphores. In the LER evaluation path the model is compiled (line 924) before the DataLoader is created (line 1057), so the conflict is triggered. The SDR path is unaffected because its DataLoader is created prior to torch.compile. Fix: when _applied_compile is True and num_workers > 0, reset num_workers to 0 so that data loading happens in the main process, avoiding the fork/spawn-after-compile hazard entirely. Fixes: https://github.com/NVIDIA/quantum-predecoder/actions/runs/23924233265/job/69777476304 Signed-off-by: Ivan Basov <ibasov@nvidia.com> Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test(metrics): guard compute_syndrome_density signature against sdr_as_percent sdr_as_percent has been accidentally passed to compute_syndrome_density() twice, each time causing a TypeError only caught by long-running GPU tests. Add a short-tier signature inspection test so the next attempt fails fast on every pre-merge CI run. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(train): remove PREDECODER_TIMING_RUN gate so env overrides always apply PREDECODER_TRAIN_EPOCHS, PREDECODER_TRAIN_SAMPLES, etc. were gated behind PREDECODER_TIMING_RUN=1. smoke_run.sh sets that flag, so full-epoch-training CI worked. run_orientations_long.sh does not set it, so orientation-inference CI silently ignored all overrides and trained with default settings (100 epochs, millions of samples), hitting the 1h30m timeout. Remove the gate so these env vars are always honoured. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * ci(orientation): increase PREDECODER_TRAIN_EPOCHS from 1 to 10 Each epoch with 32768 samples takes ~40s (3s training + ~37s fixed SDR/LER/val overhead). 4 orientations x 10 epochs x 40s ≈ 27 min, well within the 1h budget. 1 epoch was too thin; 100 (the code default) would time out. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(ci): disable SDR gate during orientation training to ensure best_model is saved With only 32k training samples, the model's syndrome density reduction stays at ~1.00x across all 10 epochs — below the hardcoded 1.5x threshold in train.py. This causes every epoch to be rejected by the SDR gate even though validation loss improves, leaving best_model/ empty and causing inference to fail with "No valid PreDecoderModelMemory files found". Setting PREDECODER_DISABLE_SDR=1 sets syndrome_density_reduction=None, so sdr_not_computed=True, bypassing the gate and allowing the best_model checkpoint to be saved. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * ci(orientation): increase PREDECODER_TRAIN_EPOCHS from 10 to 30 10 epochs completed in ~16 min, leaving headroom for 30 epochs within the 1-hour job budget. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Signed-off-by: Ivan Basov <ibasov@nvidia.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
…ility (#43) * fix(ci): disable torch.compile in orientation training to prevent segfault torch.compile=on combined with DataLoader spawn workers during LER validation causes a segfault (20 leaked semaphores, core dumped). Set PREDECODER_TORCH_COMPILE=0 for the Train all orientations step. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Revert "fix(ci): disable torch.compile in orientation training to prevent segfault" This reverts commit 7f0f6c8. * fix(mid): seed BitMatrixSampler explicitly to restore test reproducibility torch.manual_seed() does not control cuQuantum's BitMatrixSampler internal RNG, so the two mid-GPU tests that relied on it for reproducibility were non-deterministic and intermittently failing. Add an optional `seed` parameter to `dem_sampling()` and `MemoryCircuitTorch.generate_batch()`. When a seed is provided a fresh BitMatrixSampler is always created with `Options(seed=N)`, resetting its internal RNG and guaranteeing identical outputs on every call with the same seed. Production paths (seed=None) are unaffected — the cached sampler is reused as before. Update the two failing tests to use the explicit seed kwarg instead of torch.manual_seed(): - test_he_reduces_error_weight: seed=123 - test_full_pipeline_w2_reproducible: seed=100 Fixes: NVIDIA/Ising-Decoding CI run 23963347042 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * style: fix yapf line-break position in need_new condition Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test: add dem_sampling reproducibility tests for seed= parameter Add TestDEMSamplingReproducibility to test_dem_sampling.py with four cases: - same seed on CPU produces bit-exact identical frames - different seeds produce different frames - unseeded calls still reuse the cached sampler (perf regression guard) - same seed on GPU produces bit-exact identical frames (GPU-only) These tests use stochastic p values (0.1–0.9) so they would have caught the original regression: before the seed= fix, BitMatrixSampler's internal RNG was not reset between calls, making "same seed" reproducibility impossible regardless of torch.manual_seed(). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: use torch.Generator for seeded path; BitMatrixSampler RNG is not seedable Options.__init__() does not accept a 'seed' keyword — the cuST BitMatrixSampler's internal RNG is not exposed via the public API. Replace the attempted Options(seed=N) approach with a small pure-torch fallback (_torch_dem_sampling) that uses a local torch.Generator seeded to the requested value. This path is only taken when seed= is explicitly passed (tests); the production BitMatrixSampler cache path is unchanged. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: pass seed directly to BitMatrixSampler constructor BitMatrixSampler accepts seed as a constructor kwarg (not via Options). Replace the torch fallback workaround with the correct cuST API: pass seed= directly to BitMatrixSampler(..., seed=seed). A fresh sampler is created on every seeded call so its internal RNG is reset to the requested seed, guaranteeing identical outputs on repeated calls with the same value. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* DISTANCE and N_ROUNDS updates Signed-off-by: Ben Howe <bhowe@nvidia.com> * Formatting updates Signed-off-by: Ben Howe <bhowe@nvidia.com> * Revert "Formatting updates" This reverts commit 757f378. --------- Signed-off-by: Ben Howe <bhowe@nvidia.com>
add B200, H200 remove A100
reformat title and header, product positioning
* adding decode_batch path in failure_analysis and vectorizing observable projection Signed-off-by: Sachin Pisal <spisal@nvidia.com> * pass syndromes as list-of-lists to cudaq decode_batch Signed-off-by: Sachin Pisal <spisal@nvidia.com> * implementing feedback Signed-off-by: Sachin Pisal <spisal@nvidia.com> --------- Signed-off-by: Sachin Pisal <spisal@nvidia.com>
…ccurate} (#51) * fix(ci): disable torch.compile in orientation training to prevent segfault torch.compile=on combined with DataLoader spawn workers during LER validation causes a segfault (20 leaked semaphores, core dumped). Set PREDECODER_TORCH_COMPILE=0 for the Train all orientations step. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Revert "fix(ci): disable torch.compile in orientation training to prevent segfault" This reverts commit 7f0f6c8. * feat: rename pretrained models to Ising-Decoder-SurfaceCode-1-{Fast,Accurate} - Rename PreDecoderModelMemory_r9_v1.0.77.pt → Ising-Decoder-SurfaceCode-1-Fast.pt - Rename PreDecoderModelMemory_r13_v1.0.86.pt → Ising-Decoder-SurfaceCode-1-Accurate.pt - Models remain Git LFS-tracked via models/*.pt (no storage change) - Add model_checkpoint_file direct-path option to _load_model so named pretrained files (without epoch numbers) can be loaded without directory scanning - Update test_inference_public_model.py, README, and checkpoint_to_safetensors.py Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* Update config_qec_decoder_r9_fp8.yaml change model 1 to model name * Update conf/config_qec_decoder_r9_fp8.yaml --------- Co-authored-by: Ben Howe <141149032+bmhowe23@users.noreply.github.com>
* Update config_qec_decoder_r13_fp8.yaml refer to model 4 as Ising-Decoder-SurfaceCode-1-Accurate * Update conf/config_qec_decoder_r13_fp8.yaml --------- Co-authored-by: Ben Howe <141149032+bmhowe23@users.noreply.github.com>
Adds the predecoder cookbook notebook with inference quick-start, CUDA 13 / ONNX-runtime notes, and correctness-check examples. Signed-off-by: mawolf2023 <mawolf@nvidia.com.com>
Signed-off-by: mawolf2023 <mawolf@nvidia.com.com>
Signed-off-by: mawolf2023 <mawolf@nvidia.com.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adding a cookbook folder with a pre-decoder notebook tutorial. Includes a file with images too.
@ivanbasov