-
Notifications
You must be signed in to change notification settings - Fork 22
Dockerfile CPU support in progress #742
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| # Git metadata | ||
| .git | ||
| .gitignore | ||
|
|
||
| # Build artifacts | ||
| build/ | ||
| dist/ | ||
| bin/ | ||
| *.o | ||
| *.so | ||
| *.a | ||
|
|
||
| # Python junk | ||
| __pycache__/ | ||
| *.pyc | ||
| *.pyo | ||
| *.egg-info/ | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| services: | ||
| build: | ||
| image: cytnx:build | ||
| build: | ||
| context: . | ||
| dockerfile: docker/cytnx-cpu.Dockerfile | ||
| args: | ||
| - USE_OMP=OFF # This overrides everything | ||
| target: builder | ||
| env_file: | ||
| - docker/build_config.env # The base defaults | ||
| dev: | ||
| image: cytnx:dev | ||
| build: | ||
| context: . | ||
| dockerfile: docker/cytnx-cpu.Dockerfile | ||
| args: | ||
| - USE_OMP=OFF # This overrides everything | ||
| target: runtime | ||
| env_file: | ||
| - docker/build_config.env # The base defaults | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Two services are identical now. Do we need this config file?
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The reason for two targets here is I'm still debugging the build image where the cmake install directory doesn't work as intended. Ultimately there should only be the dev image. |
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| CMAKE_INSTALL_PREFIX="/opt/conda" | ||
| BUILD_PYTHON="ON" | ||
| USE_ICPC="" | ||
| USE_MKL="" | ||
| USE_OMP="" | ||
| USE_CUDA="" | ||
| USE_HPTT="" | ||
| HPTT_ENABLE_FINE_TUNE="" | ||
| HPTT_ENABLE_AVX="" | ||
| HPTT_ENABLE_ARM="" | ||
| HPTT_ENABLE_IBM="" | ||
| USE_CUTENSOR="" | ||
| USE_CUQUANTUM="" | ||
| RUN_TESTS="" | ||
| RUN_BENCHMARKS="" | ||
| USE_DEBUG="" | ||
| BUILD_DOC="" | ||
| DEV_MODE="" | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I suggest only leave the env for cmake preset here. |
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| FROM ubuntu:latest | ||
| WORKDIR /opt | ||
|
|
||
| ARG PY_VER="3.9" | ||
|
|
||
| # --- 1. System Dependencies & Download (Separate RUNs for caching) | ||
| RUN apt-get update && apt-get install -y curl wget vim bash | ||
| RUN curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh" | ||
|
|
||
| # --- 2. Conda Installation, Environment Setup, and Package Install (Crucially, one combined RUN) | ||
| RUN bash Miniforge3-$(uname)-$(uname -m).sh -p /opt/conda -b && \ | ||
| rm Miniforge3-$(uname)-$(uname -m).sh && \ | ||
| \ | ||
| # Initialize the shell for Conda (using the profile script) | ||
| . /opt/conda/etc/profile.d/conda.sh && \ | ||
| \ | ||
| # Create the environment | ||
| conda create -n cytnx python=${PY_VER} _openmp_mutex=*=*_llvm -y && \ | ||
| \ | ||
| # Activate and install (must be in the same RUN command) | ||
| conda activate cytnx && \ | ||
| conda install -c kaihsinwu cytnx=1 -y && \ | ||
| # Clean up Conda files to keep the image small | ||
| conda clean --all -f -y && \ | ||
| conda install -c conda-forge gxx=14 make cmake -y | ||
|
|
||
| # --- 3. Final Environment Configuration | ||
| # Set the PATH to include the Conda environment binaries | ||
| ENV PATH="/opt/conda/envs/cytnx/bin:/opt/conda/bin:$PATH" | ||
|
|
||
| WORKDIR /work | ||
|
|
||
| CMD ["/bin/bash"] |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,74 @@ | ||
| ### Generator Stage | ||
| FROM python:3.12-slim AS generator | ||
| WORKDIR /build | ||
|
|
||
| # Use conda cytnx package or build from source | ||
| ARG cytnx_conda="OFF" | ||
|
|
||
| # Conda packages version config | ||
| ARG cytnx_ver="1" | ||
| ARG py_ver="3.12" | ||
| ARG compilers_ver="" | ||
| ARG make_ver="" | ||
| ARG cmake_ver="" | ||
|
|
||
| # Pass to environment | ||
| ENV cytnx_conda=${cytnx_conda} | ||
| ENV cy_ver=${cy_ver} | ||
| ENV py_ver=${py_ver} | ||
| ENV compilers_ver=${compilers_ver} | ||
| ENV make_ver=${make_ver} | ||
| ENV cmake_ver=${cmake_ver} | ||
|
|
||
| # Set Cytnx compilation build config for build script | ||
| COPY docker/build_config.env . | ||
| RUN export $(cat build_config.env | xargs) | ||
|
|
||
| COPY docker/generate_conda_deps.py docker/generate_build_cytnx.py . | ||
| RUN python3 generate_conda_deps.py | ||
| RUN python3 generate_build_cytnx.py | ||
|
|
||
|
|
||
| ### Build stage | ||
| FROM continuumio/miniconda3:v25.11.1 AS builder | ||
| WORKDIR /opt | ||
|
|
||
| # Run conda dependency install script | ||
| COPY --from=generator /build/install_conda_deps.sh /build/build_cytnx.sh . | ||
| RUN bash install_conda_deps.sh && conda install -y -c conda-forge conda-pack | ||
|
|
||
| # Copy cytnx source code | ||
| # Dockerfile should be built from the root of the repo!! | ||
| COPY . ./Cytnx | ||
|
|
||
| # CMake configure, build and install Cytnx | ||
| RUN bash build_cytnx.sh | ||
|
|
||
| CMD ["/bin/bash"] | ||
|
|
||
| # 2. Pack the environment into a standalone archive | ||
| # RUN conda-pack -p /opt/conda -o /tmp/env.tar.gz && \ | ||
| # mkdir /runtime && cd /runtime && tar -xzf /tmp/env.tar.gz && \ | ||
| # rm /tmp/env.tar.gz | ||
|
|
||
|
|
||
| ### Runtime Stage | ||
| # FROM debian:bookworm-slim AS runtime | ||
| # WORKDIR /app | ||
| # | ||
| # # Install minimal system libraries (OpenMP, etc. required by C++ apps) | ||
| # RUN apt-get update && apt-get install -y --no-install-recommends \ | ||
| # libgomp1 \ | ||
| # && rm -rf /var/lib/apt/lists/* | ||
| # | ||
| # # Copy the packed environment from the builder | ||
| # COPY --from=builder /runtime /opt/venv | ||
| # | ||
| # # Set environment paths so Python and Cytnx are found | ||
| # ENV PATH="/opt/venv/bin:$PATH" | ||
| # ENV LD_LIBRARY_PATH="/opt/venv/lib:$LD_LIBRARY_PATH" | ||
| # | ||
| # # Test if it works | ||
| # # RUN python3 -c "import cytnx; print('Cytnx version:', cytnx.__version__)" | ||
| # | ||
| # CMD [ "/bin/bash" ] |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| import os | ||
|
|
||
| def generate_build_cytnx(): | ||
| # Handle CMake Build Flags | ||
| # This list maps the Docker ARG names to the desired Shell ENV names | ||
| flags = [ | ||
| "CMAKE_INSTALL_PREFIX", "BUILD_PYTHON", "USE_ICPC", "USE_MKL", | ||
| "USE_OMP", "USE_CUDA", "USE_HPTT", "USE_CUTENSOR", "USE_CUQUANTUM" | ||
| ] | ||
|
|
||
| with open("build_cytnx.sh", "w") as f: | ||
| f.write("#!/bin/bash\n") | ||
| for flag in flags: | ||
| # Hierarchy: Build Arg (env var) > Default Value ("OFF") | ||
| val = os.environ.get(flag, "") | ||
| if val: | ||
| f.write(f'export {flag}="{val}"\n') | ||
|
|
||
| # If install Cytnx from conda, skip CMake build | ||
| install_cytnx_conda = os.environ.get("cytnx_conda", "OFF") | ||
| if install_cytnx_conda == "ON": | ||
| print("cytnx_conda=ON, skipping CMake build of Cytnx.") | ||
| return | ||
|
|
||
| # CMake configure and build Cytnx | ||
| # Assuming source code mounted at ./Cytnx | ||
| with open("build_cytnx.sh", "a") as f: | ||
| f.write("\n") | ||
| f.write("cd Cytnx\n") | ||
| f.write("mkdir build\n") | ||
| f.write("cd build\n") | ||
| f.write("cmake .. \\\n") | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Cytnx uses scikit-build as the build tool. We should use it to build and install. And what is the reason that these python scripts for meta-programming are needed? It is possible to just write the pure sh scripts?
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I tried that and found leaving all the logic specifying the conda dependencies and version controlling in a few shell scripts would make a mess, which is why
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Nonetheless I'll work on scikit-build on the next commit, thanks. |
||
| for flag in flags: | ||
| val = os.environ.get(flag, "") | ||
| if val: | ||
| f.write(f' -D{flag}="${{{flag}}}" \\\n') | ||
| # f.write(" -DCMAKE_BUILD_TYPE=Release\n") | ||
| # f.write("make -j$(nproc)\n") | ||
| f.write("make\n") | ||
| f.write("make install\n") | ||
|
|
||
| with open("build_cytnx.sh", "r") as f: | ||
| print(f.read()) | ||
|
|
||
| if __name__ == "__main__": | ||
| generate_build_cytnx() | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,41 @@ | ||
| import os | ||
|
|
||
| def generate_conda_deps(): | ||
| # Handle Conda Dependencies | ||
| # Pin to latest versions as of 2026-02-05 | ||
| conda_deps = { | ||
| "python": os.environ.get("py_ver", "3.12"), | ||
| "compilers": os.environ.get("compilers_ver", ""), | ||
| "make": os.environ.get("make_ver", ""), | ||
| # "cmake": os.environ.get("cmake_ver", "3.26"), | ||
| "cmake": os.environ.get("cmake_ver", ""), | ||
| "numpy": os.environ.get("numpy_ver", ""), # Empty string means latest | ||
| "boost": os.environ.get("boost_ver", ""), | ||
| "libboost": os.environ.get("libboost_ver", ""), | ||
| "pybind11": os.environ.get("pybind11_ver", ""), | ||
| "openblas": os.environ.get("openblas_ver", ""), | ||
| "arpack": os.environ.get("arpack_ver", ""), | ||
| } | ||
|
|
||
| # Standard deps without specific version args in your Dockerfile | ||
| extra_deps = ["git", "beartype", "gtest", "benchmark"] | ||
|
|
||
| conda_cmd = ["conda", "install", "-y", "-c", "conda-forge"] | ||
| for pkg, ver in conda_deps.items(): | ||
| conda_cmd.append(f"{pkg}={ver}" if ver else pkg) | ||
| conda_cmd.extend(extra_deps) | ||
|
|
||
| # Directly install conda cytnx package if cytnx_conda="ON" | ||
| install_cytnx_conda = os.environ.get("cytnx_conda", "OFF") | ||
| if install_cytnx_conda == "ON": | ||
| cytnx_ver = os.environ.get("cytnx_ver", "1.0.0") | ||
| conda_cmd = ["conda", "install", "-y", "-c", "kaihsinwu", f"cytnx={cytnx_ver}"] | ||
| elif install_cytnx_conda != "OFF": | ||
| raise ValueError(f"Invalid value for cytnx_conda: {install_cytnx_conda}") | ||
|
|
||
| with open("install_conda_deps.sh", "w") as f: | ||
| f.write("#!/bin/bash\n") | ||
| f.write(" ".join(conda_cmd) + "\n") | ||
|
|
||
| if __name__ == "__main__": | ||
| generate_conda_deps() |
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it fine to ignore the folders that will not be used in docker? For example, docs/, example/, doxygen_config/, developer_tools/, docs/, tests/, pytests/, and so on.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes perfectly fine.