Cellposeindocker#171
Merged
Merged
Conversation
joshua-gould
reviewed
Jul 7, 2026
joshua-gould
left a comment
joshua-gould
left a comment
Collaborator
There was a problem hiding this comment.
- I get the following errors:
sed: can't read /usr/local/lib/python3.11/dist-packages/scallops/_version.py: No such file or directory
--> 73907ff085d7
STEP 18/19: RUN apt-get update -qq && DEBIAN_FRONTEND=noninteractive apt-get install -qq --no-install-recommends -y libfontconfig1 && rm -rf /var/lib/apt/lists/*
W: Failed to fetch https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/InRelease Certificate verification failed: The certificate is NOT trusted. The certificate issuer is unknown. Could not handshake: Error in the certificate verification. [IP: 69.192.139.214 443]
-
Can you remove napari and test requirements as this docker image is primarily for batch analysis?
-
Can you use python 3.12 or 3.13 instead of 3.11?
Collaborator
Author
Weird, will look into it but I cannot replicate this, when do you get it?
I do not think this is only for batch analysis... the issue raised is but the docker should contain all the things for scallops to work, IMO. Perhaps napari, but not tests?
The latest tensorflow image only have 3.11 |
Collaborator
|
I think the docker should be as minimal as possible-hence remove napari and tests. I also noticed that installing napari changed the version of tested external dependencies in requirements.txt. |
Collaborator
Author
|
OK, but I think people assume if you provide a docker all capabilities should come, batteries included, but i'll remove them |
- Switch builder base to tensorflow/tensorflow:2.21.0 (Python 3.11) so TF is pre-installed, avoiding the multi-GB re-download and disk pressure - Single-stage build to eliminate the large COPY --from layer commit that exhausted Podman's /var/tmp on constrained local disks - Install CPU-only PyTorch before ufish/cellpose to prevent the NVIDIA CUDA stack from being pulled in as a transitive dependency - Layer ordering: most-stable deps first (ufish → cellpose → extras → core) so requirements.txt changes don't invalidate the heavy cached layers - Strip tensorflow from the uv install pass (already in base image) to avoid re-downloading the 545 MB wheel - ARG TORCH_COMPUTE=cpu: override with --build-arg TORCH_COMPUTE=cu124 etc. for GPU-enabled deployments - ARG TF_VERSION / PYTHON_VERSION: single place to update the TF base image - Patch dist-info METADATA + _version.py after install so setuptools_scm does not overwrite the version with 0.0.0 when .git is absent - .dockerignore: add !scallops/_version.py so the pre-generated version file is always included in the build context; expand exclusions for IDE dirs, build artifacts, and coverage reports
…ild-arg setuptools_scm falls back to 0.0.0 when .git is absent from the build context. Fix it at every level: Dockerfile: accept SCM_VERSION as a build arg (default 0.0.0+unknown), passed by CI or docker.mk. Patch both _version.py and dist-info METADATA after install. Remove the fragile COPY scallops/_version.py that broke on fresh clones. docker.mk: local mirror of the CI workflow. Runs python -m setuptools_scm as a preflight, attaches OCI labels matching docker/metadata-action output, tags with semver + short SHA, uses docker buildx to match CI. Includes docker-push target for manual GHCR pushes. docker.yml: add fetch-depth=0 so setuptools_scm can walk git history to find tags; generate SCM_VERSION and pass it as a build-arg to build-push-action. .dockerignore: exclude _version.py from build context (version now injected via build-arg, not file copy).
…o-project to avoid TF resolution)
…ersions are invalid in Docker tag format)
…file parse error on bare lines)
Dockerfile: replace fragile sed glob with a Python one-liner that uses importlib.metadata.distribution() to find the exact dist-info path and patches both METADATA (so importlib.metadata.version() returns the right value) and _version.py (so scallops.__version__ is consistent). __main__.py: add --version argparse action so `scallops --version` works; previously the version was only visible buried in --help description text.
…ection Dockerfile: switch torch/torchvision install from uv to pip. uv uses rustls which fails with HandshakeFailure on networks with SSL inspection proxies; pip uses OpenSSL which negotiates TLS more permissively. docker.mk: use `uv run --with setuptools_scm` to resolve the package version. This creates a throwaway venv, avoiding PEP-668 externally-managed-environment errors on Homebrew/system Python without needing pip --user or pipx. Docker tag sanitizes the PEP-440 local separator (+) to (-) for compatibility.
The PyTorch CDN (download-r2.pytorch.org) is blocked by SSL inspection on some networks. The separate pip/uv torch install step was introduced to get CPU-only wheels but caused HandshakeFailure errors on Linux. Removing the step entirely: torch is now installed as a transitive dep of ufish/cellpose from PyPI, which is CUDA-capable by default and works with both CPU and GPU at runtime (--gpus all). Image is larger but builds everywhere without network-specific SSL workarounds. Also removes the unused TORCH_COMPUTE build arg.
importlib.metadata d._path is a private attribute that fails silently on Ubuntu's dist-packages layout. Use sed with | delimiters (safe for the + and . in PEP 440 versions) to patch both METADATA and _version.py directly.
- Switch builder base to tensorflow/tensorflow:2.21.0 (Python 3.11) so TF is pre-installed, avoiding the multi-GB re-download and disk pressure - Single-stage build to eliminate the large COPY --from layer commit that exhausted Podman's /var/tmp on constrained local disks - Install CPU-only PyTorch before ufish/cellpose to prevent the NVIDIA CUDA stack from being pulled in as a transitive dependency - Layer ordering: most-stable deps first (ufish → cellpose → extras → core) so requirements.txt changes don't invalidate the heavy cached layers - Strip tensorflow from the uv install pass (already in base image) to avoid re-downloading the 545 MB wheel - ARG TORCH_COMPUTE=cpu: override with --build-arg TORCH_COMPUTE=cu124 etc. for GPU-enabled deployments - ARG TF_VERSION / PYTHON_VERSION: single place to update the TF base image - Patch dist-info METADATA + _version.py after install so setuptools_scm does not overwrite the version with 0.0.0 when .git is absent - .dockerignore: add !scallops/_version.py so the pre-generated version file is always included in the build context; expand exclusions for IDE dirs, build artifacts, and coverage reports
- Remove napari, napari_ome_zarr, pytest, pytest-xdist and libfontconfig1: napari is a GUI tool useless headless; test deps have no place in production - Fix version: set SETUPTOOLS_SCM_PRETEND_VERSION via Docker ENV so it propagates into the PEP 517 subprocess that uv spawns; tested via minimal container that confirmed dist-info version matches SCM_VERSION build-arg - Remove NVIDIA apt repos before apt-get to prevent SSL failures on restricted networks (TF base image ships these repos; they fail with cert errors) - Infer Python version dynamically from interpreter instead of hardcoded ARG; print it in build logs; remove stale PYTHON_VERSION and TORCH_COMPUTE ARGs - Version flows entirely from SETUPTOOLS_SCM_PRETEND_VERSION through setuptools_scm's own write_to; no manual _version.py writes
…ld-arg setuptools_scm falls back to 0.0.0 when .git is absent from the build context. Fix it at every level: Dockerfile: accept SCM_VERSION as a build arg (default 0.0.0+unknown), passed by CI or docker.mk. Patch both _version.py and dist-info METADATA after install. Remove the fragile COPY scallops/_version.py that broke on fresh clones. docker.mk: local mirror of the CI workflow. Runs python -m setuptools_scm as a preflight, attaches OCI labels matching docker/metadata-action output, tags with semver + short SHA, uses docker buildx to match CI. Includes docker-push target for manual GHCR pushes. docker.yml: add fetch-depth=0 so setuptools_scm can walk git history to find tags; generate SCM_VERSION and pass it as a build-arg to build-push-action. .dockerignore: exclude _version.py from build context (version now injected via build-arg, not file copy).
…o-project to avoid TF resolution)
…ersions are invalid in Docker tag format)
…file parse error on bare lines)
Dockerfile: replace fragile sed glob with a Python one-liner that uses importlib.metadata.distribution() to find the exact dist-info path and patches both METADATA (so importlib.metadata.version() returns the right value) and _version.py (so scallops.__version__ is consistent). __main__.py: add --version argparse action so `scallops --version` works; previously the version was only visible buried in --help description text.
…ection Dockerfile: switch torch/torchvision install from uv to pip. uv uses rustls which fails with HandshakeFailure on networks with SSL inspection proxies; pip uses OpenSSL which negotiates TLS more permissively. docker.mk: use `uv run --with setuptools_scm` to resolve the package version. This creates a throwaway venv, avoiding PEP-668 externally-managed-environment errors on Homebrew/system Python without needing pip --user or pipx. Docker tag sanitizes the PEP-440 local separator (+) to (-) for compatibility.
The PyTorch CDN (download-r2.pytorch.org) is blocked by SSL inspection on some networks. The separate pip/uv torch install step was introduced to get CPU-only wheels but caused HandshakeFailure errors on Linux. Removing the step entirely: torch is now installed as a transitive dep of ufish/cellpose from PyPI, which is CUDA-capable by default and works with both CPU and GPU at runtime (--gpus all). Image is larger but builds everywhere without network-specific SSL workarounds. Also removes the unused TORCH_COMPUTE build arg.
importlib.metadata d._path is a private attribute that fails silently on Ubuntu's dist-packages layout. Use sed with | delimiters (safe for the + and . in PEP 440 versions) to patch both METADATA and _version.py directly.
for more information, see https://pre-commit.ci
The TF base image ships NVIDIA and deadsnakes apt sources that fail SSL verification on networks with certificate inspection. These repos are unused: CUDA comes from self-contained PyPI wheels (nvidia-*-cu13), not apt packages, and the Python version is already pinned in the base image. Removing them before any apt-get update eliminates the SSL warnings for all users.
- Replace grep filter (only caught first line of multi-line error) with
2>/dev/null so the py312_intrinsics SyntaxError block is fully silenced
- Print a clear WARNING in build logs when SCM_VERSION is the default
0.0.0+unknown so users know to use make -f docker.mk docker
compileall prints '*** Error compiling...' to stdout and the SyntaxError details to stderr — 2>/dev/null only suppressed the latter. Use >/dev/null 2>&1 to silence both streams completely.
Collaborator
|
Looks good. Do we want to add https://docs.rapids.ai/install/#selector when building for GPU? |
joshua-gould
approved these changes
Jul 9, 2026
Collaborator
Author
|
uh! that is a great point will add and test, although, do we use RAPIDs anywhere in the code right now? |
Collaborator
|
We currently only use RAPIDs for PCA |
Dockerfile:
- ARG IS_GPU (default 0) lifted into ENV IS_GPU so it persists across all
RUN steps and is inspectable in running containers
- IS_GPU controls NVIDIA apt repo retention (kept for GPU, removed for CPU)
and RAPIDS installation (GPU only)
- RAPIDS (cuDF, cuML, dask-cudf) installed from pypi.nvidia.com when IS_GPU=1;
build fails with a clear error if RAPIDS_VERSION is not set for GPU builds
- ARG/ENV ordering cleaned up: IS_GPU declared after FROM (not needed before)
docker.mk:
- IS_GPU derived automatically from TF_VERSION via findstring — users never
need to pass it explicitly
- docker-gpu target validates RAPIDS_VERSION before starting the build
- _check_prereqs: after failed uv install attempt, verify success and print
a clear error with the manual install URL if everything fails
- RAPIDS_VERSION, RAPIDS_CUDA, IS_GPU forwarded as build-args in _build
…mands
- Replace prose GPU description with a table showing which library uses GPU,
which commands are affected, and whether the default or GPU image is needed
- Document docker-gpu target and required RAPIDS_VERSION arg
- Explain IS_GPU env var and how to inspect it at runtime
- Add RAPIDS and current RAPIDS release links
Collaborator
Author
|
Added an option to build with GPU that has rapids.... This might fit neatly if we eventually add GPU to the features (https://github.com/rapidsai/cucim) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Attending to Ana Caroline Leote message in scallops slack (https://roche.enterprise.slack.com/archives/C0427EMS4SX/p1783034473804959), I've refactor the docker to include all dependencies. A lot of changes were necesary beacause of the sizes of TF and torch.