Skip to content

SINGLE_FRAME_INTEGRATION.md

Latest commit

 

History

History
232 lines (175 loc) · 10.9 KB

SINGLE_FRAME_INTEGRATION.md

File metadata and controls

232 lines (175 loc) · 10.9 KB
Raw

SHARP Single-Frame Depth-Injection — Integration Brief

Hand this whole file to a fresh Claude Code session to integrate the single-frame splat generator into another application. It is self-contained: API, install, a runnable example, and every hard-won gotcha from building the original project. Ignore everything about 4DGS / video sequences — this brief is the single image → single splat path only.

What this module does

Given one RGB image and one metric depth map (camera-space Z, in meters), it produces a 3D Gaussian Splat (.ply) with correct metric scale and plausible parallax under modest camera moves. It is a thin, runtime extension on top of Apple's ml-sharp — no upstream files are edited; a composer subclass is hot-swapped in at runtime.

Pipeline it was built for (you only need step 3):

  1. Houdini → beauty render + Karma cam_zdepth AOV
  2. An image model (e.g. Nano Banana) → photoreal RGB
  3. SHARP (this module): RGB + Houdini depth → metric Gaussian splat ← this is what you integrate

Source repo: https://github.com/gitcapoom/Sharp_Depth_Injection Built against apple/ml-sharp pinned at commit cdb4ddc6.

The entire public API

from sharp_ext import predict_image_with_depth, load_depth_exr

# 1) build the SHARP predictor ONCE (model download is automatic, ~/.cache)
predictor = build_predictor(device="cuda")          # see snippet below

# 2) load a metric depth EXR (handles Karma sky=0 / clipFar automatically)
depth = load_depth_exr("depth.exr")                  # -> (H, W) float32, meters

# 3) predict the splat
gaussians = predict_image_with_depth(
    predictor,
    image,            # np.ndarray (H, W, 3) uint8, RGB
    f_px,             # float, focal length in pixels (see formula below)
    depth,            # np.ndarray (H, W) float32, METRIC METERS, +Z forward
    device="cuda",
    blend_alpha=0.4,  # 0.0 = full external override, 1.0 = vanilla SHARP, 0.4 = good default
)

# 4) write a standard 3DGS .ply (colors auto-converted linearRGB -> sRGB)
from sharp.utils.gaussians import save_ply
h, w = image.shape[:2]
save_ply(gaussians, f_px, (h, w), "out.ply")

That's the whole surface. gaussians is an ml-sharp Gaussians3D NamedTuple (mean_vectors, singular_values, quaternions, colors, opacities), batch dim B=1.

build_predictor (copy this verbatim)

import torch
from sharp.models import PredictorParams, create_predictor

DEFAULT_MODEL_URL = "https://ml-site.cdn-apple.com/models/sharp/sharp_2572gikvuh.pt"

def build_predictor(device="cuda"):
    state_dict = torch.hub.load_state_dict_from_url(DEFAULT_MODEL_URL, progress=True)
    predictor = create_predictor(PredictorParams())
    predictor.load_state_dict(state_dict)
    predictor.eval()
    predictor.to(device)
    return predictor

f_px — focal length in pixels

f_px = (focal_length_mm / horizontal_aperture_mm) * image_width_px
  • If your image model upscaled the render (common), use the processed image's width: f_px = f_px_render * (processed_width / render_width). Equivalent to just using the formula with the processed width.
  • If you have no lens data, SHARP also works from EXIF via sharp.utils.io.load_rgb(path) which returns (image, icc, f_px).

Inputs / outputs contract (precise)

arg type notes
image np.ndarray (H,W,3) uint8 RGB, not BGR. Strip alpha. Any resolution; it's resized to 1536 internally.
f_px float pixels, in the image's coordinate frame (use the image's width).
external_depth np.ndarray (H,W) float32 camera-space Z, meters, positive forward. NOT ray-distance. Can differ in resolution from image but must be the same view; it's resized (max-pool) to the 768×768 Gaussian grid.
blend_alpha float 0.0=trust depth fully, 1.0=ignore depth (vanilla SHARP), 0.4 is a good metric-anchor default.
returns Gaussians3D B=1; feed to save_ply or ml-sharp's gsplat renderer.

Coordinate convention is OpenCV (x right, y down, z forward); scene center ≈ (0, 0, +z).

Install

Target Python 3.13. Done on Windows + RTX 4090; a single 1536² inference is ~4 s, ~6 GB VRAM.

# 1) ml-sharp at the pinned commit (the extension targets this exact version)
git clone https://github.com/apple/ml-sharp.git ml-sharp
git -C ml-sharp checkout cdb4ddc6

# 2) Python env
python -m venv .venv && .venv/Scripts/activate     # (Windows) or source .venv/bin/activate
pip install -r ml-sharp/requirements.txt
pip install -e ml-sharp                              # so `import sharp` resolves

# 3) the extension package
git clone https://github.com/gitcapoom/Sharp_Depth_Injection.git
pip install -e Sharp_Depth_Injection                 # so `import sharp_ext` resolves

# 4) depth EXR reader (imageio has NO EXR backend on Windows)
pip install OpenEXR>=3.4

CRITICAL — CUDA torch. pip install -r requirements.txt pulls CPU-only torch by default, which silently runs on CPU. Force the CUDA build (match your CUDA; cu128 used here):

pip install --force-reinstall torch==2.8.0 torchvision==0.23.0 --index-url https://download.pytorch.org/whl/cu128

Verify: python -c "import torch; print(torch.cuda.is_available())" must print True.

gsplat is only needed if you want to render the splat to images/video in-process; producing the .ply does not require it. (gsplat JIT-compiles CUDA kernels and needs MSVC cl.exe + a CUDA toolkit on PATH — skip it unless you render.)

Minimal-copy alternative (embed without the full package)

If you don't want the whole package, copy these 5 files from sharp_ext/ — they are the entire single-frame core:

  • external_depth.py — ExternalDepthGaussianComposer (the subclass)
  • swap_composer.py — install_external_composer (runtime hot-swap)
  • _predict_at_res.py — resolution-parameterized copy of predict_image
  • predict_with_depth.py — predict_image_with_depth (the entry point)
  • depth_io.py — load_depth_exr

Do not copy batch_4dgs.py, frame_source.py, recolor.py, prune.py, compress_plys.py (those are all 4DGS/sequence machinery). If you copy files individually, import from the submodules directly (from external_depth import ...) or write a tiny __init__.py exporting just the two public functions — the repo's __init__.py imports the 4DGS modules too.

Locked learnings / gotchas (do not relitigate — these cost real debugging time)

  1. Decoder stride is 2, not 4. The Gaussian grid is 768×768 (= 1536/2), with the checkpoint sharp_2572gikvuh.pt. The original spec guessed 4; it's 2. Already baked into the code.

  2. Two depth layers — propagate the correction to layer 1+. SHARP predicts 2 monodepth layers per pixel (layer 0 = visible surface, layer 1 = disocclusion hallucination). Injecting depth into layer 0 only, leaving layer 1 at the network's predicted NDC value, makes layer 1 land at an inconsistent metric depth → ghost duplicates of every object. Fix (built in, propagate_to_other_layers=True): scale all layers by the same per-pixel factor as layer 0's correction, preserving relative layer ordering while keeping absolute meters consistent.

  3. Resize depth with max-pool on inverse-depth, not bilinear. When downsampling the depth to the 768 grid, bilinear blends thin foreground (a pole at 30 m) with distant background (sky at 9988 m) into bogus midway depths → vertical-streak halos around thin features. Use max-pool of inverse-depth (= min-pool of depth = "closest surface in the cell"). Built into set_external_depth.

  4. Sky / no-hit depth pixels. Karma writes 0 at sky/no-hit (which maps to inverse-depth ∞ → "at the camera" → halo). load_depth_exr remaps 0 and non-finite pixels to a far value; by default it auto-picks the file's own max finite depth so the 0-fill merges with any existing clipFar fill. Channel auto-detect tries depth.Z, Z, Y, depth, R, hitPz, else the single channel; pass channel="..." if needed.

  5. Gaussian scale correction is already handled. base_scale_on_predicted_mean=True (the checkpoint default) makes the composer scale singular_values by (zz_external / zz_network) automatically. Do not add the spec's "Mode B" explicit singular-value rescale on top — it double-applies and overshoots.

  6. Colors are linearRGB internally; save_ply converts to sRGB for public renderers. If you consume gaussians.colors directly, they're linearRGB.

  7. Internal resolution is locked to 1536. The SPN encoder (sharp/models/encoders/spn_encoder.py) only tiles cleanly at 1536². 3072 (and other sizes) fail at split time — don't bother trying to raise it for more detail.

  8. Subclass, don't fork. The whole design rule: never edit anything under ml-sharp/. ExternalDepthGaussianComposer subclasses GaussianComposer and is swapped into the built predictor at runtime by install_external_composer. This keeps clean against upstream.

  9. Depth must be camera-space Z, positive forward, in meters. Not world-space Z (Pz), not ray-distance (those differ off the optical axis). If your scene isn't in meters, convert before injection (check stage metersPerUnit).

  10. The injection is a soft global prior, not pixel-exact ground truth. If the RGB came from an image model, it has drifted a few pixels vs the depth — blend_alpha ≈ 0.4 anchors metric scale and layout without fighting the drift. Don't try to align pixel-for-pixel or add a trust mask; a uniform global blend is the design.

Self-test the integration works

import numpy as np
from sharp.utils.gaussians import save_ply
from sharp_ext import predict_image_with_depth   # load_depth_exr if you have an EXR

predictor = build_predictor("cuda")

# synthetic 1-meter wall: flat RGB + constant 5 m depth
H = W = 512
image = (np.ones((H, W, 3)) * 180).astype(np.uint8)
depth = np.full((H, W), 5.0, np.float32)
f_px = (24.0 / 35.0) * W

g = predict_image_with_depth(predictor, image, f_px, depth, device="cuda", blend_alpha=0.4)
print("gaussians:", tuple(g.mean_vectors.shape))   # expect (1, 1179648, 3)
save_ply(g, f_px, (H, W), "selftest.ply")

Open selftest.ply in any 3DGS viewer (e.g. https://playcanvas.com/supersplat/editor). A correct result is a roughly fronto-parallel sheet ~5 m from the camera. If it's at the wrong distance, your f_px or depth units are off (revisit gotcha #9 and the f_px formula).

What to tell the new session about your app

The integration is just: build the predictor once (it's stateful — keep it alive), then call predict_image_with_depth(...) per (image, depth) pair and do whatever your app needs with the Gaussians3D or the saved .ply. The predictor holds the swapped composer; you can reuse it across many calls. There is no per-call setup beyond the function call itself.