Support for distorted camera models / COLMAP fisheye cameras

[DuVogel87]

Hi,

I would like to experiment with ZipSplat on datasets that use distorted camera models, in particular COLMAP-style fisheye / thin-prism camera models.

From what I have found so far, ZipSplat currently seems to assume a pinhole-style camera representation. The public Camera class appears to store only [w, h, fx, fy, cx, cy], and I did not find support for distortion parameters such as radial/tangential/thin-prism coefficients. In the paper, camera priors also seem to be represented mainly through pose and field-of-view information, rather than a full distorted projection model.

My use case is a dataset reconstructed with COLMAP using a distorted camera model, for example THIN_PRISM_FISHEYE. I would prefer not to undistort or crop the images before running ZipSplat, because I am trying to preserve as much image content as possible.

I would like to try adding support for distorted camera models, but I am not fully sure which parts of the codebase would need to be changed.

Could you point me to the relevant places where camera projection / ray generation / camera priors are handled?

In particular, I would appreciate guidance on questions like:

• Is the current model architecture fundamentally tied to pinhole cameras, or could distorted camera models be added mostly by extending the camera representation?
• Where would distortion-aware projection or ray generation need to be implemented?
• If using use_priors=True, where should additional camera parameters such as COLMAP distortion coefficients be encoded?
• Would adding distorted cameras require retraining the model, or could this potentially work for inference/fine-tuning with modified camera handling?
• Are there any assumptions in the preprocessing, cropping, resizing, or Gaussian parameter prediction that would break for strongly distorted fisheye images?

I am happy to experiment and possibly contribute a patch, but I would really appreciate some pointers on the intended architecture and the minimum set of files/functions that would need to be adapted.

Thanks!

[veichta]

Hi, thanks for the questions!

ZipSplat is currently pinhole-only, and there are four places where that assumption lives:

• Camera representation (zipsplat/camera.py): stores exactly [w, h, fx, fy, cx, cy], no distortion parameters.
• Camera priors (zipsplat/backbone.py, CameraEnc): when use_priors=True, the model is conditioned on a 9-dim encoding [t_w2c (3), quat (4), vfov, hfov], so distortion is invisible to the model. Adding COLMAP coefficients here means widening the encoder input, which only makes sense with retraining or fine-tuning.
• Rendering (zipsplat/gaussians.py, _render_one): calls gsplat.rasterization with a 3x3 K and the default pinhole camera model.
• Preprocessing (zipsplat/predictor.py / zipsplat/utils.py): inputs are center-cropped to square and resized to 252px. For a fisheye this would discard the content you want to keep.

Those are quite easy fixes, but for this to work, I think the main bottleneck will be the backbone, which only saw pinhole images during pretraining. I am not aware of any model that can reliably reconstruct distorted images, and I am not sure how well fine-tuning the model for it would work.

Some useful pointers if you want to attempt it:

• gsplat >= 1.5 supports OpenCV radial/tangential/thin-prism camera models via its 3DGUT mode, so the rendering part should work.
• GeoCalib has some differentiable camera model implementations that you can borrow.

A pragmatic workaround for your case (calibration + poses known): undistort the images to pinhole for the feed-forward pass to get an initial Gaussian set, then refine the Gaussians per-scene (test-time optimization) by rendering with gsplat against the original distorted frames. This should work well and converge really fast (1-3 sec, see the TTO results in the paper).

I hope this helps!