Skip to content

Added pipelines and evals #3

Open
jamesthesnakegatech wants to merge 11 commits intoLilCSharp:mainfrom
jamesthesnakegatech:main
Open

Added pipelines and evals #3
jamesthesnakegatech wants to merge 11 commits intoLilCSharp:mainfrom
jamesthesnakegatech:main

Conversation

@jamesthesnakegatech
Copy link
Copy Markdown

@jamesthesnakegatech jamesthesnakegatech commented Jun 6, 2025

feat: Integrate and evaluate NeuS2 and OpenMVS

This commit introduces initial support and evaluation for NeuS2 and OpenMVS, alongside our existing SuGaR implementation.

Key Findings:

  • Performance: SuGaR remains the fastest (32 mins), outperforming NeuS2 (79 mins) and OpenMVS (62 mins).
  • Quality & Detail:
    • NeuS2 demonstrates superior overall quality.
    • OpenMVS excels in capturing object textures but is less effective for outdoor scenes. It also produces the highest vertex count.
    • SuGaR's output can sometimes be unusable.
  • Memory: NeuS2 is noted as being more memory-efficient.

Further tuning will be required, especially for OpenMVS which was not optimized in this evaluation.

Mip-NeRF 360 Benchmark and Optimization Pipeline

This script provides a comprehensive framework for benchmarking and optimizing Neural Radiance Field (NeRF) methods, specifically SuGaR and NeuS2, on the challenging Mip-NeRF 360 dataset.

It automates the entire pipeline, from downloading the dataset to running experiments with hyperparameter tuning and generating detailed performance reports. The primary goal is to identify the optimal set of hyperparameters for each method on a per-scene or per-scene-type (indoor/outdoor) basis.

Key Features:

  • Automated Dataset Management: Downloads and prepares the Mip-NeRF 360 dataset.
  • Multi-Method Integration: Supports benchmarking for both SuGaR and NeuS2 out-of-the-box.
  • Advanced Hyperparameter Optimization:
    • Grid Search: Systematically evaluates a predefined set of hyperparameters.
    • Bayesian Optimization (Optuna): Intelligently searches the hyperparameter space to find the best configurations more efficiently.
  • Unbounded Scene Support: Hyperparameter spaces and configurations are specifically tailored for the challenges of unbounded 360° scenes.
  • Robust Evaluation:
    • Calculates a 'quality_score' based on mesh properties like watertightness, vertex count, and face regularity.
    • Measures processing time, memory usage (implicitly via success/failure), and other relevant metrics.
  • Comprehensive Reporting:
    • Generates detailed Markdown reports summarizing the benchmark results.
    • Creates visualizations to compare methods, analyze time-quality trade-offs, and rank scene difficulty.
  • Command-Line Interface: Allows for easy configuration of benchmark runs, including selecting scenes, methods, and optimization strategies.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant