ScanNet++ Dataset

[alvinjinsung]

Hello! Thank you for sharing great work!

I wanted to ask you if you can provide details regarding ScanNet++ dataset, specifically,

• dataset preparation procedure (and code) (ex. do you use fisheye camera as in original ScanNet++ or convert it to Pinhole)
• specific configuration
• evaluation code

These information would greatly help me to test the code on the benchmark you established!
Thank you and I will be waiting for your response!

[yyfz]

Hi, we convert fisheye camera images to pinhole images using the tool provided in https://github.com/scannetpp/scannetpp by running the following command: python -m dslr.undistort dslr/configs/undistort.yml. The evaluation is performed using the following code:

def nn_correspondance(verts1, verts2):
    indices = []
    distances = []
    if len(verts1) == 0 or len(verts2) == 0:
        return indices, distances

    kdtree = KDTree(verts1)
    distances, indices = kdtree.query(verts2)
    distances = distances.reshape(-1)

    return distances

def write_color_distances(path, pcd, distances, max_distance):
    o3d.utility.set_verbosity_level(o3d.utility.VerbosityLevel.Debug)
    # cmap = plt.get_cmap("afmhot")
    cmap = plt.get_cmap("hot_r")
    distances = np.array(distances)
    colors = cmap(np.minimum(distances, max_distance) / max_distance)[:, :3]
    pcd.colors = o3d.utility.Vector3dVector(colors)
    o3d.io.write_point_cloud(path, pcd)


def evaluate(mesh_pred, mesh_trgt, threshold=.05, down_sample=.02, scale=1, offset=[0., 0, 0], vis_err=False):
    pcd_trgt = o3d.geometry.PointCloud()
    pcd_pred = o3d.geometry.PointCloud()
    
    vertices = mesh_trgt.vertices[:, :3]
    vertices -= offset
    vertices *= scale
    mesh_trgt.vertices[:, :3] = vertices

    pcd_trgt.points = o3d.utility.Vector3dVector(mesh_trgt.vertices[:, :3])
    pcd_pred.points = o3d.utility.Vector3dVector(mesh_pred.vertices[:, :3])

    if down_sample:
        pcd_pred = pcd_pred.voxel_down_sample(down_sample)
        pcd_trgt = pcd_trgt.voxel_down_sample(down_sample)
    
    colors = np.random.rand(np.array(pcd_trgt.points).shape[0], 3)
    pcd_trgt.colors = o3d.utility.Vector3dVector(colors)
    colors = np.random.rand(np.array(pcd_pred.points).shape[0], 3)
    pcd_pred.colors = o3d.utility.Vector3dVector(colors)

    verts_pred = np.asarray(pcd_pred.points)
    verts_trgt = np.asarray(pcd_trgt.points)

    dist1 = nn_correspondance(verts_pred, verts_trgt)
    dist2 = nn_correspondance(verts_trgt, verts_pred)

    #########
    if vis_err:
        source_n_fn ="evaluate_result/precision.ply"
        target_n_fn = "evaluate_result/recall.ply"

        print("[ViewDistances] Add color coding to visualize error")
        # eval_str_viewDT = (
        #     OPEN3D_EXPERIMENTAL_BIN_PATH
        #     + "ViewDistances "
        #     + source_n_fn
        #     + " --max_distance "
        #     + str(threshold * 3)
        #     + " --write_color_back --without_gui"
        # )
        # os.system(eval_str_viewDT)
        write_color_distances(source_n_fn, pcd_pred, dist2, 3 * threshold)

        print("[ViewDistances] Add color coding to visualize error")
        # eval_str_viewDT = (
        #     OPEN3D_EXPERIMENTAL_BIN_PATH
        #     + "ViewDistances "
        #     + target_n_fn
        #     + " --max_distance "
        #     + str(threshold * 3)
        #     + " --write_color_back --without_gui"
        # )
        # os.system(eval_str_viewDT)
        write_color_distances(target_n_fn, pcd_trgt, dist1, 3 * threshold)
        ###########

    # print(dist1.max(), dist1.min())
    # print(dist2.max(), dist2.min())
    precision = np.mean((dist2 < threshold).astype('float'))
    recal = np.mean((dist1 < threshold).astype('float'))
    fscore = 2 * precision * recal / (precision + recal)
    metrics = {
        'Acc': np.mean(dist2),
        'Comp': np.mean(dist1),
        'Prec': precision,
        'Recal': recal,
        'F-score': fscore,
    }
    return metrics

[alvinjinsung]

Thank you for detailed explanation!
I have a few questions regarding the pre-processing of Scannet++ dataset

In the paper it is mentioned

"we downsample the images from 1752 × 1168 to 876 × 584 for training"
-> is this downsample_factor=0.5 for pipeline.datamanger.downsample_factor ?

"For methods that do use prior knowledge, we do something similar to MonoSDF.
We first crop the images to 1152 × 1152, then downsample them to 384 × 384 before feeding them into the Omnidata model [6] to predict geometry cues."
-> is this --mono-prior=True, --crop-mult=3 for process_nerfstudio_to_sdfstudio.py code?

"First, we scale the pose to be centered within a bounding sphere of radius equal to 1. Subsequently, we scale the camera pose by 0.8 to ensure that all scene boundaries are contained within the bounding sphere."
-> is this scale_factor=1.0 for dataparser config and scene_scale_mult=0.8 for process_nerfstudio_to_sdfstudio.py code?

I am trying to check so that I correctly preprocess the dataset and guarantee fair comparison with other methods.
Thank you and I will be waiting for your response!

[yyfz]

1. Yes, the downsample_factor is set to 0.5.
2. Please refer to the following URL for details: extract_monocular_cues.py. The resolution is already adjusted to 384x384 after extracting monocular cues.
3. We only use scale_factor=0.8. As far as I know, I have never set scene_scale_mult.

[alvinjinsung]

Thank you for quick answer!

For question 3,
where is "we scale the pose to be centered within a bounding sphere of radius equal to 1" set then?
I was assuming two sentences are meaning different things or is it?

[yyfz]

If we set scale_factor=1, I observed that some areas of the room extend beyond the bounding box with a radius of 1. Therefore, we set it to 0.8.

This adjustment ensures that when scale_factor=0.8, the pose is scaled to be centered within a bounding sphere with a radius equal to 1.

[alvinjinsung]

Thank you! It is really clearer now.

One last thing, for the omnidata,
which version did you use (pretrained model)? can you specify the file structure?

[yyfz]

We use omnidata v2 model.
This is the link to the depth model: Download Depth Model
This is the link to the normal model: Download Normal Model

[alvinjinsung]

sorry for asking again but can you provide the full evaluation code and command you used?
The results seem to be weird..

Specifically,
is mesh_pred raw ply file directly after applying marching cube to the output?
is mesh_trgt mesh_aligned_0.05.ply file from the scannet++ dataset?

Is there no scene scale issue?

I am getting weird evaluation result and it be of much help if you can provide every details of the evaluation process.

Thank you in advance!

[yyfz]

This is the complete script I use for evaluation.

# import open3d as o3d
from pathlib import Path
import json
import trimesh
import numpy as np
import open3d as o3d
from sklearn.neighbors import KDTree
import argparse
import matplotlib.pyplot as plt

parser = argparse.ArgumentParser()
parser.add_argument('--mesh', type=str, default='mesh.ply')
parser.add_argument("--vis_err", action="store_true")
args = parser.parse_args()

# data_list = ['Caterpillar', 'Courthouse']

def load_from_json(filename: Path):
    """Load a dictionary from a JSON filename.

    Args:
        filename: The filename to load from.
    """
    assert filename.suffix == ".json"
    with open(filename, encoding="UTF-8") as file:
        return json.load(file)

def nn_correspondance(verts1, verts2):
    indices = []
    distances = []
    if len(verts1) == 0 or len(verts2) == 0:
        return indices, distances

    kdtree = KDTree(verts1)
    distances, indices = kdtree.query(verts2)
    distances = distances.reshape(-1)

    return distances

def write_color_distances(path, pcd, distances, max_distance):
    o3d.utility.set_verbosity_level(o3d.utility.VerbosityLevel.Debug)
    # cmap = plt.get_cmap("afmhot")
    cmap = plt.get_cmap("hot_r")
    distances = np.array(distances)
    colors = cmap(np.minimum(distances, max_distance) / max_distance)[:, :3]
    pcd.colors = o3d.utility.Vector3dVector(colors)
    o3d.io.write_point_cloud(path, pcd)


def evaluate(mesh_pred, mesh_trgt, threshold=.05, down_sample=.02, scale=1, offset=[0., 0, 0], vis_err=False):
    pcd_trgt = o3d.geometry.PointCloud()
    pcd_pred = o3d.geometry.PointCloud()
    
    vertices = mesh_trgt.vertices[:, :3]
    vertices -= offset
    vertices *= scale
    mesh_trgt.vertices[:, :3] = vertices

    pcd_trgt.points = o3d.utility.Vector3dVector(mesh_trgt.vertices[:, :3])
    pcd_pred.points = o3d.utility.Vector3dVector(mesh_pred.vertices[:, :3])

    if down_sample:
        pcd_pred = pcd_pred.voxel_down_sample(down_sample)
        pcd_trgt = pcd_trgt.voxel_down_sample(down_sample)
    
    colors = np.random.rand(np.array(pcd_trgt.points).shape[0], 3)
    pcd_trgt.colors = o3d.utility.Vector3dVector(colors)
    colors = np.random.rand(np.array(pcd_pred.points).shape[0], 3)
    pcd_pred.colors = o3d.utility.Vector3dVector(colors)

    verts_pred = np.asarray(pcd_pred.points)
    verts_trgt = np.asarray(pcd_trgt.points)

    dist1 = nn_correspondance(verts_pred, verts_trgt)
    dist2 = nn_correspondance(verts_trgt, verts_pred)

    #########
    if vis_err:
        source_n_fn ="evaluate_result/precision.ply"
        target_n_fn = "evaluate_result/recall.ply"

        print("[ViewDistances] Add color coding to visualize error")
        # eval_str_viewDT = (
        #     OPEN3D_EXPERIMENTAL_BIN_PATH
        #     + "ViewDistances "
        #     + source_n_fn
        #     + " --max_distance "
        #     + str(threshold * 3)
        #     + " --write_color_back --without_gui"
        # )
        # os.system(eval_str_viewDT)
        write_color_distances(source_n_fn, pcd_pred, dist2, 3 * threshold)

        print("[ViewDistances] Add color coding to visualize error")
        # eval_str_viewDT = (
        #     OPEN3D_EXPERIMENTAL_BIN_PATH
        #     + "ViewDistances "
        #     + target_n_fn
        #     + " --max_distance "
        #     + str(threshold * 3)
        #     + " --write_color_back --without_gui"
        # )
        # os.system(eval_str_viewDT)
        write_color_distances(target_n_fn, pcd_trgt, dist1, 3 * threshold)
        ###########

    # print(dist1.max(), dist1.min())
    # print(dist2.max(), dist2.min())
    precision = np.mean((dist2 < threshold).astype('float'))
    recal = np.mean((dist1 < threshold).astype('float'))
    fscore = 2 * precision * recal / (precision + recal)
    metrics = {
        'Acc': np.mean(dist2),
        'Comp': np.mean(dist1),
        'Prec': precision,
        'Recal': recal,
        'F-score': fscore,
    }
    return metrics
    
# for data in data_list:
#     # mesh = o3d.io.read_point_cloud(f'./meshes/pure-neus-reproduce-{data}.ply')
#     mesh = trimesh.load_mesh(f'./meshes/pure-neus-reproduce-{data}.ply')
#     meta = load_from_json(Path(f'/mnt/data/oss_beijing/wangyifan/TanksAndTemples/{data}/transforms.json'))
#     mesh.vertices = mesh.vertices * meta["sphere_radius"] + np.array(meta["sphere_center"])
#     # o3d.io.write_point_cloud(f'/mesh/pure-neus-reproduce-{data}.ply', mesh)
#     mesh.export(f'./meshes/pure-neus-reproduce-{data}.ply')


mesh_path = args.mesh
# data = mesh_path.split('/')[-1].split('.')[0].split('-')[1]
data = mesh_path.split('/')[-1].split('.')[0].split('-')[2]
# data = mesh_path.split('/')[-1].split('.')[0].split('-')[-1]
# data = '0e75f3c4d9'
mesh = trimesh.load_mesh(mesh_path)
meta = load_from_json(Path(f'/home/bingxing2/ailab/group/3dv/data/sdf/scannetpp/{data}/meta_data.json'))
worldtogt = np.array(meta['worldtogt'])
mesh.vertices = mesh.vertices / 0.8
# mesh.vertices = mesh.vertices
mesh.vertices = (worldtogt[:3, :3] @ mesh.vertices.T).T + worldtogt[:3, 3][None, ...]
# o3d.io.write_point_cloud(f'/mesh/pure-neus-reproduce-{data}.ply', mesh)
# mesh.export(f'test.ply')

gt_mesh = trimesh.load_mesh(f'data/scannetpp/data/{data}/scans/mesh_aligned_0.05.ply')
bbox = gt_mesh.bounds
vertices = np.asarray(mesh.vertices)
mask = (vertices > bbox[0]).all(axis=1) & (vertices < bbox[1]).all(axis=1)

# Compute a modified mesh with only the vertices inside the bounding box
vertices_in_box = vertices[mask]

# Compute a list of faces after removing the ones outside the bounding box
# faces_in_box = mesh.faces[np.all(mask[mesh.faces], axis=1)]

# Create a new mesh
mesh_in_box = trimesh.Trimesh(vertices=vertices_in_box)

# mesh_in_box.export(f'test.ply')

metrics = evaluate(mesh_in_box, gt_mesh, threshold=0.025, vis_err=args.vis_err)
print(f'Evaluation result for {mesh_path}:')
print(metrics)

[alvinjinsung]

doesn't extracted first need to converted from nerfstudio corrdinate to COLMAP's/OPENCV's camera coordinate?

before mesh.vertices = (worldtogt[:3, :3] @ mesh.vertices.T).T + worldtogt[:3, 3][None, ...]
I think
mesh_vertices[:, 1] *= -1
mesh_vertices[:, 2] *= -1

needs to be done as sdfstudio_dataparser has

Convert from COLMAP's/OPENCV's camera coordinate system to nerfstudio
camera_to_worlds[:, 0:3, 1:3] *= -1

Am I right?

[alvinjinsung]

I think it is still not aligning... am I missing something?

[yyfz]

How do you obtain the worldtogt?

[alvinjinsung]

After downloading the data
I first used the tool provided in scannetpp github by running the following command: python -m dslr.undistort dslr/configs/undistort.yml

that's how I got undistorted images and transforms_undistorted.json

Then, Use the script from sdfstudio to convert it to the sdfstudio format.

This gave me meta_data.json with worldtogt in it

[yyfz]

I'm sorry, but as far as I know, my process is the same as yours. You can check: Is the mesh at the same scale? How about the rotation and translation?

[alvinjinsung]

can you check if worldtogt is same as mine?
this is worldtogt matrix for 0e75f3c4d9

    "worldtogt": [
        [
            21.800817489624023,
            0.0,
            0.0,
            4.046741008758545
        ],
        [
            0.0,
            21.800817489624023,
            0.0,
            4.931745529174805
        ],
        [
            0.0,
            0.0,
            21.800817489624023,
            -1.2336795330047607
        ],
        [
            0.0,
            0.0,
            0.0,
            1.0
        ]

I would really appreciate it