Question on experiment data

[lingxin12]

Why is the PSNR 19.22 while training on LLFF with vanilla 3DGS (3000 iterations). But the work is lower than it.

[Fictionarry]

Hi, I don't know how you get the results. Maybe you can list more details like what kind of initialization and which codebase are in use. I guess the setting where you get the results is different from us.

[hahahaha5606]

That's quite interesting. I have got a similar issue. I always got different results when training on the same data (for example, LLFF / flower). And I have tried to find any random events that may cause such issue, my experiment environment is almost the same with the environment.yml, i.e. cuda 11.3. Could anyone tell me why and how to get stable results? @lingxin12 @Fictionarry

[Fictionarry]

That's quite interesting. I have got a similar issue. I always got different results when training on the same data (for example, LLFF / flower). And I have tried to find any random events that may cause such issue, my experiment environment is almost the same with the environment.yml, i.e. cuda 11.3. Could anyone tell me why and how to get stable results?

Randomness comes from two parts: 1) the random point cloud initialization; and 2) randomness in the training process, including optimizing, Gaussian splitting, and so on. As discussed in README, you can use some more robust initialization methods like the patch-match MVS used in FSGS (may not provide sufficient points in some situations like 3-view DTU), or try our new work CoR-GS (https://github.com/jiaw-z/CoR-GS) that aims to solve these randomness problems.

Additionally, I noticed that the results you provided seem to be abnormally low. To reproduce your problem, I have just tried using the current repo on my server to run "flowers" five times, the results of which are all better than the provided, with PSNR all above 19 and mean PSNR about 19.2. If the PSNR you reproduced is always lower than 19, please see if there is any inconsistency in your code with the current repo. If not, I have no idea either.

[hahahaha5606]

@Fictionarry
Thank you for your reply! It helps a lot, I will give it a try.

Besides, the reason why PSNR is abnormally low is that I have changed some hyperparameters in configuration.

[lingxin12]

That's quite interesting. I have got a similar issue. I always got different results when training on the same data (for example, LLFF / flower). And I have tried to find any random events that may cause such issue, my experiment environment is almost the same with the environment.yml, i.e. cuda 11.3. Could anyone tell me why and how to get stable results?

Randomness comes from two parts: 1) the random point cloud initialization; and 2) randomness in the training process, including optimizing, Gaussian splitting, and so on. As discussed in README, you can use some more robust initialization methods like the patch-match MVS used in FSGS (may not provide sufficient points in some situations like 3-view DTU), or try our new work CoR-GS (https://github.com/jiaw-z/CoR-GS) that aims to solve these randomness problems.

Additionally, I noticed that the results you provided seem to be abnormally low. To reproduce your problem, I have just tried using the current repo on my server to run "flowers" five times, the results of which are all better than the provided, with PSNR all above 19 and mean PSNR about 19.2. If the PSNR you reproduced is always lower than 19, please see if there is any inconsistency in your code with the current repo. If not, I have no idea either.

Yes, I note that the number of points at initialization is not same, one is 5024 (DNGaussian), another (work on initial 3dgs) is 10091.

[Fictionarry]

Yes, I note that the number of points at initialization is not same, one is 5024 (DNGaussian), another (work on initial 3dgs) is 10091.

OK, now I can confirm where the problem you posted exactly is... Take the scene "fern" you used as example, the abnormally higher results you get through the original 3DGS are initialized with a full-view SfM point cloud (10091 points), which includes the test data in our setting. In other words, the test data has been partially mixed in your training. So, the results you got are actually wrong. The initialization we used is a random point cloud (5024 points), without such a data leakage problem. To fairly compare the methods, the full-view SfM point cloud should not be used.