Metrics on SD3.5-L are much higher than reported in the paper

[Niculuse]

I use SD3.5 to generate images, and then evaluate metrics. The results are as followed:

All metrics are much higher than reported in the paper. Why?

[DnkNju]

Hello, different settings can lead to inconsistent results. This project evaluated SD3.5 using a resolution of 1024x1024, a seed of 0, 28 inference steps (consistent with the official script), an A6000 compute device, and BF16 floating-point precision. These settings must be identical to ensure identical results.

Also, based on the results you provided, only Word Accuracy and NED show a significant gap with the results reported in the paper. I believe this is reasonable, considering the differences in environment and settings.