Duration of combat with large dataset

[dklose-amsterdam]

Dear developer,

how long do you expect a matrix of 800k CpGs x 366 samples to run using the ComBat_met function? It has been a while now for me (~2h?) and I don't understand if this is normal or not.

All the best,
Dennis

[dklose-amsterdam]

(I have 8 batches)

[JmWangBio]

Hi Dennis - Thanks for the feedback. The computational time you are observing is expected, and primarily stems from the fact that the algorithm is performing beta regression. Unlike Gaussian regression, beta regression does not have a closed form solution, relying on iterations to find the optimal parameter fit.

To help estimate the runtime, I would suggest running your analysis on a small, representative subset of your features, taking note of the time cost for this subset, and linearly scaling this time to estimate the full runtime. To significantly reduce the runtime, you can leverage parallel processing by setting the ncores parameter in ComBat_met(). I hope this helps. Please feel free to let me know if you have more questions.

[JmWangBio]

@dklose-amsterdam Hi Dennis - Thanks again for your feedback. I have recently rewritten the core beta regression engine in C++ using LAPACK/BLAS routines. This has eliminated the betareg dependency and resulted in a massive speed boost.

If you have any large-scale methylation datasets, I’d love for you to give this new version a try. Any feedback would be greatly appreciated.