Support Parallel computing

[geertvandeweyer]

This PR includes:

• Activation of parallel processing through snakemake scatter/gather
• strip 'chr' prefix in contigs

For benchmarking, see : https://hub.docker.com/repository/docker/cmgantwerpen/cadd/general

[visze]

thanks @geertvandeweyer cfor the PR. I am aware of it just found no time yet to look deeper into it. Maybe you can give some more explanations what was the intention with the rule specific threding limits.
But personally I like that you start using resources in snakemake!

[visze]

I really like your effort on the ressources. I have to look depper into that. But it seems to be very good. But since there are more side changes it is difficult to me to merge the whole PR

[geertvandeweyer]

thanks @geertvandeweyer cfor the PR. I am aware of it just found no time yet to look deeper into it. Maybe you can give some more explanations what was the intention with the rule specific threding limits. But personally I like that you start using resources in snakemake!

Sorry for the late reply. The rule specific threading was setup rather empirically. Some threads used high memory, so there thatwas limit (hence the resource scaling); for others (I think vep), it was mainly disk I/O. So the rules I had max'd out my SSD setup. Any higher for those would bring I/O into massive wait states... I'm not saying it's a good for all set, but it might be worth adding options to tune the threading per step somehow ?