Scalability for large datasets

[malosreet]

Hello,

Thank you for this useful tool. I was wondering if it is possible to use this approach with large datasets (>=100k cells). My understanding may be wrong, so I wanted to clarify here.

In its current form line 54 of 1_seurat_pipeline.R seems to create a matrix of size (numbers of cells * number of cells), which wouldn't be feasible for larger datasets.

Would it be possible to calculate silhouette scores using sub-samples of cells at this step? Any suggestions are welcome. I was also wondering if there are plans to make the tool more scalable to larger datasets.

Thanks!

[MenonLab]

Hi there,

Apologies for the slow response. Thanks for mentioning this - we have not yet optimized chooseR for very large data sets, but are working on an update that trades the large memory allocation (cells x cells) for increased computational time. As you mentioned in your comment, the best way around this issue right now is to take advantage of substructure at the top level of your data set. In essence, if there are clearly discrete subgroups of cells at the top level, you could run chooseR separately on these subsets - usually, the subsets are also where there is more confusion about what parameter values are the most robust.