Support loading of data directly as an in-memory dgCMatrix

[LTLA]

The package currently uses HDF5-based object stores, returning SingleCellExperiment objects wrapping HDF5Matrix count matrices. This is understandable given the motivation for this package, but for general use by the wider community, it would be a good idea to also offer to return the conventional dgCMatrix. One can imagine many users would just want to analyze the data, not necessarily for testing access speed.

[kasperdanielhansen]

I don't view this package as suitable for biological analysis. IMO that would mean we are making some kind of assurance that we have made sensible choices, and I don't feel comfortable doing that.

I do think, for testing purposes, it would make sense to have easy access to sparse, in-memory data representation as well as dense, in-memory. The best way forward is IMO to make a general converter which does this efficiently and converts all assays in a SingleCellExperiment or SummarizedExperiment in a convenient form. Perhaps something like

tenx = assayRepresentation(tenx, "dgCMatrix")

This could be used outside of this package and it could also be used in code when we chunk over HDF5 objects.

[LTLA]

I'm not proposing addition of biological analysis methods to this package. My point is that, if other people want to do biological analysis on the data from this package, they would probably prefer to work with wholly in-memory representations (like the dgCMatrix) rather than paying the performance penalty of file-backed representations when they're not necessary for the relatively small datasets involved.

A coercion method is one approach for provision of these in-memory representations, though it still relies on reading through the entirety of the HDF5 file (zeroes and all). I was thinking of a separate object store, possibly a dgCMatrix saved as an RDS file, which can be directly loaded into memory.

[kasperdanielhansen]

I know you were thinking of another object store, and we could for sure consider adding it. I am asking how slow it would be to read the HDF5 into memory and do the coercion. That might be slow for the 68k dataset, but (I am guessing) really quick for all the other much smaller datasets.

update: it takes 5s on my laptop for the 3k dataset which actually is slower than I expected.
update: I forgot, but the 68k dataset is too big to be in a standard matrix, ie. as.matrix(counts(tenx)) fails. That is both a vote for having an additional data store and also a vote for having some method for reading it directly in as a dgCMatrix object.

[stephaniehicks]

Following up on GitHub issues today. Where do we stand with this?