Scarf core package

[oskbor]

Usually scarf methods looks a bit like this

my_func(cell_key: str, feat_key:str):
	# load stuff from datastore
	# do stuff
	# save results to datastore

it would be awesome if we could extract all the "do stuff" to a core library inside scarf. So the new code in scarf would look something like

import do_stuff from "scarf.core"

my_func(cell_key: str, feat_key:str):
	# load stuff from datastore
	results = do_stuff(loaded_stuff)
	# save results to datastore

Basically all the functions in core should aim to be pure, they return their results rather than writing them to disk.
For complex things, we could return a stream or iterable that can be consumed or written to disk as needed.

Installing the scarf core package should not include all the graphing, progress bars etc.

Other loose thoughts

• Have a few data types. Vector, matrix, HNSW index?, graph?, featureSet, cellSet. all of these should be hashable.
• Nothing shall ever be overwritten. If its the same inputs, return the old output. Store every run to disk
• Every task does something like this:
  1. All inputs must be hashable. All tasks have an id. On call, hash the input and check if store.has("<id>:<hash>"). If so return the cached content.
  2. If there is nothing in the store, compute the object, store it in the store and return it (or return its hash).

type DataType = "cellSubset" | "featureSubset" | "vector" | "matrix" | "etc" | "sparseMatrix"
type Store = {
  get(key: string) : DataType | null,
  has(key: string): boolean,
  save(key: string, val: DataType),
}