What are the datastores in roadmap.

[seeni-dev]

@Alexander255
I get that FileSystemDataStore is almost done. But what are the other DataStores in priority list?

====
We can fix some important <5 datastores and then we can concentrate on it.

[ntninja]

That it depends if your more interested in developing more datastore related stuff or getting py-ipfs off-the-ground. 😉

In the later case, the next step would be to add an https://github.com/ipfs/go-ds-flatfs compatible adapter/shim so that we may access the contents of the go-ipfs block storage.

wrt to py-ipfs there are some other backends that I see as useful in the future, but not strictly necessary right now:

• datastore.pylru – For in-memory caching recently used blocks
• datastore.badgerds – For accessing the maybe-at-some-point-default new datastore of go-ipfs; I'm not aware of any Python implementations/bindings for this however
• datastore.leveldb – For accessing the “datastore” subdirectory of go-ipfs (admittedly I don't actually know what that contains atm)

[ntninja]

I have pushed my current branch to https://github.com/alexander255/py-datastore. It's completely untested atm, but should give you an idea of how I envision the library's interface will look like after #8 when taking into account #11. If you want to port more datastores it may be a good idea to target that library version for that, but things may break as there is no definite design yet.

[seeni-dev]

I am focussed on py-ipfs only. So we can concentrate on the adapter.

[ntninja]

@seenivasanseeni: Do you have any time/interest in working on some IPFS related stuff?

[adamjsawicki]

@Alexander255 Do you want the py-ds-flatfs to be its own PyPi package? If so, should it exactly mirror the functionality in https://github.com/ipfs/go-ds-flatfs?

[seeni-dev]

@Alexander255 I am interested. But I am little busy now due to work. Is there anything important that we need to finish fast?

[ntninja]

@seenivasanseeni: Could you review #13?

[ntninja]

@swixx: I think its fine to add as an adapter to this repo. The nested path adapter is probably a good starting point for this – unless I'm missing something you will really just need to change the nest_key method to instead dispatch to the relevant flatfs sharding function (prefix, suffix, next-to-last/*) + add some code to read and write the SHARDING file so that you know which one to call.

[adamjsawicki]

@Alexander255 Thanks.

So for _FlatFsAdapter, the __init__ method is the following

def __init__(self, *args, path_to_sharding_file: str, **kwargs):

Or is the SHARDING file specified elsewhere?

[ntninja]

Looks like it's hard-coded: https://github.com/ipfs/go-ds-flatfs/blob/1bc2b870c4bcedee0f75eb9b9fc6fad901ac1eb0/shard.go#L17 & https://github.com/ipfs/go-ds-flatfs/blob/1bc2b870c4bcedee0f75eb9b9fc6fad901ac1eb0/shard.go#L114

So either do the same or keep the parameter, but let it default to that name.

[adamjsawicki]

What is the difference between Filesystem and a normal adapter?

As far as I can tell, it exists in isolation because it doesn't follow the pattern of being able to use both a BinaryDatastore and an ObjectDatastore. Is this accurate?

[ntninja]

@swixx: In the nomenclature I've ended up using for this project an “Adapter” is something that applies some transformation to an input key, then forwards it to an underlying child datastore (for some, like sharding, tiered and mount, there may be more than one possible datastore that may receive such a request) – it could also apply some transformation to the result, but none of our current adapters do that. Similarly a “Serializer” is an adapter that translates object datastore values to binary and vice-versa. A plain “Datastore” (or datastore backend, maybe?) is something that actually persists or shares the received values to some place outside of the datastore library (such as your disk, a database or an in-memory store) – it's the thing this library abstracts over, by providing generic transformations and interfaces on top.
Sorry this caused confusion! I guess this would be a good thing to add to the documentation‽

BTW: Any kind of datastore can be either binary or object, but most adapters are both, hence why they use that mixin subclass construction for maximum code sharing.