Scaling out

[kouhate]

Hi @siddontang, I have a question about scaling out of xcodis.

Currently, xcodis relies on the database separation.
So, scaling out is limited to the number of it.

Do you have any plan to support tons of databases?

[siddontang]

Well, ledisdb uses 1 byte for db index, so now it may only support maximum 256 dbs (my fault). Supporting larger (using 2 byte for db index) must upgrade the backend data store, it is dangerous to do this.

btw, I think 256 dbs may be OK too.

[siddontang]

@kouhate , now xcodis and ledisdb support configuring databases, (slot_num in xcodis config, databases in ledisdb config), you can decide how many databases you want to use.

Because of my old fault, ledisdb only supports maximum 256, and I don't want to change this now to prevent massive backend data upgrade.

But I still think 256 is ok, or even 64 too. If you use 64 machines (one runs a single ledisdb server for a database), I think the whole data storage is very very big.

[kouhate]

@siddontang , thank you!

When I set databases = 256, I got this error.

Type mismatch for 'config.Config.databases': Value '256' is out of range for uint8.

So your config message of Maximum databases should be 255 not 256.
After changing databases = 255, I couldn't switch to index 255.
It's ok because you mentioned in config dbindex must be in [0, databases - 1].

localhost:6380> select 255
(error) ERR invalid db index 255
localhost:6380> select 254
OK

This is a trivial thing, but ledisdb support [0, 254], not [0, 255].

Btw, I think 256 dbs are not yet enough because distributed slot is still limited to 256.
It is true the whole storage become bigger, but the chance which some busy keys sit on the same slot may at least 1/256.

Redis cluster(not stable) support 16384, so probably those chances are 1/16384.
What I worry about is not the storage size but the equalization of the distribution feature.

[siddontang]

Hi @kouhate, sorry the bug for 256 config.

the chance for busy keys on the same slot may be big, I think we can customize the routing rule, e.g, detect the busy key and set their own routing rule, not just mode 256.

btw, the default codis slot num is 1024, so the change is 1/1024, a little big too.

[kouhate]

Hi @siddontang, sorry for my long silence.

Custom routing for busy key looks good, but this may bring some difficulty for maintenance in the future.
Is it possible to automate the busy key detection and distribute them equally?
If we always have to detect and distribute manually, the bigger keymap may get out of hand.

I think the same is true for big slot. If some keys are not busy but have too big values, the probability of which those keys sit on the same slot is also 1/256 at least.

As we know, custom routing needs a routing table, and always have to check whether the key is in it every time we send a command.
There is few matter while the table is small, but the bigger will be harmful, I think.

Where will you intend to put the table?
Only on zookeeper may decrease performance, but caching on xcodis will be also problematic.
Perhaps, without direct checking or every sec snapshot, xcodis have to keep the cache of not existing keys in the table.

[siddontang]

Hi @kouhate, there is no perfect solution, so we have to weigh the cost of different ways and find a acceptable one. A NoSQL can not fit all needs in the world. :-)

I don't want people to store huge big data in one key, migrating this key is a horrible thing and I think the architect is responsible for designing a proper data structure or data usage scenario.

Customizing busy key routing rule may not be a good way, I still want to use a big slot instead, of course, 256 is too small, codis uses 1024, and I heard that elasticsearch also uses 1024 (maybe wrong), but redis uses 16384 which I think too big, 😄

At the same time, even some busy keys mapped into same slot, they may not kill the performance badly, the backend rocksdb has a awesome performance, and we can read data from slaves if we don't consider data consistency too much. Even we can use cache to solve it.

If we want to use a big routing table, saving it into zookeeper is not a wise decision, but we can save it in a LedisDB which only serves xcodis, and use zookeeper to coordinate them.