need some tips about benmark results

[calvin2021y]

hi @freeekanayaka

I hope you are doing well and in good health.

I recently started playing around with raft again, I notice the pubic api almost is the same with dqlite master branch.

So i write a tiny test with 8 byte count, do some simple benchmark. (use tmpfs as data dir)

I only change the header search dir and library file, compare dqlite with cowsql raft.

dqlite around 63.82k, but cowsql raft around 8.17k.

I hope to know the specific reason, I notice with LIBRAFT_TRACE=1 the cowsql give very details logs, dqlite not response for this env.

What other factors do you think influenced the results?

[calvin2021y]

I set threshold=65536, trailing=65536. not sure related.

3 node cluster

[freeekanayaka]

Hello @calvin2021y, I'm good, hope you too!

Is it possible for you to share the source code of the benchmark?

[calvin2021y]

Thanks and I am good too.

the benchmark code is a massy and I need time to tidy it up. (I will try modify and send to you, If you can accept binary I can send it ASAP)

After run more test, I guess it cloud be related to my code. here is a updated results:

Number of parallel connections	COWSQL RAFT	DQLITE RAFT
1	18.68k QPS, 569.65us	20.35k,507.99u
2	30.71k QPS,806.57us	33.09k, 1.46ms
5	40.71k QPS, 1.13ms	46.96k,1.36ms
10	47.97k QPS, 1.24ms	57.54k,2.54ms
20	55.72k QPS,3.04ms	62.10k, 2.69ms
50	8.49k QPS, 12.68ms	12.23k,8.82ms
100	8.87k QPS, 16.24ms	18.18k,10.27ms
200	11.01k QPS, 20.97ms	28.65k,9.87ms
500	11.95k QPS, 44.63ms	63.43k,10.41ms
1000	8.24k QPS, 121.44ms	57.44k,19.41ms
2000	8.50k QPS, 234.5ms	61.23k,34.13ms
5000	7.98k QPS, 908.83ms	50.69k,98.17ms

The time is Avg time of each request. I run each test many time and the result is stable. I try 10s duration and 300s duration, restul also stable.

The cowsql version test run few test ( 5000 connection level) will blocked and no response.

I test with threshold=65536, trailing=65536, timeout=50ms, election=200ms. reduce threshold to 1024 will make the speed much slow. (I guess cause by too much async snapshot)

I change the snapshot size to 16 byte, contains 2 int64_t number.

[calvin2021y]

for each connection, they wait the log committed then return result to client, client start next request to modify FSM value. they all connected to leader.