Infinistore Server crashed after free the client - free(): invalid pointer #179
Description
[2025-05-15 20:19:26.604] [infini] [info] ServerConfig: service_port=12345, manage_port=8088, log_level='debug', dev_name='mlx5_1', ib_port=1, link_type='Ethernet', prealloc_size=16, minimal_allocate_size=64, auto_increase=False, evict_min_threshold=0.6, evict_max_threshold=0.8, evict_interval=5, hint_gid_index=7
[2025-05-15 20:19:26.605] [infini] [info] open rdma device mlx5_1, link_type Ethernet, hint_gid_index 7,
[2025-05-15 20:19:26.606] [infini] [info] Can't find or failed to open the specified device, try to open the default device mlx5_2
[2025-05-15 20:19:26.632] [infini] [warning] [rdma.cpp:100] RoCE choose user specified gid index 7
[2025-05-15 20:19:26.632] [infini] [info] gid index 7, gid 33.2.121.163
[2025-05-15 20:19:26.632] [infini] [info] Memory pool size: 17179869184 bytes, block size: 65536 bytes, total blocks: 262144, it may take a while
[2025-05-15 20:19:26.632] [infini] [info] Memory pool allocated at 0x7f80af28f000
[2025-05-15 20:19:29.043] [infini] [info] register server done
[2025-05-15 20:19:29.043] [infini] [info] set oom_score_adj to -1000 to prevent OOM
[2025-05-15 20:19:29.044] [infini] [warning] server started
[2025-05-15 20:27:45.331] [infini] [info] new connection...
[2025-05-15 20:27:45.390] [infini] [info] free client resources
[2025-05-15 20:44:39.398] [infini] [info] new connection...
[2025-05-15 20:44:39.453] [infini] [info] free client resources
[2025-05-15 20:45:30.034] [infini] [info] new connection...
[2025-05-15 20:45:30.089] [infini] [info] free client resources
[2025-05-15 20:49:48.545] [infini] [info] new connection...
[2025-05-15 20:49:48.572] [infini] [debug] HEADER: op: E, body_size :30
[2025-05-15 20:49:48.572] [infini] [debug] reading body, bytes_read: 0, expected_bytes: 30
[2025-05-15 20:49:48.572] [infini] [debug] body read done, size 30
[2025-05-15 20:49:48.572] [infini] [info] do rdma exchange...
[2025-05-15 20:49:48.575] [infini] [debug] local rdma_conn_info: psn: 0, qpn: 530, gid: 33.2.121.163, enum mtu: 5
[2025-05-15 20:49:48.575] [infini] [debug] remote rdma_conn_info: psn: 0, qpn: 529, gid: 33.2.57.165, enum mtu: 5
[2025-05-15 20:49:48.577] [infini] [info] RDMA exchange done
[2025-05-15 20:49:49.207] [infini] [info] new connection...
[2025-05-15 20:49:49.235] [infini] [debug] HEADER: op: E, body_size :30
[2025-05-15 20:49:49.235] [infini] [debug] reading body, bytes_read: 0, expected_bytes: 30
[2025-05-15 20:49:49.235] [infini] [debug] body read done, size 30
[2025-05-15 20:49:49.235] [infini] [info] do rdma exchange...
[2025-05-15 20:49:49.238] [infini] [debug] local rdma_conn_info: psn: 2116118, qpn: 531, gid: 33.2.121.163, enum mtu: 5
[2025-05-15 20:49:49.238] [infini] [debug] remote rdma_conn_info: psn: 10238434, qpn: 532, gid: 33.2.57.165, enum mtu: 5
[2025-05-15 20:49:49.239] [infini] [info] RDMA exchange done
[2025-05-15 20:49:49.843] [infini] [info] free client resources
[2025-05-15 20:50:19.880] [infini] [info] new connection...
[2025-05-15 20:50:19.907] [infini] [debug] HEADER: op: E, body_size :30
[2025-05-15 20:50:19.907] [infini] [debug] reading body, bytes_read: 0, expected_bytes: 30
[2025-05-15 20:50:19.907] [infini] [debug] body read done, size 30
[2025-05-15 20:50:19.907] [infini] [info] do rdma exchange...
[2025-05-15 20:50:19.910] [infini] [debug] local rdma_conn_info: psn: 5515815, qpn: 532, gid: 33.2.121.163, enum mtu: 5
[2025-05-15 20:50:19.910] [infini] [debug] remote rdma_conn_info: psn: 13658313, qpn: 536, gid: 33.2.57.165, enum mtu: 5
[2025-05-15 20:50:19.910] [infini] [info] RDMA exchange done
[2025-05-15 20:50:20.583] [infini] [info] free client resources
[2025-05-15 20:50:50.610] [infini] [info] new connection...
[2025-05-15 20:50:50.638] [infini] [debug] HEADER: op: E, body_size :30
[2025-05-15 20:50:50.638] [infini] [debug] reading body, bytes_read: 0, expected_bytes: 30
[2025-05-15 20:50:50.638] [infini] [debug] body read done, size 30
[2025-05-15 20:50:50.638] [infini] [info] do rdma exchange...
[2025-05-15 20:50:50.641] [infini] [debug] local rdma_conn_info: psn: 10238434, qpn: 533, gid: 33.2.121.163, enum mtu: 5
[2025-05-15 20:50:50.641] [infini] [debug] remote rdma_conn_info: psn: 2809818, qpn: 539, gid: 33.2.57.165, enum mtu: 5
[2025-05-15 20:50:50.641] [infini] [info] RDMA exchange done
[2025-05-15 20:50:51.091] [infini] [info] free client resources
[2025-05-15 20:51:21.118] [infini] [info] new connection...
[2025-05-15 20:51:21.146] [infini] [debug] HEADER: op: E, body_size :30
[2025-05-15 20:51:21.146] [infini] [debug] reading body, bytes_read: 0, expected_bytes: 30
[2025-05-15 20:51:21.146] [infini] [debug] body read done, size 30
[2025-05-15 20:51:21.146] [infini] [info] do rdma exchange...
[2025-05-15 20:51:21.148] [infini] [debug] local rdma_conn_info: psn: 11225430, qpn: 536, gid: 33.2.121.163, enum mtu: 5
[2025-05-15 20:51:21.148] [infini] [debug] remote rdma_conn_info: psn: 4389417, qpn: 544, gid: 33.2.57.165, enum mtu: 5
[2025-05-15 20:51:21.149] [infini] [info] RDMA exchange done
[2025-05-15 20:51:21.509] [infini] [info] free client resources
[2025-05-15 20:51:51.537] [infini] [info] new connection...
[2025-05-15 20:51:51.562] [infini] [debug] HEADER: op: E, body_size :30
[2025-05-15 20:51:51.562] [infini] [debug] reading body, bytes_read: 0, expected_bytes: 30
[2025-05-15 20:51:51.562] [infini] [debug] body read done, size 30
[2025-05-15 20:51:51.562] [infini] [info] do rdma exchange...
[2025-05-15 20:51:51.564] [infini] [debug] local rdma_conn_info: psn: 11581620, qpn: 539, gid: 33.2.121.163, enum mtu: 5
[2025-05-15 20:51:51.564] [infini] [debug] remote rdma_conn_info: psn: 8497408, qpn: 548, gid: 33.2.57.165, enum mtu: 5
[2025-05-15 20:51:51.565] [infini] [info] RDMA exchange done
[2025-05-15 20:51:51.963] [infini] [info] free client resources
[2025-05-15 20:52:21.982] [infini] [info] new connection...
[2025-05-15 20:52:22.010] [infini] [debug] HEADER: op: E, body_size :30
[2025-05-15 20:52:22.010] [infini] [debug] reading body, bytes_read: 0, expected_bytes: 30
[2025-05-15 20:52:22.010] [infini] [debug] body read done, size 30
[2025-05-15 20:52:22.010] [infini] [info] do rdma exchange...
[2025-05-15 20:52:22.012] [infini] [debug] local rdma_conn_info: psn: 13658313, qpn: 540, gid: 33.2.121.163, enum mtu: 5
[2025-05-15 20:52:22.012] [infini] [debug] remote rdma_conn_info: psn: 5067550, qpn: 552, gid: 33.2.57.165, enum mtu: 5
[2025-05-15 20:52:22.012] [infini] [info] RDMA exchange done
[2025-05-15 20:52:22.359] [infini] [info] free client resources
[2025-05-15 20:52:52.386] [infini] [info] new connection...
[2025-05-15 20:52:52.410] [infini] [debug] HEADER: op: E, body_size :30
[2025-05-15 20:52:52.410] [infini] [debug] reading body, bytes_read: 0, expected_bytes: 30
[2025-05-15 20:52:52.410] [infini] [debug] body read done, size 30
[2025-05-15 20:52:52.410] [infini] [info] do rdma exchange...
[2025-05-15 20:52:52.412] [infini] [debug] local rdma_conn_info: psn: 6103629, qpn: 543, gid: 33.2.121.163, enum mtu: 5
[2025-05-15 20:52:52.412] [infini] [debug] remote rdma_conn_info: psn: 658986, qpn: 557, gid: 33.2.57.165, enum mtu: 5
[2025-05-15 20:52:52.413] [infini] [info] RDMA exchange done
[2025-05-15 20:52:52.823] [infini] [info] free client resources
[2025-05-15 20:53:22.843] [infini] [info] new connection...
[2025-05-15 20:53:22.871] [infini] [debug] HEADER: op: E, body_size :30
[2025-05-15 20:53:22.871] [infini] [debug] reading body, bytes_read: 0, expected_bytes: 30
[2025-05-15 20:53:22.871] [infini] [debug] body read done, size 30
[2025-05-15 20:53:22.871] [infini] [info] do rdma exchange...
[2025-05-15 20:53:22.872] [infini] [debug] local rdma_conn_info: psn: 7114140, qpn: 544, gid: 33.2.121.163, enum mtu: 5
[2025-05-15 20:53:22.872] [infini] [debug] remote rdma_conn_info: psn: 12330751, qpn: 560, gid: 33.2.57.165, enum mtu: 5
[2025-05-15 20:53:22.873] [infini] [info] RDMA exchange done
[2025-05-15 20:53:23.216] [infini] [info] free client resources
[2025-05-15 20:53:53.236] [infini] [info] new connection...
[2025-05-15 20:53:53.262] [infini] [debug] HEADER: op: E, body_size :30
[2025-05-15 20:53:53.262] [infini] [debug] reading body, bytes_read: 0, expected_bytes: 30
[2025-05-15 20:53:53.262] [infini] [debug] body read done, size 30
[2025-05-15 20:53:53.262] [infini] [info] do rdma exchange...
[2025-05-15 20:53:53.264] [infini] [debug] local rdma_conn_info: psn: 2809818, qpn: 547, gid: 33.2.121.163, enum mtu: 5
[2025-05-15 20:53:53.264] [infini] [debug] remote rdma_conn_info: psn: 647507, qpn: 565, gid: 33.2.57.165, enum mtu: 5
[2025-05-15 20:53:53.264] [infini] [info] RDMA exchange done
[2025-05-15 20:53:53.603] [infini] [info] free client resources
[2025-05-15 20:54:23.623] [infini] [info] new connection...
[2025-05-15 20:54:23.650] [infini] [debug] HEADER: op: E, body_size :30
[2025-05-15 20:54:23.650] [infini] [debug] reading body, bytes_read: 0, expected_bytes: 30
[2025-05-15 20:54:23.650] [infini] [debug] body read done, size 30
[2025-05-15 20:54:23.650] [infini] [info] do rdma exchange...
[2025-05-15 20:54:23.652] [infini] [debug] local rdma_conn_info: psn: 14207609, qpn: 548, gid: 33.2.121.163, enum mtu: 5
[2025-05-15 20:54:23.652] [infini] [debug] remote rdma_conn_info: psn: 1108139, qpn: 568, gid: 33.2.57.165, enum mtu: 5
[2025-05-15 20:54:23.652] [infini] [info] RDMA exchange done
[2025-05-15 20:54:23.995] [infini] [info] free client resources
[2025-05-15 20:54:54.013] [infini] [info] new connection...
[2025-05-15 20:54:54.038] [infini] [debug] HEADER: op: E, body_size :30
[2025-05-15 20:54:54.038] [infini] [debug] reading body, bytes_read: 0, expected_bytes: 30
[2025-05-15 20:54:54.038] [infini] [debug] body read done, size 30
[2025-05-15 20:54:54.038] [infini] [info] do rdma exchange...
[2025-05-15 20:54:54.039] [infini] [debug] local rdma_conn_info: psn: 6800625, qpn: 551, gid: 33.2.121.163, enum mtu: 5
[2025-05-15 20:54:54.039] [infini] [debug] remote rdma_conn_info: psn: 6944197, qpn: 573, gid: 33.2.57.165, enum mtu: 5
[2025-05-15 20:54:54.040] [infini] [info] RDMA exchange done
[2025-05-15 20:54:54.395] [infini] [info] free client resources
[2025-05-15 20:55:24.414] [infini] [info] new connection...
[2025-05-15 20:55:24.438] [infini] [debug] HEADER: op: E, body_size :30
[2025-05-15 20:55:24.438] [infini] [debug] reading body, bytes_read: 0, expected_bytes: 30
[2025-05-15 20:55:24.438] [infini] [debug] body read done, size 30
[2025-05-15 20:55:24.438] [infini] [info] do rdma exchange...
[2025-05-15 20:55:24.440] [infini] [debug] local rdma_conn_info: psn: 4389417, qpn: 552, gid: 33.2.121.163, enum mtu: 5
[2025-05-15 20:55:24.440] [infini] [debug] remote rdma_conn_info: psn: 9055212, qpn: 576, gid: 33.2.57.165, enum mtu: 5
[2025-05-15 20:55:24.440] [infini] [info] RDMA exchange done
[2025-05-15 20:55:24.764] [infini] [info] free client resources
[2025-05-15 20:55:54.783] [infini] [info] new connection...
[2025-05-15 20:55:54.806] [infini] [debug] HEADER: op: E, body_size :30
[2025-05-15 20:55:54.806] [infini] [debug] reading body, bytes_read: 0, expected_bytes: 30
[2025-05-15 20:55:54.806] [infini] [debug] body read done, size 30
[2025-05-15 20:55:54.806] [infini] [info] do rdma exchange...
[2025-05-15 20:55:54.807] [infini] [debug] local rdma_conn_info: psn: 11939177, qpn: 555, gid: 33.2.121.163, enum mtu: 5
[2025-05-15 20:55:54.807] [infini] [debug] remote rdma_conn_info: psn: 2653298, qpn: 581, gid: 33.2.57.165, enum mtu: 5
[2025-05-15 20:55:54.808] [infini] [info] RDMA exchange done
[2025-05-15 20:55:55.122] [infini] [info] free client resources
[2025-05-15 20:56:25.142] [infini] [info] new connection...
[2025-05-15 20:56:25.170] [infini] [debug] HEADER: op: E, body_size :30
[2025-05-15 20:56:25.170] [infini] [debug] reading body, bytes_read: 0, expected_bytes: 30
[2025-05-15 20:56:25.170] [infini] [debug] body read done, size 30
[2025-05-15 20:56:25.170] [infini] [info] do rdma exchange...
[2025-05-15 20:56:25.171] [infini] [debug] local rdma_conn_info: psn: 3021149, qpn: 556, gid: 33.2.121.163, enum mtu: 5
[2025-05-15 20:56:25.171] [infini] [debug] remote rdma_conn_info: psn: 15503146, qpn: 584, gid: 33.2.57.165, enum mtu: 5
[2025-05-15 20:56:25.172] [infini] [info] RDMA exchange done
[2025-05-15 20:56:25.512] [infini] [info] free client resources
[2025-05-15 20:56:55.530] [infini] [info] new connection...
[2025-05-15 20:56:55.558] [infini] [debug] HEADER: op: E, body_size :30
[2025-05-15 20:56:55.558] [infini] [debug] reading body, bytes_read: 0, expected_bytes: 30
[2025-05-15 20:56:55.558] [infini] [debug] body read done, size 30
[2025-05-15 20:56:55.558] [infini] [info] do rdma exchange...
[2025-05-15 20:56:55.559] [infini] [debug] local rdma_conn_info: psn: 8497408, qpn: 559, gid: 33.2.121.163, enum mtu: 5
[2025-05-15 20:56:55.559] [infini] [debug] remote rdma_conn_info: psn: 11388669, qpn: 589, gid: 33.2.57.165, enum mtu: 5
[2025-05-15 20:56:55.560] [infini] [info] RDMA exchange done
[2025-05-15 20:56:55.911] [infini] [info] free client resources
free(): invalid pointer
[2025-05-15 20:56:55.920] [infini] [info] Interrupt signal (6) received.
[2025-05-15 20:56:55.921] [infini] [error] [utils.cpp:99] Stacktrace:
0# signal_handler(int) in /usr/local/lib/python3.10/dist-packages/infinistore/_infinistore.cpython-310-x86_64-linux-gnu.so
1# 0x00007F84B62BF520 in /lib/x86_64-linux-gnu/libc.so.6
2# pthread_kill in /lib/x86_64-linux-gnu/libc.so.6
3# raise in /lib/x86_64-linux-gnu/libc.so.6
4# abort in /lib/x86_64-linux-gnu/libc.so.6
5# 0x00007F84B6306677 in /lib/x86_64-linux-gnu/libc.so.6
6# 0x00007F84B631DCFC in /lib/x86_64-linux-gnu/libc.so.6
7# 0x00007F84B631FA44 in /lib/x86_64-linux-gnu/libc.so.6
8# free in /lib/x86_64-linux-gnu/libc.so.6
9# Client::~Client() in /usr/local/lib/python3.10/dist-packages/infinistore/_infinistore.cpython-310-x86_64-linux-gnu.so
10# on_close(uv_handle_s*) in /usr/local/lib/python3.10/dist-packages/infinistore/_infinistore.cpython-310-x86_64-linux-gnu.so
11# uv_run in /usr/local/lib/python3.10/dist-packages/uvloop/loop.cpython-310-x86_64-linux-gnu.so
12# 0x00007F84B041A8C3 in /usr/local/lib/python3.10/dist-packages/uvloop/loop.cpython-310-x86_64-linux-gnu.so
13# 0x00007F84B041ACA0 in /usr/local/lib/python3.10/dist-packages/uvloop/loop.cpython-310-x86_64-linux-gnu.so
14# 0x00007F84B0403983 in /usr/local/lib/python3.10/dist-packages/uvloop/loop.cpython-310-x86_64-linux-gnu.so
15# 0x00007F84B042BD1E in /usr/local/lib/python3.10/dist-packages/uvloop/loop.cpython-310-x86_64-linux-gnu.so
16# _PyEval_EvalFrameDefault in /usr/bin/python3
17# _PyFunction_Vectorcall in /usr/bin/python3
18# _PyEval_EvalFrameDefault in /usr/bin/python3
19# 0x000055D125D3CF56 in /usr/bin/python3
20# PyEval_EvalCode in /usr/bin/python3
21# 0x000055D125D63808 in /usr/bin/python3
22# 0x000055D125D5E00F in /usr/bin/python3
23# 0x000055D125D635A5 in /usr/bin/python3
24# _PyRun_SimpleFileObject in /usr/bin/python3
25# _PyRun_AnyFileObject in /usr/bin/python3
26# Py_RunMain in /usr/bin/python3
27# Py_BytesMain in /usr/bin/python3
28# 0x00007F84B62A6D90 in /lib/x86_64-linux-gnu/libc.so.6
29# __libc_start_main in /lib/x86_
Version: 0.2.42