Bug Description
taosdump 3.4.1.6 fails an entire database dump with return code 255 when a table receives writes while it is being dumped:
ERROR: writeResultToAvro() LN3774, actual dump out rows not equal to query count. actual dump out: 5000114, count 5000097 dbName=test tbName=d0
ERROR: super table (meters) dump failed, 1 child table(s) have errors. close taos connections...
ERROR: Database test dumped failed
For each child table, writeResultToAvro() takes SELECT COUNT(*) (queryCount) at the start, then scans and counts the rows (fetchedCount). If fetchedCount != queryCount it is fatal (return -1), which fails the child table -> the super table -> the whole database. Concurrent inserts during the dump make fetchedCount > queryCount (more rows, not truncation), so a single actively-written table aborts the whole dump.
Regression introduced by PR #35096 (commit 807472524a); present in 3.4.1.6, and a 3.3.6.9 build completes the identical workload with return code 0.
To Reproduce
Minimal local reproduction — a table written to while it is dumped. Needs a high-rate writer; a slow one-insert-at-a-time writer often won't trip it.
docker run -d --name td --hostname td tdengine/tsdb:3.4.1.6
# wait until the dnode is ready, then:
docker exec td taosBenchmark -y -t 2 -n 5000000 # db test, stable meters, tables d0/d1
docker exec -d td sh -c 'yes "INSERT INTO test.d0 VALUES (now,1,220,0.5);" | taos' # concurrent writer
docker exec td sh -c 'mkdir -p /tmp/dump && taosdump -D test -o /tmp/dump'; echo "rc=$?"
-> rc=255 with the error above. Also reproduces with out-of-order/backfill writes (now-400d), and -E <past> does not prevent it when backdated writes land inside the window.
Expected Behavior
A count difference caused by concurrent ingestion is expected for a live time-series database, not corruption. taosdump should not fail the whole database over it.
Screenshots
Not attaching log screenshots to avoid exposing sensitive information; the relevant log is included as text above.
Environment:
- OS: Linux (Kubernetes pods, Ubuntu-based)
- Memory: ~31,3 GB
- Disk: ~496 GB
- CPU: 16
- TDengine Server Version: 3.4.1.6
- taosdump Version: 3.4.1.6
- Architecture: Linux-x64
- Deployment: Kubernetes triggered by Airflow DAG
- Observed in production on a database with ~9,600 subtables and ~3.4B rows.
Additional Context
Bug Description
taosdump3.4.1.6 fails an entire database dump with return code 255 when a table receives writes while it is being dumped:For each child table,
writeResultToAvro()takesSELECT COUNT(*)(queryCount) at the start, then scans and counts the rows (fetchedCount). IffetchedCount != queryCountit is fatal (return -1), which fails the child table -> the super table -> the whole database. Concurrent inserts during the dump makefetchedCount > queryCount(more rows, not truncation), so a single actively-written table aborts the whole dump.Regression introduced by PR #35096 (commit
807472524a); present in 3.4.1.6, and a 3.3.6.9 build completes the identical workload with return code 0.To Reproduce
Minimal local reproduction — a table written to while it is dumped. Needs a high-rate writer; a slow one-insert-at-a-time writer often won't trip it.
->
rc=255with the error above. Also reproduces with out-of-order/backfill writes (now-400d), and-E <past>does not prevent it when backdated writes land inside the window.Expected Behavior
A count difference caused by concurrent ingestion is expected for a live time-series database, not corruption. taosdump should not fail the whole database over it.
Screenshots
Not attaching log screenshots to avoid exposing sensitive information; the relevant log is included as text above.
Environment:
Additional Context
tools/taos-tools/src/taosdump.c, functionwriteResultToAvro— inver-3.4.1.6, line 3636 takes the initialSELECT COUNT(*)(queryCount) and line 3771 is thefetchedCount != queryCountguard.