Skip to content

[taosdump] aborts DB dump with rc 255 on tables written during the dump #35415

Description

@rpyxao

Bug Description

taosdump 3.4.1.6 fails an entire database dump with return code 255 when a table receives writes while it is being dumped:

ERROR: writeResultToAvro() LN3774, actual dump out rows not equal to query count. actual dump out: 5000114, count 5000097 dbName=test tbName=d0
ERROR: super table (meters) dump failed, 1 child table(s) have errors. close taos connections...
ERROR: Database test dumped failed

For each child table, writeResultToAvro() takes SELECT COUNT(*) (queryCount) at the start, then scans and counts the rows (fetchedCount). If fetchedCount != queryCount it is fatal (return -1), which fails the child table -> the super table -> the whole database. Concurrent inserts during the dump make fetchedCount > queryCount (more rows, not truncation), so a single actively-written table aborts the whole dump.

Regression introduced by PR #35096 (commit 807472524a); present in 3.4.1.6, and a 3.3.6.9 build completes the identical workload with return code 0.

To Reproduce

Minimal local reproduction — a table written to while it is dumped. Needs a high-rate writer; a slow one-insert-at-a-time writer often won't trip it.

docker run -d --name td --hostname td tdengine/tsdb:3.4.1.6
# wait until the dnode is ready, then:
docker exec td taosBenchmark -y -t 2 -n 5000000                                   # db test, stable meters, tables d0/d1
docker exec -d td sh -c 'yes "INSERT INTO test.d0 VALUES (now,1,220,0.5);" | taos' # concurrent writer
docker exec td sh -c 'mkdir -p /tmp/dump && taosdump -D test -o /tmp/dump'; echo "rc=$?"

-> rc=255 with the error above. Also reproduces with out-of-order/backfill writes (now-400d), and -E <past> does not prevent it when backdated writes land inside the window.

Expected Behavior

A count difference caused by concurrent ingestion is expected for a live time-series database, not corruption. taosdump should not fail the whole database over it.

Screenshots

Not attaching log screenshots to avoid exposing sensitive information; the relevant log is included as text above.

Environment:

  • OS: Linux (Kubernetes pods, Ubuntu-based)
  • Memory: ~31,3 GB
  • Disk: ~496 GB
  • CPU: 16
  • TDengine Server Version: 3.4.1.6
  • taosdump Version: 3.4.1.6
  • Architecture: Linux-x64
  • Deployment: Kubernetes triggered by Airflow DAG
  • Observed in production on a database with ~9,600 subtables and ~3.4B rows.

Additional Context

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions