DBMS_STATS.GATHER_SCHEMA_STATS triggers 2013 connection loss and repeated observer log submission/freeze/compaction errors: -6268/-4023/-4012

[cms-cms]

Environment

• Version:

OceanBase seekdb
MySQL protocol version: 5.7.25-OceanBase seekdb-v
REVISION: 1-928324ce1523e964142a8357fb968e432c9511f4
BUILD_BRANCH: HEAD
BUILD_TIME: Aug 7 2026 17:04:55
BUILD_FLAGS: RelWithDebInfo

• Is this an RD-provided temporary version?: No
• Reproducibility: Not specified
• Test changes: Not specified
• Other environmental anomalies or changes: Not specified

Description

During a 4-hour regression test, three independent observer instances were started on the same machine using the same binary. The ddltest_duomo_lite instance failed when executing CALL DBMS_STATS.GATHER_SCHEMA_STATS. The client reported error 2013 Lost connection to MySQL server during query, which was flagged as fatal in strict mode. Concurrently, the observer logs showed repeated errors related to log submission, freeze, compaction, and tablet GC, including -6268 OB_TX_NOLOGCB, -4023 OB_EAGAIN, and -4012 OB_TIMEOUT.

The other two instances (multi_modal_lite and stress_duomo_lite) passed their tests but also exhibited warnings like submit log callback use too mush time and -4023 errors. This suggests the issue is not specific to a single script but indicates a degradation in the observer's log submission/flush/checkpoint/freeze pipeline under load.

Steps to Reproduce / Scenario

1. On the machine, three independent observer instances were started for a 4-hour regression test:
   • ddltest_duomo_lite -> :2881
   • multi_modal_lite -> :3881
   • stress_duomo_lite -> :4881
2. The test was executed using the following commands:

   cd /obdata/data/three_lite_regression
   bash run_all.sh prepare 2>&1 | tee prepare.log
   bash run_all.sh build   2>&1 | tee build.log
   bash run_all.sh start   2>&1 | tee start.log
   bash run_all.sh test    2>&1 | tee test.log

3. Each task had a runtime of time 14400.
4. The failure occurred in ddltest_duomo_lite when executing the specific SQL:

   CALL DBMS_STATS.GATHER_SCHEMA_STATS('test', degree=>64, no_invalidate=>FALSE, cascade=>TRUE);

Environment Information

• Database connection strings:

  mysql -h  -P2881 -uroot
  mysql -h  -P3881 -uroot
  mysql -h  -P4881 -uroot
  

• Instance directories:

  /obdata/data/ddltest_duomo_lite_db
  /obdata/data/multi_modal_lite_db
  /obdata/data/stress_duomo_lite_db
  

• SSH access: ssh hudson@ oceanbaseV5
• Is the scene preserved?: Yes

Log Information & Evidence

Workload-side Failure Evidence (ddltest_duomo_lite)

Time: 2026-08-07 20:01:48.792
Connection: :2881
Triggering SQL:

CALL DBMS_STATS.GATHER_SCHEMA_STATS('test', degree=>64,no_invalidate=>FALSE,cascade=>TRUE);

Client Error:

(2013, 'Lost connection to MySQL server during query (timed out)')

Strict Mode Judgment:
[DDLTEST_STRICT][fatal] RESULT ddltest_duomo_lite FAIL

Log Paths:

• /obdata/data/three_lite_regression/logs/ddltest_duomo_lite/workload-console.log
• /obdata/data/three_lite_regression/test.log
• /obdata/data/three_lite_regression/start.log
• /obdata/data/ddltest_duomo_lite_db/log/seekdb.log*

Preserved Scene: /obdata/data/three_lite_regression/evidence/ddltest_duomo_lite-20260807-200148/ containing:

• workload-console.log
• test.log
• start.log
• seekdb-at-failure.log

Observer-side Evidence

ddltest_duomo_lite (:2881)

Logs: /obdata/data/ddltest_duomo_lite_db/log/seekdb.log*

Key error logs captured:

[2026-08-10 02:05:35.139880] EDIAG [STORAGE.TRANS] ~ObMvccWriteGuard
(ob_mvcc_ctx.cpp:388) errcode=-6268
failed to submit log if neccesary(ret=-6268)

[2026-08-10 02:00:58.551477] EDIAG [STORAGE.COMPACTION] schedule_freeze_dag
(ob_basic_schedule_tablet_func.cpp:93) errcode=-4023
failed to schedule batch force freeze tablets dag

[2026-08-10 00:03:16.503924] EDIAG [STORAGE.COMPACTION] try_schedule_compaction_after_mini
(ob_tablet_merge_ctx.cpp:227) errcode=-4677
failed to schedule special tablet minor merge(tablet_id={id:49402})

[2026-08-09 20:02:10.613014] EDIAG [STORAGE] runTimerTask
(ob_tablet_gc_service.cpp:191) errcode=-4012
fail to wait unpersist tablet ids flushed(ret=-4012, ret="OB_TIMEOUT")

Subsequent similar freeze failures:

[2026-08-10 10:03:21.273654] WDIAG [STORAGE] freeze
(ob_tx_data_memtable_mgr.cpp:226) errcode=-4023
freeze tx data memtable fail.(ret=-4023, ret="OB_EAGAIN")
tablet_id:{id:49402}

multi_modal_lite (:3881)

Logs: /obdata/data/multi_modal_lite_db/log/seekdb.log*

Key log:

[2026-08-09 14:16:10.827108] EDIAG [STORAGE.TRANS] submit_log_
(ob_id_service.cpp:98) errcode=-4023
submit log callback use too mush time

Corresponding PALF logs show high log flush/callback latency:

WDIAG [PALF] sliding_cb ... errcode=-4389
log_task life cost too much time
submit_to_flush cost=348069
log_life_time=348130

Followed by a callback success:

INFO [STORAGE.TRANS] handle_submit_callback
success=true, used_ts=348154

stress_duomo_lite (:4881)

Logs: /obdata/data/stress_duomo_lite_db/log/seekdb.log*

Key logs:

[2026-08-10 02:00:12.022705] EDIAG [STORAGE.COMPACTION] schedule_freeze_dag
(ob_basic_schedule_tablet_func.cpp:93) errcode=-4023
failed to schedule batch force freeze tablets dag

[2026-08-09 14:28:42.871343] EDIAG [STORAGE.TRANS] submit_log_
(ob_id_service.cpp:98) errcode=-4023
submit log callback use too mush time

Error Code Interpretation

From source code definitions:

-6268 = OB_TX_NOLOGCB     = No log callbacks available
-4023 = OB_EAGAIN         = Try again
-4012 = OB_TIMEOUT        = Timeout
-4677 = OB_NO_NEED_MERGE  = no need to merge
-4389 = OB_ERR_TOO_MUCH_TIME

While -4677 OB_NO_NEED_MERGE alone might be a scheduling race or harmless warning, its occurrence alongside -6268/-4023/-4012/-4389 and the actual client disconnection in the DDL workload suggests this is not merely noise.

Other Notes

This issue was identified during internal testing. The workload failure (ddltest_duomo_lite) and the widespread warnings in the other instances point to a systemic issue with the log submission and freeze mechanisms under concurrent load.

[hnwyllmm]

Dima personal assistant added

1. 2013 Error Analysis

Two GATHER_SCHEMA_STATS calls were initiated at 20:00:18.674 and 20:00:18.800, respectively, and finally reported an error at 20:01:48.792/796, taking about 90 seconds.

(2013, 'Lost connection to MySQL server during query (timed out)')

The ddltest connection code sets the PyMySQL read_timeout to 45 seconds; the current environment is PyMySQL 1.0.2, DBUtils 3.0.3. DBUtils will reconnect and retry once when execution fails outside a transaction, so the observed time in this case was:

45-second read timeout + waiting another ~45 seconds after reconnection and retry ≈ 2013 error after ~90 seconds

At the same time, other DDL/DML connections were still executing normally, and no phenomenon was observed where the seekdb process or all connections were disconnected simultaneously. In strict test mode, after receiving the first fatal error, os._exit(86) is called, and the workload exits immediately.

Two identical calls were running concurrently at the time:

CALL DBMS_STATS.GATHER_SCHEMA_STATS(
'test', degree=>64, no_invalidate=>FALSE, cascade=>TRUE
);

The database already contained about 301 ddlkitty_% tables, 4778 columns, and 3499 index statistics records, along with a large amount of concurrent DDL/DML. Whether the statistics collection exceeding 45 seconds also involves server-side performance issues needs to be re-verified using a more appropriate client timeout. What can be confirmed now is that the client's 45-second read timeout directly triggered this 2013 error; DBUtils' transparent retry for long-running stored procedures may also resubmit the task, amplifying concurrency.

2. Incomplete Log Coverage

The client error time was 20:01:48, but the earliest observer log in seekdb-at-failure.log is around 20:02:13, and the latest is 20:03:28. There are no logs from 20:01 in this file. This file actually records a log segment from about 25 seconds after the failure and cannot be used to judge the server-side execution status during the period 20:00:18 to 20:01:48.

The original seekdb.log* files have been rotated, and the files corresponding to August 7th are currently unrecoverable. Therefore, existing evidence cannot further confirm at which specific stage the statistics collection was stuck on the server side.

Suggestions:

1. Configure an independent read_timeout for GATHER_SCHEMA_STATS that matches long-running operations, instead of reusing the generic 45-second value.
2. Disable or bypass DBUtils' transparent execution retry for this call to avoid resubmitting the statistics task after a timeout.
3. Re-execute the call with a sufficiently long client timeout, while simultaneously saving statistics collection monitoring information, SQL audit/trace IDs, and complete observer logs.
4. When collecting evidence for a failure, preserve all rotated seekdb.log* files covering the window before and after the failure, not just copy the current log segment from after the failure occurred.