util: fix incorrect statement summary Others rows

[nolouch]

What problem does this PR solve?

Issue Number: close #69913

Problem Summary:

The v1 current statement-summary reader can expose the latest Others row
even when it belongs to a previous interval. Separately, reapplying
tidb_stmt_summary_internal_query=OFF scans the LRU with Get, which reverses
the LRU order and causes hot statements to be evicted into Others.

What changed and how does it work?

• Filter the current Others row with the same current-interval boundary used
  for regular statement-summary rows.
• Add a non-mutating SimpleLRUCache.Peek operation and use it in both v1 and
  persistent v2 while removing internal statement summaries, preserving the
  LRU order.
• Initialize the v1 internal-only flag from the first execution, then keep the
  existing AND semantics so mixed internal/external digests survive cleanup.
• Add regressions for a stale previous-window Others row, a capacity-20 LRU
  with two hot statements, actual internal-row cleanup, mixed digest retention,
  and the equivalent persistent-v2 eviction behavior.
• Add a cross-layer Domain regression proving that LoadSysVarCacheLoop
  reapplies the unchanged OFF value through the statement-summary callback.
• Refresh generated Bazel shard counts after rebasing; this also picks up
  TestIssue68550, which was added on master without its generated shard
  update.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No need to test

  • I checked and no code files have been changed.

Commands:

make bazel_prepare
make lint
go test -run '^TestGet$' -count=1 -tags=intest,deadlock ./pkg/util/kvcache
./tools/check/failpoint-go-test.sh pkg/util/stmtsummary -count=1
(cd pkg/util/stmtsummary/v2 && env GOCACHE=/tmp/tidb-go-build-cache go test -count=1 -tags=intest,deadlock)
./tools/check/failpoint-go-test.sh pkg/domain -run '^TestLoadSysVarCacheLoopReappliesStmtSummaryInternalQuery$' -count=1

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

Fix an issue where statement summary could return an `Others` row from an earlier window or incorrectly evict hot statements after system variable reloads.

Summary by CodeRabbit

• New Features

  • Added a non-disruptive cache peek option to read a cached value without changing its LRU position.

• Bug Fixes

  • Fixed statement-summary current rows to exclude evicted “other” entries from previous intervals.
  • Preserved LRU/digest ordering when internal query tracking is disabled.
  • Improved statement-summary internal cleanup to safely handle missing entries.

• Documentation

  • Refined TiDB test reference descriptions to reflect cache and statement-summary behaviors more accurately.

• Tests

  • Added regression coverage for interval handling and periodic sysvar cache behavior; adjusted Bazel test sharding.

[coderabbitai]

📝 Walkthrough

Walkthrough

Adds a non-mutating SimpleLRUCache.Peek method and uses it to preserve LRU ordering during statement-summary cleanup. Current statement-summary rows now exclude evictions from earlier intervals, with regression tests, domain sysvar-reload coverage, and updated Bazel sharding.

Changes

Statement summary cache correctness

Layer / File(s)	Summary
Non-mutating LRU lookup pkg/util/kvcache/simple_lru.go, pkg/util/kvcache/simple_lru_test.go, .agents/skills/tidb-test-guidelines/references/util-case-map.md	Adds Peek for value lookup without LRU movement and validates its return value and ordering behavior.
Statement summary eviction and cleanup pkg/util/stmtsummary/statement_summary.go, pkg/util/stmtsummary/reader.go, pkg/util/stmtsummary/statement_summary_test.go, pkg/util/stmtsummary/v2/stmtsummary.go, pkg/util/stmtsummary/v2/stmtsummary_test.go, pkg/util/stmtsummary/BUILD.bazel, .agents/skills/tidb-test-guidelines/references/util-case-map.md	Corrects internal-query initialization, preserves LRU order during cleanup, filters stale evicted rows by interval, adds regression tests, and increases test sharding from 28 to 30.
Domain sysvar reload regression pkg/domain/domain_test.go, pkg/domain/BUILD.bazel, .agents/skills/tidb-test-guidelines/references/domain-case-map.md	Adds a restricted SQL test context and verifies periodic sysvar-cache loading reapplies statement-summary internal-query settings.
Test execution metadata pkg/bindinfo/tests/BUILD.bazel	Increases bindinfo test sharding from 25 to 26.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Suggested labels: component/statistics

Suggested reviewers: d3hunter, yudongusa, wjhuang2016

Poem

A rabbit peeked without a nudge,
While hot hops kept their rightful grudge.
Old “Others” rows now fade from sight,
Fresh interval counts sit right.
Tests thump softly—hop, hop, hooray!

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The changes address the linked issue's stale Others row, LRU-order preservation, internal-only cleanup, mixed-digest retention, and sysvar reload behavior.
Out of Scope Changes check	✅ Passed	No clear out-of-scope code changes appear; the BUILD and test-map edits support the stated fixes and test updates.
Title check	✅ Passed	The title is concise, specific, and matches the main change to statement-summary Others row handling.
Description check	✅ Passed	The description follows the required template and includes the issue link, problem summary, changes, test checklist, side effects, and release note.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign djshow832, hawkingrei for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS
• pkg/bindinfo/OWNERS
• pkg/domain/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[codecov]

Codecov Report

❌ Patch coverage is 64.70588% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.5450%. Comparing base (66be7a6) to head (46dc2b3).
⚠️ Report is 7 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #69914        +/-   ##
================================================
- Coverage   76.3193%   75.5450%   -0.7743%     
================================================
  Files          2041       2086        +45     
  Lines        559929     583081     +23152     
================================================
+ Hits         427334     440489     +13155     
- Misses       131694     140445      +8751     
- Partials        901       2147      +1246     

Flag	Coverage Δ
integration	45.6451% <64.7058%> (+5.9398%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	60.4471% <ø> (ø)
parser	∅ <ø> (∅)
br	63.9869% <ø> (+1.2656%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[XuHuaiyu]

The persistent v2 implementation appears to have the same ordering issue in StmtSummary.ClearInternal: it iterates s.window.lru.Keys() and calls Get() for every key, which reverses the remaining entries just like the v1 path. When persistence is enabled, the same tidb_stmt_summary_internal_query=OFF callback dispatches here on each successful sysvar-cache rebuild. Once the current v2 window is full, subsequent new keys can therefore evict recently used records into Others.

Was v2 intentionally left out of this PR? If so, could we create and link a follow-up issue? Otherwise, could we use the new non-mutating lookup here as well and add an equivalent v2 regression?

[XuHuaiyu]

Could we also cover clearing an actual internal entry? All entries in the new regression currently have IsInternal == false. On a cache miss, AddStatement creates a zero-valued stmtSummaryByDigest and then evaluates summary.isInternal = summary.isInternal && sei.IsInternal; therefore the flag can never become true.

I reproduced this at the PR head: after enabling internal summaries, adding an IsInternal=true statement, and disabling them again, summaryMap.Size() is still 1 instead of 0. v2 initializes IsInternal from the first record and ANDs subsequent records. Could v1 do the equivalent and add a mixed internal/non-internal regression that verifies both deletion and survivor LRU order? If this pre-existing bug is out of scope, could we track it in a separate issue?

[nolouch]

@XuHuaiyu Addressed both review points in 46dc2b36c2:

• For the persistent-v2 comment, StmtSummary.ClearInternal now uses the non-mutating Peek lookup. The v2 regression verifies that a pure-internal entry is removed, a mixed internal/external digest remains, survivor LRU order is unchanged, and the subsequent inserts evict the original cold records rather than the hot records.
• For the v1 internal-flag comment, a new digest now initializes isInternal from its first execution. Existing digests retain the AND update, so only digests observed exclusively as internal are removed. The v1 regression covers both pure-internal deletion and mixed-digest retention together with the existing LRU/Others assertions.

The regressions failed before the fixes exactly as expected: v1 retained pure_internal_digest, while v2 reversed the survivor order. They pass after the fixes, along with the complete pkg/util/stmtsummary and pkg/util/stmtsummary/v2 package tests and make lint.

[nolouch]

/retest unit-test

[ti-chi-bot]

@nolouch: The /retest command does not accept any targets.
The following commands are available to trigger required jobs:

/test build

/test check-dev

/test check-dev2

/test mysql-test

/test pull-br-integration-test

/test pull-build-next-gen

/test pull-integration-e2e-test

/test pull-integration-realcluster-test-next-gen

/test pull-lightning-integration-test

/test pull-mysql-client-test

/test pull-mysql-client-test-next-gen

/test pull-unit-test-ddlv1

/test pull-unit-test-next-gen

/test unit-test

The following commands are available to trigger optional jobs:

/test canary-long-unit-test

/test canary-unit-test

/test pull-br-integration-test-next-gen

/test pull-check-deps

/test pull-common-test

/test pull-e2e-test

/test pull-error-log-review

/test pull-integration-common-test

/test pull-integration-copr-test

/test pull-integration-ddl-test

/test pull-integration-ddl-test-next-gen

/test pull-integration-e2e-test-next-gen

/test pull-integration-jdbc-test

/test pull-integration-mysql-test

/test pull-integration-nodejs-test

/test pull-integration-python-orm-test

/test pull-mysql-test-next-gen

/test pull-sqllogic-test

/test pull-tiflash-integration-test

Use /test all to run the following jobs that were automatically triggered:

pingcap/tidb/ghpr_build

pingcap/tidb/ghpr_check

pingcap/tidb/ghpr_check2

pingcap/tidb/ghpr_mysql_test

pingcap/tidb/ghpr_unit_test

pingcap/tidb/pull_br_integration_test

pingcap/tidb/pull_build_next_gen

pingcap/tidb/pull_integration_e2e_test

pingcap/tidb/pull_integration_realcluster_test_next_gen

pingcap/tidb/pull_lightning_integration_test

pingcap/tidb/pull_mysql_client_test

pingcap/tidb/pull_mysql_client_test_next_gen

pingcap/tidb/pull_unit_test_next_gen

pull-check-deps

pull-error-log-review

Details

In response to this:

/retest unit-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[nolouch]

/test unit-test