statistics: stabilize TestNonLiteInitStatsWithTableIDs case (#67971)#70017
statistics: stabilize TestNonLiteInitStatsWithTableIDs case (#67971)#70017ti-chi-bot wants to merge 1 commit into
Conversation
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
ti-chi-bot
added
component/statistics
do-not-merge/hold
|
@0xPoe This PR has conflicts, I have hold it. |
ti-chi-bot
Bot
added
the
cherry-pick-approved
|
@ti-chi-bot: ## If you want to know how to resolve it, please read the guide in TiDB Dev Guide. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
📝 WalkthroughWalkthroughStatistics cache interfaces now expose asynchronous-update synchronization. LFU operations and tests use it for deterministic visibility, while statistics bootstrap drains updates between loading phases and conditionally publishes temporary cache contents. ChangesStats cache synchronization contract
LFU visibility semantics and tests
Statistics bootstrap synchronization
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant InitStats
participant TemporaryStatsCache
participant LFU
participant GlobalStatsCache
InitStats->>TemporaryStatsCache: Load metadata and statistic chunks
TemporaryStatsCache->>LFU: Put chunk entries
InitStats->>LFU: WaitForAsyncUpdates between phases
InitStats->>GlobalStatsCache: Publish each table
GlobalStatsCache->>LFU: WaitForAsyncUpdates
InitStats->>TemporaryStatsCache: Close temporary cache
Possibly related PRs
Suggested labels: Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 golangci-lint (2.12.2)Error: can't load config: unsupported version of the configuration: "" See https://golangci-lint.run/docs/product/migration-guide for migration instructions Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@pkg/statistics/handle/bootstrap.go`:
- Around line 362-369: Remove all cherry-pick conflict markers and reconcile the
bootstrap flow with the selected release branch. In
pkg/statistics/handle/bootstrap.go:362-369, use the release-compatible
initStatsHistograms4Chunk call while retaining per-chunk WaitForAsyncUpdates
synchronization; at 498-505, adapt the TopN invocation to the branch’s actual
helper signature; at 814-832 and 904-926, update targeted-cache and final-cache
publication plus cleanup to the release branch APIs and behavior.
- Around line 907-909: Update the direct bucket-loading loop in the bootstrap
flow to call cache.WaitForAsyncUpdates() after each chunk is processed, before
the next initStatsBuckets4Chunk invocation can read the cache. Match the
per-chunk wait placement used by initStatsBucketsByPaging, while retaining the
existing final drain and completion log.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 34b4145c-b517-44d3-8dcc-7fb07b6101e6
📒 Files selected for processing (9)
pkg/statistics/handle/bootstrap.gopkg/statistics/handle/cache/internal/inner.gopkg/statistics/handle/cache/internal/lfu/lfu_cache.gopkg/statistics/handle/cache/internal/lfu/lfu_cache_test.gopkg/statistics/handle/cache/internal/mapcache/map_cache.gopkg/statistics/handle/cache/statscache.gopkg/statistics/handle/cache/statscache_test.gopkg/statistics/handle/cache/statscacheinner.gopkg/statistics/handle/types/interfaces.go
| <<<<<<< HEAD | ||
| h.initStatsHistograms4Chunk(is, cache, iter, false) | ||
| ======= | ||
| h.initStatsHistograms4Chunk(is, cache, iter, isFullCache(cache, totalMemory)) | ||
| // The same table may continue in the next chunk. Drain LFU async admission/rejection | ||
| // before the next chunk reads or mutates it again. | ||
| cache.WaitForAsyncUpdates() | ||
| >>>>>>> 933e59fbe58 (statistics: stabilize TestNonLiteInitStatsWithTableIDs case (#67971)) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🔴 Critical | ⚡ Quick win
Resolve the remaining cherry-pick conflicts before merge.
The <<<<<<<, =======, and >>>>>>> markers make this package uncompilable; the incoming alternatives also reference signatures and variables not present in the selected release branch.
pkg/statistics/handle/bootstrap.go#L362-L369: reconcile histogram initialization and retain the intended per-chunk synchronization.pkg/statistics/handle/bootstrap.go#L498-L505: reconcile the TopN call with the release branch’s actual helper signature.pkg/statistics/handle/bootstrap.go#L814-L832: reconcile targeted-cache publication with the release branch API.pkg/statistics/handle/bootstrap.go#L904-L926: reconcile final cache publication and cleanup behavior.
📍 Affects 1 file
pkg/statistics/handle/bootstrap.go#L362-L369(this comment)pkg/statistics/handle/bootstrap.go#L498-L505pkg/statistics/handle/bootstrap.go#L814-L832pkg/statistics/handle/bootstrap.go#L904-L926
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@pkg/statistics/handle/bootstrap.go` around lines 362 - 369, Remove all
cherry-pick conflict markers and reconcile the bootstrap flow with the selected
release branch. In pkg/statistics/handle/bootstrap.go:362-369, use the
release-compatible initStatsHistograms4Chunk call while retaining per-chunk
WaitForAsyncUpdates synchronization; at 498-505, adapt the TopN invocation to
the branch’s actual helper signature; at 814-832 and 904-926, update
targeted-cache and final-cache publication plus cleanup to the release branch
APIs and behavior.
| // CalcPreScalar writes tables back; drain before replacing/publishing the cache. | ||
| cache.WaitForAsyncUpdates() | ||
| statslogutil.StatsLogger().Info("Complete loading the bucket", zap.Duration("duration", time.Since(start))) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Drain the direct bucket-loading loop between chunks.
This final drain occurs after all chunks, so it cannot prevent a following initStatsBuckets4Chunk call from reading a stale snapshot written by the preceding chunk. Add the same per-chunk wait used by initStatsBucketsByPaging after line 703.
Proposed fix
h.initStatsBuckets4Chunk(cache, iter)
+ cache.WaitForAsyncUpdates()
}🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@pkg/statistics/handle/bootstrap.go` around lines 907 - 909, Update the direct
bucket-loading loop in the bootstrap flow to call cache.WaitForAsyncUpdates()
after each chunk is processed, before the next initStatsBuckets4Chunk invocation
can read the cache. Match the per-chunk wait placement used by
initStatsBucketsByPaging, while retaining the existing final drain and
completion log.
|
@ti-chi-bot: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
2 participants
This is an automated cherry-pick of #67971
What problem does this PR solve?
Issue Number: ref #67176
Problem Summary:
The LFU stats cache is backed by Ristretto. New/non-resident entries are admitted asynchronously, but init stats builds a fresh stats cache in phases/chunks and later phases may read entries written by earlier phases/chunks. Without draining the async path at the required boundaries, init stats can observe missing or stale table snapshots.
What changed and how does it work?
WaitForAsyncUpdatesto the stats-cache interfaces and implementations.Wait().CalcPreScalarwrites tables back, before replacing/publishing the cache;Check List
Tests
Side effects
Documentation
Release note
Please refer to Release Notes Language Style Guide to write a quality release note.
Summary by CodeRabbit
Bug Fixes
Tests