Feature/agent memory kickoff

[BornChanger]

What problem does this PR solve?

Issue Number: ref #xxx

Problem Summary:

From the user spec and design package, the core problem is not just missing code paths—it is missing product-level memory guarantees for agent applications:

• No unified memory behavior across Claude Code / OpenCode / OpenClaw.
• No strict, end-to-end tenant fail-closed contract across SQL + adapter paths.
• No deterministic and token-bounded retrieval->assembly path for stable prompt quality.
• No auditable, explainable, and rollout-safe production hardening baseline.

This PR delivers the first end-to-end TiDB Agent Memory baseline to close that gap.

What changed and how does it work?

1) Business value: consistent memory UX across clients

Aligned with the user spec goal (consistent memory experience with TiDB guarantees), this PR establishes a unified memory data plane contract so client integrations can rely on one behavior model for store/search/pack/trace workflows.

2) Business value: safe multi-tenant foundation you can trust

Milestone A outcomes are now executable:

• canonical schema/profile baseline + compatibility views,
• mandatory tenant context with fail-closed behavior,
• baseline audit trail for read/write/delete/policy-denied actions,
• upgrade/rollback compatibility evidence.

This reduces isolation risk and gives operators a governance baseline for shared-agent deployments.

3) Business value: predictable recall quality and cost

Milestone B delivers deterministic hybrid retrieval and token-budgeted context assembly with guardrails, plus lifecycle core controls (pause/resume/cancel, idempotence/recovery). This improves answer consistency under retries and limits latency/cost spikes from unbounded candidate expansion.

4) Business value: production hardening for debugging and compliance

Milestone C delivers observability/explain surfaces, adapter conformance path, and security hardening (redaction/privileged access/purge safety), so teams can diagnose why memory was selected or denied and enforce policy consistently.

5) Business value: rollout confidence and operator readiness

A5/B5/C5 evidence docs and backlog gate reconciliation were added to make release readiness explicit and reproducible (canary/rollback rehearsal scope + command evidence), enabling staged rollout with lower operational risk.

User-facing outcomes (from spec perspective)

• Persistent memory behavior is stable across sessions/retries.
• Team-shared memory remains tenant/namespace isolated by default.
• Retrieval and context packing are explainable (selected/dropped reasons).
• Policy and budget failures are explicit and actionable.
• Security/audit posture is verifiable before broader rollout.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No need to test

  • I checked and no code files have been changed.

Manual validation commands executed:

• make bazel_prepare
• make lint
• go test -run '^(TestBootstrap|TestUpgrade|TestVersionedBootstrapSchemas)$' -tags=intest,deadlock ./pkg/session -count=1
• go test -run '^(TestAgentMemoryTenantContextFailClosed|TestRunAgentMemoryPipelineTenantContextFailClosed|TestAgentMemoryAuditStorageCardinalityUnderLoad)$' -tags=intest,deadlock ./pkg/planner/core -count=1
• go test -run '^TestAgentMemoryContextSysVarsConcurrentIsolation$' -tags=intest,deadlock ./pkg/sessionctx/variable -count=1
• make failpoint-enable && (go test ./pkg/planner/core -run 'TestObserveAgentMemoryPipelineNormalPath|TestObserveAgentMemoryPipelineErrorPathAlwaysSampled|TestObserveAgentMemoryPipelineSuccessSamplingControl|TestObserveAgentMemoryPipelinePayloadCaptureOptIn|TestBuildAgentMemoryExplainStages|TestBuildAgentMemoryExplainStagesErrorPath|TestRenderAgentMemoryExplainRowsStable|TestQueryAgentMemoryTraceRowsStableOrder|TestAssembleAgentMemoryContextBudgetAndManifest|TestRunAgentMemoryPipelineDeterministicOutput|TestRunAgentMemoryPipelineLifecycleStateInfluence|TestAgentMemoryTenantContextFailClosed|TestAgentMemoryRetrievalFallbackWarning|TestRankAgentMemoryCandidatesDeterministic' -tags=intest,deadlock && go test ./pkg/session -run 'TestAgentMemoryAdapterVersionNegotiation|TestAgentMemoryAdapterConformanceEndToEnd|TestAgentMemoryAdapterCompatibilityMatrix|TestApplyAgentMemoryRedactionByRole|TestApplyAgentMemorySecurityPolicyTenantIsolation|TestApplyAgentMemorySecurityPolicyPrivilegedTenantScoped|TestValidateAgentMemoryPrivilegedAccessReason|TestAgentMemoryPurgeWorkflowCheckpointRecovery|TestAgentMemoryLifecycleTaskIdempotenceAcrossRetries|TestAgentMemoryLifecyclePauseResumeCancelConcurrentSafe|TestAgentMemoryLifecycleCheckpointRecovery|TestAgentMemoryLifecycleFilterArchivedDefault' -tags=intest,deadlock && go test ./pkg/sessionctx/variable -run 'TestAgentMemoryContextSysVarsConcurrentIsolation|TestAgentMemoryContextSysVars|TestAgentMemoryHybridRetrievalSysVars' -tags=intest,deadlock); rc=$?; make failpoint-disable; exit $rc
• go test ./br/pkg/restore/snap_client -run TestMonitorTheSystemTableIncremental -tags=intest,deadlock

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

Introduce TiDB Agent Memory kickoff baseline with cross-client memory contract consistency, tenant fail-closed isolation, deterministic retrieval/context assembly, lifecycle controls, observability/explainability, and security hardening evidence for staged production rollout.