PXB-3144 [DOCS] - update How XtraBackup works 8.4

docs/differences-logical-backup.md

@@ -42,7 +42,7 @@ Use physical backups with XtraBackup when you need fast, non-blocking backups an
 
 !!! admonition "See also"
 
-    [How Percona XtraBackup works](how-xtrabackup-works.md)
+    [How Percona XtraBackup works: what happens underneath](how-xtrabackup-works-explanation.md)
 
     [Backup process overview](backup-overview.md)
 

docs/features.md

@@ -22,4 +22,4 @@ Percona XtraBackup automatically uses backup locks, a lightweight alternative to
 
 !!! admonition "See also"
 
-    [How Percona XtraBackup works](how-xtrabackup-works.md)
+    [How Percona XtraBackup works: what happens underneath](how-xtrabackup-works-explanation.md)

docs/how-xtrabackup-works-explanation.md

@@ -0,0 +1,163 @@
+# Percona XtraBackup internals
+
+Percona XtraBackup (PXB) runs three sequential phases: backup, prepare, and restore. This page describes the internal work each phase performs.
+
+For commands and flag selection, see [How-to: backup lifecycle operations](how-xtrabackup-works-how-to.md). For compact tables of privileges and defaults, see [Reference: backup lifecycle](how-xtrabackup-works-reference.md).
+
+## Three phases at a glance
+
+The following table maps each operator phase to the InnoDB-internal work the phase performs:
+
+| Phase | Internal work |
+|-------|---------------|
+| Backup (hot copy) | Copy data files and capture redo for the full backup window |
+| Prepare | Replay captured redo, then roll back transactions open at backup end |
+| Restore | Place the prepared files in the destination `datadir` |
+
+The prepare phase runs the same pipeline as InnoDB crash recovery, but offline on the backup tree.
+
+## Backup phase
+
+The backup phase produces a directory that holds everything `mysqld` needs to start a fresh server. The output includes data files for every storage engine, a captured slice of the InnoDB redo log, and a small set of metadata files.
+
+PXB queries `performance_schema.log_status` to stamp a start log sequence number (LSN), then runs two parallel streams of work:
+
+* A data-file stream copies on-disk files for every storage engine in use. InnoDB tablespaces (`.ibd` files), the shared tablespace (`ibdata1`), and undo tablespaces make up most of the volume. PXB also copies files for MyISAM, CSV, MyRocks, and other recognized engines.
+
+* A background redo-log thread tails the InnoDB redo log from the start LSN through the end of the backup window. The thread writes the capture into `xtrabackup_logfile` inside the backup directory.
+
+PXB picks the backup end LSN during the final lock posture. While `LOCK BINLOG FOR BACKUP` is held, PXB reads `performance_schema.log_status`. The query records the current LSN, binary log file and position, and replica coordinates. The redo thread keeps draining records past that sync-point LSN, then stops. PXB records both LSNs in `xtrabackup_checkpoints`. `to_lsn` is the sync-point LSN. `last_lsn` is the highest LSN the redo thread actually copied. For the LSN fields recorded in incremental backups, see [Incremental backups](create-incremental-backup.md).
+
+InnoDB incremental backups can skip the full data-file scan with page tracking. The `mysqlbackup` server component records the LSN of every page write. The `--page-tracking` flag lets PXB copy only the pages changed since the previous backup. For setup and limitations, see [Take an incremental backup using page tracking](page-tracking.md).
+
+MyRocks copies are always wholesale. The MyRocks storage layout does not expose per-page change tracking. PXB copies every MyRocks file on each backup, including incremental backups. For the full storage-engine support list, see [About Percona XtraBackup](about-xtrabackup.md).
+
+The dual capture lets the prepare phase advance the copy to a single LSN. PXB also writes metadata files that prepare and restore consume, including `xtrabackup_checkpoints`, `xtrabackup_info`, `xtrabackup_binlog_info`, and `backup-my.cnf`. For the full list of artifacts, see [Index of files created by Percona XtraBackup](xtrabackup-files.md).
+
+PXB stages locking across the backup so InnoDB data manipulation language (DML) keeps running through the bulk file copy. Two flags shape the trade-offs:
+
+| Flag | Default | Effect |
+|------|---------|--------|
+| `--lock-ddl` | `ON` | Controls when the backup lock takes effect |
+| `--register-redo-log-consumer` | `OFF` | Holds redo files on the server until PXB finishes reading them |
+
+### Redo capture and why the copy is not yet consistent
+
+InnoDB keeps writing while PXB reads files, so the copied pages reflect mixed points in time. Pages copied early carry older LSNs than pages copied later.
+
+To bridge that gap, PXB stamps a start LSN and copies InnoDB data files. A background thread streams the InnoDB redo log in parallel for the entire backup window.
+
+The redo capture matters for two reasons:
+
+* The `.ibd` page reads span real clock time. Without a continuous redo log, prepare cannot advance the copy to a single LSN.
+
+* InnoDB redo files wrap and recycle. The redo thread must read every record before the server overwrites the underlying file.
+
+Without redo capture, the directory holds physically mixed page images. The copy then has no path to run InnoDB recovery against a single LSN. The dataset therefore becomes consistent during prepare, not during the file copy.
+
+### Three lock postures
+
+PXB stages locking across the backup so InnoDB DML keeps running through the bulk copy.
+
+The following table lists the three sub-phases in order:
+
+| # | Lock posture | Internal work |
+|---|--------------|---------------|
+| 1 | No global lock | Copy InnoDB data files and stream redo while DML continues |
+| 2 | Backup lock | Copy non-InnoDB files such as `.frm`, `.MRG`, `.MYD`, `.MYI`, `.CSM`, `.CSV`, `.sdi`, and `.par`. Data definition language (DDL) is restricted; DML continues, with details that depend on the server build. |
+| 3 | `LOCK BINLOG FOR BACKUP` (short) | Pin binary log or replica coordinates, read `performance_schema.log_status`, and release locks |
+
+For background on the underlying lock primitives, see [Reduction in locks](reduction-in-locks.md) and the upstream MySQL documentation for [`LOCK INSTANCE FOR BACKUP`](https://dev.mysql.com/doc/refman/{{vers}}/en/lock-instance-for-backup.html).
+
+### `--lock-ddl=ON` compared with `--lock-ddl=REDUCED`
+
+The `--lock-ddl` flag controls when PXB takes the backup lock that gates DDL.
+
+The following table compares the two modes:
+
+| Value | Backup lock taken | DDL impact |
+|-------|-------------------|------------|
+| `ON` (default) | Backup start | DDL is restricted for the full backup window |
+| `REDUCED` | After the InnoDB copy finishes | DDL is restricted for a shorter window. DDL runs concurrently with the InnoDB copy, so PXB tracks tablespace changes from redo and recopies affected tablespaces when the lock is taken. |
+
+InnoDB DML continues through the main copy under both modes.
+
+### `--register-redo-log-consumer`: keep redo until PXB finishes
+
+On a high-traffic host, the server can recycle redo files before PXB finishes reading them. A purged redo file makes the backup unusable.
+
+The `--register-redo-log-consumer` flag registers PXB as a redo consumer. The server then retains a redo file until PXB has copied that file.
+
+Review the trade-offs before enabling the flag:
+
+* Redo files stay on disk longer, which raises peak redo size during the run.
+
+* Free space can drop sharply on a long backup of a write-heavy server.
+
+* The server can briefly stall writes when the consumer advances.
+
+!!! warning
+
+    Monitor free disk during the backup. Abort with Ctrl+C or `SIGTERM` to `xtrabackup` if disk space reaches a critical threshold. The consumer releases on abort, and the server can then purge redo.
+
+The default value is `OFF`. Enable the flag only when spare disk space is available.
+
+### When PXB can skip backup locks
+
+PXB skips backup locks only when every table in every schema, including the `mysql` system schema, uses InnoDB. Most installations still hold CSV or MyISAM tables in `mysql`, such as `general_log`, so PXB takes backup locks in practice.
+
+Replication coordinate handling depends on the server build:
+
+* Percona Server for MySQL can fold relay coordinates into `log_status`, which reduces the locks needed for `--slave-info`.
+
+* Stock MySQL can still require `FLUSH TABLES WITH READ LOCK` for some relay-position scenarios.
+
+## Prepare phase: roll forward, then roll back
+
+`xtrabackup --prepare` runs two coordinated passes against the backup directory.
+
+Roll forward applies the captured redo to the copied tablespaces. The pass is physical: each redo record describes a page-level change. InnoDB reapplies that change to the page image on disk.
+
+Roll back removes transactions that had not committed at the backup end LSN. Redo replay can have written those changes to pages, so prepare must reverse the changes at the logical level.
+
+Roll back relies on two structures inside the tablespaces:
+
+* Undo records describe how to reverse the row-level effects of each open transaction.
+
+* Serialized Dictionary Information (SDI) carries table and index definitions inside each tablespace. SDI supplies the schema context that older `.frm` files once provided to rollback.
+
+Non-InnoDB tables match the same point in time because PXB copied them under backup locks during the backup phase.
+
+After prepare finishes, the dataset matches the backup end LSN with open transactions stripped.
+
+## Restore phase: file placement
+
+The restore phase reads destination paths from the server configuration, including `datadir`, `innodb_data_home_dir`, `innodb_data_file_path`, and `innodb_log_group_home_dir`.
+
+PXB places files in a fixed order:
+
+* MyISAM-family files first, including `.MRG`, `.MYD`, `.MYI`, `.CSM`, `.CSV`, `.sdi`, and `.par`.
+
+* InnoDB tablespaces and indexes second.
+
+* Log files last.
+
+File attributes survive the copy. A successful backup writes binary log coordinates to standard error (STDERR). Redirect STDERR to a file when the workflow needs to capture those coordinates.
+
+## Cloud and streaming: same mechanics, longer windows
+
+Streaming and `xbcloud` workflows run the same internal phases as a local backup. Slow networks extend the PXB final-sync phase, which holds the short backup locks, including the binary log lock, for longer.
+
+For workflow procedures, see the following references:
+
+* [Take a streaming backup](take-streaming-backup.md)
+
+* [xbcloud binary overview](xbcloud-binary-overview.md)
+
+## See also
+
+* [How-to: backup lifecycle operations](how-xtrabackup-works-how-to.md)
+
+* [Reference: backup lifecycle](how-xtrabackup-works-reference.md)
+
+* [Index of files created by Percona XtraBackup](xtrabackup-files.md)

docs/how-xtrabackup-works-how-to.md

@@ -0,0 +1,136 @@
+# Backup lifecycle
+
+A Percona XtraBackup (PXB) workflow runs three sequential phases: backup, prepare, and restore. This how-to guide describes each phase and the operational flags that shape locking, redo retention, and recovery output.
+
+Apply the following principles across every lifecycle phase:
+
+* Grant only the privileges that the workflow requires.
+
+* Run backup, prepare, and restore in sequence.
+
+* Capture binary log coordinates for recovery and replication.
+
+* Choose operational flags, such as `--lock-ddl` and `--register-redo-log-consumer`, to match the workload and lock tolerance.
+
+## Grant the minimum privileges
+
+PXB needs database privileges that match the locking operations and metadata reads in the workflow.
+
+Complete the following privilege steps:
+
+1. Grant `BACKUP_ADMIN` so PXB can read `performance_schema.log_status`.
+
+2. Confirm `BACKUP_ADMIN` allows PXB to take `LOCK INSTANCE FOR BACKUP`, `LOCK TABLES FOR BACKUP`, and `LOCK BINLOG FOR BACKUP`.
+
+3. Add `RELOAD`, `LOCK TABLES`, or `REPLICATION CLIENT` for workflows that need extra capabilities. Examples include `FLUSH TABLES WITH READ LOCK` and the `--slave-info` option.
+
+For privilege lists and worked examples, see [Connection and privileges needed](privileges.md).
+
+## Run a backup and capture binary log coordinates
+
+Capture binary log coordinates during the backup phase to support recovery and replica bootstrap.
+
+Complete the following backup steps:
+
+1. Run `xtrabackup` against the target directory with the standard options.
+
+2. Redirect STDERR to a log file because PXB writes the binary log coordinates to STDERR. For example, `xtrabackup ... 2> backupout.log`.
+
+3. Confirm the command returns exit code `0`.
+
+For the artifact list a backup produces, see [Index of files created by Percona XtraBackup](xtrabackup-files.md).
+
+## Choose a `--lock-ddl` mode
+
+PXB uses backup locks to copy non-InnoDB files without blocking InnoDB DML, when the server supports backup locks.
+
+The following table compares the two `--lock-ddl` modes:
+
+| Option | Effect |
+|--------|--------|
+| `--lock-ddl=ON` (default) | Takes the backup lock at start. DDL stays blocked for the full run, unless the workflow changes. |
+| `--lock-ddl=REDUCED` | Takes the lock after InnoDB is copied. DDL blocks for a shorter window. This mode accepts DDL contention during the InnoDB phase in exchange for a shorter DDL block. |
+
+!!! warning
+
+    Both modes block schema changes (DDL) for part of the backup window. Schedule DDL outside the backup, or accept the contention.
+
+For a compact comparison, see [Reference: backup lifecycle](how-xtrabackup-works-reference.md).
+
+## When to enable `--register-redo-log-consumer`
+
+Enable `--register-redo-log-consumer` when heavy write throughput risks purging redo before PXB finishes reading the redo stream. The flag prevents failures caused by missing redo.
+
+Review the following checks before enabling the flag:
+
+* Plan for redo bloat and possible disk exhaustion, because the server retains redo longer.
+
+* Monitor free space and I/O throughout the run.
+
+* Abort the backup with Ctrl+C or `SIGTERM` to the `xtrabackup` process if disk space reaches a critical threshold. The consumer then releases, the server purges redo, and the operator can free space and retry.
+
+The default value is `OFF`. Enable the flag only when spare disk space is available.
+
+## Prepare the backup with `--prepare`
+
+The prepare phase applies redo and rolls back uncommitted transactions to produce a consistent snapshot.
+
+Complete the following prepare steps:
+
+1. Run `xtrabackup --prepare` with `--target-dir` set to the backup directory.
+
+2. For large or incremental backups, tune prepare performance with the following options:
+
+    * Raise `--use-memory` to between 1 GB and 2 GB when RAM allows, because the default value is small.
+
+    * Pass `--parallel` to apply `.delta` files in parallel on incremental backups, starting in 8.4.0-3. The flag does not parallelize first-pass redo on a full backup.
+
+The following command shows a tuned prepare run:
+
+```bash
+xtrabackup --prepare --use-memory=2G --parallel=4 --target-dir=/data/backups/
+```
+
+!!! warning
+
+    Leave RAM headroom for the OS and `mysqld` when prepare runs on the same host as production. Insufficient headroom triggers OOM kills.
+
+For flag references, see [`--use-memory`](xtrabackup-option-reference.md#use-memory) and [`--parallel`](xtrabackup-option-reference.md#parallel).
+
+## Restore with `--copy-back` or `--move-back`
+
+The restore phase places prepared backup files into the destination `datadir` for `mysqld` startup.
+
+Complete the following restore steps:
+
+1. Stop `mysqld` if the process owns the destination `datadir`.
+
+2. Run `xtrabackup --copy-back` or `xtrabackup --move-back` with `--target-dir` set to the prepared backup. PXB reads destination paths from the configuration, including `datadir`, InnoDB paths, log paths, and related settings.
+
+3. Set ownership and permissions before starting `mysqld`. For example, run `chown -R mysql:mysql /var/lib/mysql`. Backup files belong to the user that ran `xtrabackup`, while the server expects the `mysql` user.
+
+4. Start `mysqld`.
+
+!!! warning
+
+    `--move-back` destroys the original backup directory because the command moves files instead of copying them. Use `--move-back` only when disk space is constrained, and confirm a separate backup copy is available.
+
+For full restore flows covering full, incremental, and compressed backups, see [Restore full, incremental, and compressed backups](restore-a-backup.md).
+
+## Stream backups or upload to object storage
+
+Streaming backups and cloud uploads follow the same lifecycle as a local backup. Slow networks extend the PXB final-sync phase and hold the short backup locks, including the binary log lock, for longer.
+
+Plan streaming and upload runs with the following guidance:
+
+* Run streaming backups inside a maintenance window.
+
+* Size network bandwidth and destination throughput to keep the final-sync phase short.
+
+* Confirm the destination object store accepts the backup payload size.
+
+For workflow details, see the following references:
+
+* [Take a streaming backup](take-streaming-backup.md)
+
+* [xbcloud binary overview](xbcloud-binary-overview.md)