From 65b83355e6d62d33c7201c6c4871e08cac743d6f Mon Sep 17 00:00:00 2001 From: Patrick Birch <48594400+patrickbirch@users.noreply.github.com> Date: Fri, 20 Feb 2026 10:56:19 -0600 Subject: [PATCH] PXB-3144 [DOCS] - update How XtraBackup works 8.4 modified: docs/how-xtrabackup-works.md --- docs/differences-logical-backup.md | 2 +- docs/features.md | 2 +- docs/how-xtrabackup-works-explanation.md | 163 +++++++++++++++++++++++ docs/how-xtrabackup-works-how-to.md | 136 +++++++++++++++++++ docs/how-xtrabackup-works-reference.md | 92 +++++++++++++ docs/how-xtrabackup-works.md | 114 ---------------- docs/index-contents.md | 4 +- docs/reduction-in-locks.md | 2 +- docs/server-backup-version-comparison.md | 2 +- mkdocs-base.yml | 4 +- 10 files changed, 401 insertions(+), 120 deletions(-) create mode 100644 docs/how-xtrabackup-works-explanation.md create mode 100644 docs/how-xtrabackup-works-how-to.md create mode 100644 docs/how-xtrabackup-works-reference.md delete mode 100644 docs/how-xtrabackup-works.md diff --git a/docs/differences-logical-backup.md b/docs/differences-logical-backup.md index 4eaf720d3..3bd2afdda 100644 --- a/docs/differences-logical-backup.md +++ b/docs/differences-logical-backup.md @@ -42,7 +42,7 @@ Use physical backups with XtraBackup when you need fast, non-blocking backups an !!! admonition "See also" - [How Percona XtraBackup works](how-xtrabackup-works.md) + [How Percona XtraBackup works: what happens underneath](how-xtrabackup-works-explanation.md) [Backup process overview](backup-overview.md) diff --git a/docs/features.md b/docs/features.md index a41cfadeb..ef5110b48 100644 --- a/docs/features.md +++ b/docs/features.md @@ -22,4 +22,4 @@ Percona XtraBackup automatically uses backup locks, a lightweight alternative to !!! admonition "See also" - [How Percona XtraBackup works](how-xtrabackup-works.md) + [How Percona XtraBackup works: what happens underneath](how-xtrabackup-works-explanation.md) diff --git a/docs/how-xtrabackup-works-explanation.md b/docs/how-xtrabackup-works-explanation.md new file mode 100644 index 000000000..89364ea0f --- /dev/null +++ b/docs/how-xtrabackup-works-explanation.md @@ -0,0 +1,163 @@ +# Percona XtraBackup internals + +Percona XtraBackup (PXB) runs three sequential phases: backup, prepare, and restore. This page describes the internal work each phase performs. + +For commands and flag selection, see [How-to: backup lifecycle operations](how-xtrabackup-works-how-to.md). For compact tables of privileges and defaults, see [Reference: backup lifecycle](how-xtrabackup-works-reference.md). + +## Three phases at a glance + +The following table maps each operator phase to the InnoDB-internal work the phase performs: + +| Phase | Internal work | +|-------|---------------| +| Backup (hot copy) | Copy data files and capture redo for the full backup window | +| Prepare | Replay captured redo, then roll back transactions open at backup end | +| Restore | Place the prepared files in the destination `datadir` | + +The prepare phase runs the same pipeline as InnoDB crash recovery, but offline on the backup tree. + +## Backup phase + +The backup phase produces a directory that holds everything `mysqld` needs to start a fresh server. The output includes data files for every storage engine, a captured slice of the InnoDB redo log, and a small set of metadata files. + +PXB queries `performance_schema.log_status` to stamp a start log sequence number (LSN), then runs two parallel streams of work: + +* A data-file stream copies on-disk files for every storage engine in use. InnoDB tablespaces (`.ibd` files), the shared tablespace (`ibdata1`), and undo tablespaces make up most of the volume. PXB also copies files for MyISAM, CSV, MyRocks, and other recognized engines. + +* A background redo-log thread tails the InnoDB redo log from the start LSN through the end of the backup window. The thread writes the capture into `xtrabackup_logfile` inside the backup directory. + +PXB picks the backup end LSN during the final lock posture. While `LOCK BINLOG FOR BACKUP` is held, PXB reads `performance_schema.log_status`. The query records the current LSN, binary log file and position, and replica coordinates. The redo thread keeps draining records past that sync-point LSN, then stops. PXB records both LSNs in `xtrabackup_checkpoints`. `to_lsn` is the sync-point LSN. `last_lsn` is the highest LSN the redo thread actually copied. For the LSN fields recorded in incremental backups, see [Incremental backups](create-incremental-backup.md). + +InnoDB incremental backups can skip the full data-file scan with page tracking. The `mysqlbackup` server component records the LSN of every page write. The `--page-tracking` flag lets PXB copy only the pages changed since the previous backup. For setup and limitations, see [Take an incremental backup using page tracking](page-tracking.md). + +MyRocks copies are always wholesale. The MyRocks storage layout does not expose per-page change tracking. PXB copies every MyRocks file on each backup, including incremental backups. For the full storage-engine support list, see [About Percona XtraBackup](about-xtrabackup.md). + +The dual capture lets the prepare phase advance the copy to a single LSN. PXB also writes metadata files that prepare and restore consume, including `xtrabackup_checkpoints`, `xtrabackup_info`, `xtrabackup_binlog_info`, and `backup-my.cnf`. For the full list of artifacts, see [Index of files created by Percona XtraBackup](xtrabackup-files.md). + +PXB stages locking across the backup so InnoDB data manipulation language (DML) keeps running through the bulk file copy. Two flags shape the trade-offs: + +| Flag | Default | Effect | +|------|---------|--------| +| `--lock-ddl` | `ON` | Controls when the backup lock takes effect | +| `--register-redo-log-consumer` | `OFF` | Holds redo files on the server until PXB finishes reading them | + +### Redo capture and why the copy is not yet consistent + +InnoDB keeps writing while PXB reads files, so the copied pages reflect mixed points in time. Pages copied early carry older LSNs than pages copied later. + +To bridge that gap, PXB stamps a start LSN and copies InnoDB data files. A background thread streams the InnoDB redo log in parallel for the entire backup window. + +The redo capture matters for two reasons: + +* The `.ibd` page reads span real clock time. Without a continuous redo log, prepare cannot advance the copy to a single LSN. + +* InnoDB redo files wrap and recycle. The redo thread must read every record before the server overwrites the underlying file. + +Without redo capture, the directory holds physically mixed page images. The copy then has no path to run InnoDB recovery against a single LSN. The dataset therefore becomes consistent during prepare, not during the file copy. + +### Three lock postures + +PXB stages locking across the backup so InnoDB DML keeps running through the bulk copy. + +The following table lists the three sub-phases in order: + +| # | Lock posture | Internal work | +|---|--------------|---------------| +| 1 | No global lock | Copy InnoDB data files and stream redo while DML continues | +| 2 | Backup lock | Copy non-InnoDB files such as `.frm`, `.MRG`, `.MYD`, `.MYI`, `.CSM`, `.CSV`, `.sdi`, and `.par`. Data definition language (DDL) is restricted; DML continues, with details that depend on the server build. | +| 3 | `LOCK BINLOG FOR BACKUP` (short) | Pin binary log or replica coordinates, read `performance_schema.log_status`, and release locks | + +For background on the underlying lock primitives, see [Reduction in locks](reduction-in-locks.md) and the upstream MySQL documentation for [`LOCK INSTANCE FOR BACKUP`](https://dev.mysql.com/doc/refman/{{vers}}/en/lock-instance-for-backup.html). + +### `--lock-ddl=ON` compared with `--lock-ddl=REDUCED` + +The `--lock-ddl` flag controls when PXB takes the backup lock that gates DDL. + +The following table compares the two modes: + +| Value | Backup lock taken | DDL impact | +|-------|-------------------|------------| +| `ON` (default) | Backup start | DDL is restricted for the full backup window | +| `REDUCED` | After the InnoDB copy finishes | DDL is restricted for a shorter window. DDL runs concurrently with the InnoDB copy, so PXB tracks tablespace changes from redo and recopies affected tablespaces when the lock is taken. | + +InnoDB DML continues through the main copy under both modes. + +### `--register-redo-log-consumer`: keep redo until PXB finishes + +On a high-traffic host, the server can recycle redo files before PXB finishes reading them. A purged redo file makes the backup unusable. + +The `--register-redo-log-consumer` flag registers PXB as a redo consumer. The server then retains a redo file until PXB has copied that file. + +Review the trade-offs before enabling the flag: + +* Redo files stay on disk longer, which raises peak redo size during the run. + +* Free space can drop sharply on a long backup of a write-heavy server. + +* The server can briefly stall writes when the consumer advances. + +!!! warning + + Monitor free disk during the backup. Abort with Ctrl+C or `SIGTERM` to `xtrabackup` if disk space reaches a critical threshold. The consumer releases on abort, and the server can then purge redo. + +The default value is `OFF`. Enable the flag only when spare disk space is available. + +### When PXB can skip backup locks + +PXB skips backup locks only when every table in every schema, including the `mysql` system schema, uses InnoDB. Most installations still hold CSV or MyISAM tables in `mysql`, such as `general_log`, so PXB takes backup locks in practice. + +Replication coordinate handling depends on the server build: + +* Percona Server for MySQL can fold relay coordinates into `log_status`, which reduces the locks needed for `--slave-info`. + +* Stock MySQL can still require `FLUSH TABLES WITH READ LOCK` for some relay-position scenarios. + +## Prepare phase: roll forward, then roll back + +`xtrabackup --prepare` runs two coordinated passes against the backup directory. + +Roll forward applies the captured redo to the copied tablespaces. The pass is physical: each redo record describes a page-level change. InnoDB reapplies that change to the page image on disk. + +Roll back removes transactions that had not committed at the backup end LSN. Redo replay can have written those changes to pages, so prepare must reverse the changes at the logical level. + +Roll back relies on two structures inside the tablespaces: + +* Undo records describe how to reverse the row-level effects of each open transaction. + +* Serialized Dictionary Information (SDI) carries table and index definitions inside each tablespace. SDI supplies the schema context that older `.frm` files once provided to rollback. + +Non-InnoDB tables match the same point in time because PXB copied them under backup locks during the backup phase. + +After prepare finishes, the dataset matches the backup end LSN with open transactions stripped. + +## Restore phase: file placement + +The restore phase reads destination paths from the server configuration, including `datadir`, `innodb_data_home_dir`, `innodb_data_file_path`, and `innodb_log_group_home_dir`. + +PXB places files in a fixed order: + +* MyISAM-family files first, including `.MRG`, `.MYD`, `.MYI`, `.CSM`, `.CSV`, `.sdi`, and `.par`. + +* InnoDB tablespaces and indexes second. + +* Log files last. + +File attributes survive the copy. A successful backup writes binary log coordinates to standard error (STDERR). Redirect STDERR to a file when the workflow needs to capture those coordinates. + +## Cloud and streaming: same mechanics, longer windows + +Streaming and `xbcloud` workflows run the same internal phases as a local backup. Slow networks extend the PXB final-sync phase, which holds the short backup locks, including the binary log lock, for longer. + +For workflow procedures, see the following references: + +* [Take a streaming backup](take-streaming-backup.md) + +* [xbcloud binary overview](xbcloud-binary-overview.md) + +## See also + +* [How-to: backup lifecycle operations](how-xtrabackup-works-how-to.md) + +* [Reference: backup lifecycle](how-xtrabackup-works-reference.md) + +* [Index of files created by Percona XtraBackup](xtrabackup-files.md) diff --git a/docs/how-xtrabackup-works-how-to.md b/docs/how-xtrabackup-works-how-to.md new file mode 100644 index 000000000..dd4e6b2b1 --- /dev/null +++ b/docs/how-xtrabackup-works-how-to.md @@ -0,0 +1,136 @@ +# Backup lifecycle + +A Percona XtraBackup (PXB) workflow runs three sequential phases: backup, prepare, and restore. This how-to guide describes each phase and the operational flags that shape locking, redo retention, and recovery output. + +Apply the following principles across every lifecycle phase: + +* Grant only the privileges that the workflow requires. + +* Run backup, prepare, and restore in sequence. + +* Capture binary log coordinates for recovery and replication. + +* Choose operational flags, such as `--lock-ddl` and `--register-redo-log-consumer`, to match the workload and lock tolerance. + +## Grant the minimum privileges + +PXB needs database privileges that match the locking operations and metadata reads in the workflow. + +Complete the following privilege steps: + +1. Grant `BACKUP_ADMIN` so PXB can read `performance_schema.log_status`. + +2. Confirm `BACKUP_ADMIN` allows PXB to take `LOCK INSTANCE FOR BACKUP`, `LOCK TABLES FOR BACKUP`, and `LOCK BINLOG FOR BACKUP`. + +3. Add `RELOAD`, `LOCK TABLES`, or `REPLICATION CLIENT` for workflows that need extra capabilities. Examples include `FLUSH TABLES WITH READ LOCK` and the `--slave-info` option. + +For privilege lists and worked examples, see [Connection and privileges needed](privileges.md). + +## Run a backup and capture binary log coordinates + +Capture binary log coordinates during the backup phase to support recovery and replica bootstrap. + +Complete the following backup steps: + +1. Run `xtrabackup` against the target directory with the standard options. + +2. Redirect STDERR to a log file because PXB writes the binary log coordinates to STDERR. For example, `xtrabackup ... 2> backupout.log`. + +3. Confirm the command returns exit code `0`. + +For the artifact list a backup produces, see [Index of files created by Percona XtraBackup](xtrabackup-files.md). + +## Choose a `--lock-ddl` mode + +PXB uses backup locks to copy non-InnoDB files without blocking InnoDB DML, when the server supports backup locks. + +The following table compares the two `--lock-ddl` modes: + +| Option | Effect | +|--------|--------| +| `--lock-ddl=ON` (default) | Takes the backup lock at start. DDL stays blocked for the full run, unless the workflow changes. | +| `--lock-ddl=REDUCED` | Takes the lock after InnoDB is copied. DDL blocks for a shorter window. This mode accepts DDL contention during the InnoDB phase in exchange for a shorter DDL block. | + +!!! warning + + Both modes block schema changes (DDL) for part of the backup window. Schedule DDL outside the backup, or accept the contention. + +For a compact comparison, see [Reference: backup lifecycle](how-xtrabackup-works-reference.md). + +## When to enable `--register-redo-log-consumer` + +Enable `--register-redo-log-consumer` when heavy write throughput risks purging redo before PXB finishes reading the redo stream. The flag prevents failures caused by missing redo. + +Review the following checks before enabling the flag: + +* Plan for redo bloat and possible disk exhaustion, because the server retains redo longer. + +* Monitor free space and I/O throughout the run. + +* Abort the backup with Ctrl+C or `SIGTERM` to the `xtrabackup` process if disk space reaches a critical threshold. The consumer then releases, the server purges redo, and the operator can free space and retry. + +The default value is `OFF`. Enable the flag only when spare disk space is available. + +## Prepare the backup with `--prepare` + +The prepare phase applies redo and rolls back uncommitted transactions to produce a consistent snapshot. + +Complete the following prepare steps: + +1. Run `xtrabackup --prepare` with `--target-dir` set to the backup directory. + +2. For large or incremental backups, tune prepare performance with the following options: + + * Raise `--use-memory` to between 1 GB and 2 GB when RAM allows, because the default value is small. + + * Pass `--parallel` to apply `.delta` files in parallel on incremental backups, starting in 8.4.0-3. The flag does not parallelize first-pass redo on a full backup. + +The following command shows a tuned prepare run: + +```bash +xtrabackup --prepare --use-memory=2G --parallel=4 --target-dir=/data/backups/ +``` + +!!! warning + + Leave RAM headroom for the OS and `mysqld` when prepare runs on the same host as production. Insufficient headroom triggers OOM kills. + +For flag references, see [`--use-memory`](xtrabackup-option-reference.md#use-memory) and [`--parallel`](xtrabackup-option-reference.md#parallel). + +## Restore with `--copy-back` or `--move-back` + +The restore phase places prepared backup files into the destination `datadir` for `mysqld` startup. + +Complete the following restore steps: + +1. Stop `mysqld` if the process owns the destination `datadir`. + +2. Run `xtrabackup --copy-back` or `xtrabackup --move-back` with `--target-dir` set to the prepared backup. PXB reads destination paths from the configuration, including `datadir`, InnoDB paths, log paths, and related settings. + +3. Set ownership and permissions before starting `mysqld`. For example, run `chown -R mysql:mysql /var/lib/mysql`. Backup files belong to the user that ran `xtrabackup`, while the server expects the `mysql` user. + +4. Start `mysqld`. + +!!! warning + + `--move-back` destroys the original backup directory because the command moves files instead of copying them. Use `--move-back` only when disk space is constrained, and confirm a separate backup copy is available. + +For full restore flows covering full, incremental, and compressed backups, see [Restore full, incremental, and compressed backups](restore-a-backup.md). + +## Stream backups or upload to object storage + +Streaming backups and cloud uploads follow the same lifecycle as a local backup. Slow networks extend the PXB final-sync phase and hold the short backup locks, including the binary log lock, for longer. + +Plan streaming and upload runs with the following guidance: + +* Run streaming backups inside a maintenance window. + +* Size network bandwidth and destination throughput to keep the final-sync phase short. + +* Confirm the destination object store accepts the backup payload size. + +For workflow details, see the following references: + +* [Take a streaming backup](take-streaming-backup.md) + +* [xbcloud binary overview](xbcloud-binary-overview.md) diff --git a/docs/how-xtrabackup-works-reference.md b/docs/how-xtrabackup-works-reference.md new file mode 100644 index 000000000..db0d800b9 --- /dev/null +++ b/docs/how-xtrabackup-works-reference.md @@ -0,0 +1,92 @@ +# Backup lifecycle + +Quick-scan facts: privileges, phases, defaults, and behavior. For the internals narrative, see [What happens underneath](how-xtrabackup-works-explanation.md). For commands and flag selection, see [How-to: backup lifecycle operations](how-xtrabackup-works-how-to.md). + + +## Privileges (summary) + +| Privilege | Typical use | +|-----------|-------------| +| `BACKUP_ADMIN` | Read `performance_schema.log_status`; use `LOCK INSTANCE FOR BACKUP`, `LOCK TABLES FOR BACKUP`, `LOCK BINLOG FOR BACKUP` when applicable | +| `RELOAD` | Workflows using `FLUSH TABLES WITH READ LOCK` or similar | +| `LOCK TABLES` | Workflows that issue table-level locks | +| `REPLICATION CLIENT` | `--slave-info` and related replication metadata | + +[Connection and privileges needed](privileges.md) has the full list. + +## Three operator-facing phases + +| Step | Phase | What happens | +|------|-------|----------------| +| 1 | Backup (hot copy) | Copy data files; stream redo for the whole run | +| 2 | Prepare | Replay redo; roll back uncommitted work | +| 3 | Restore | `--copy-back` or `--move-back` into `datadir` | + +## Three in-backup sub-phases (locking) + +With backup locks, Percona XtraBackup (PXB) typically runs the stages in the table below. Each numbered row is a phase in order: the Locks column is the global lock posture for that phase, and the Work column is what PXB copies or records while those locks apply—InnoDB pages and redo first (often with no global backup lock), then non-InnoDB files under the backup lock (with DDL restricted), then a short `LOCK BINLOG FOR BACKUP` to pin binlog or replica coordinates. Exact steps depend on server version and which storage engines are in use. + +| # | Locks | Work | +|---|--------|------| +| 1 | None global | Copy InnoDB files and redo while data manipulation language (DML) runs | +| 2 | Backup lock | Copy non-InnoDB files (e.g. `.frm`, `.MRG`, `.MYD`, `.MYI`, `.CSM`, `.CSV`, `.sdi` (serialized dictionary information), `.par`); data definition language (DDL) restricted, DML usually allowed—details are server-specific | +| 3 | `LOCK BINLOG FOR BACKUP` (short) | Pin binlog/replica coordinates; read `performance_schema.log_status`; drop locks | + +Backup locks vs FTWRL (`FLUSH TABLES WITH READ LOCK`): [Percona Server backup locks](https://docs.percona.com/percona-server/innovation-release/backup-locks.html). +MySQL {{vers}}: [`LOCK INSTANCE FOR BACKUP`](https://dev.mysql.com/doc/refman/{{vers}}/en/lock-instance-for-backup.html). + +## `--lock-ddl` + +| Value | When the backup lock is taken (typical) | +|-------|----------------------------------------| +| `ON` (default) | Backup start | +| `REDUCED` | After InnoDB data copy finishes | + +`ON` blocks DDL for the whole window from the beginning; `REDUCED` delays that until InnoDB is copied (shorter DDL stall, different overlap with DDL during the InnoDB phase). InnoDB DML usually keeps running through the main copy. + +## `--register-redo-log-consumer` + +| Field | Detail | +|-------|--------| +| Default | Off | +| Does | Registers PXB as a redo consumer so the server won’t purge a redo file until PXB copies that file | +| Cost | Redo can pile up (“redo bloat”); disk use may spike | +| Writes | Server may stall writes briefly while the consumer advances | + +## Prepare options (subset) + +| Option | Notes | +|--------|--------| +| `--use-memory` | Random-access memory (RAM) for prepare only (default 100MB); larger often helps | +| `--parallel` | 8.4.0-3+: parallel `.delta` apply for incrementals; not the same as parallel full-backup redo replay | + +Full flags: [xtrabackup option reference](xtrabackup-option-reference.md). + +## Restore (facts) + +* Pulls paths from `my.cnf` (e.g. `datadir`, `innodb_data_home_dir`, `innodb_data_file_path`, `innodb_log_group_home_dir`). + +* Order: MyISAM-family files first (`.MRG`, `.MYD`, `.MYI`, `.CSM`, `.CSV`, `.sdi`, `.par`), then InnoDB tables/indexes, then logs. + +* Keeps file attributes. + +* Successful backup prints binlog coordinates to standard error (STDERR)—redirect if you need a file. + +## When backup locks are skipped (“lockless”) + +Locks stay off only if every table in every schema—including `mysql`—is InnoDB. Commonly `mysql` still holds CSV/MyISAM tables (e.g. `general_log`), so PXB usually takes backup locks anyway. + +| Server | Notes | +|--------|--------| +| Percona Server for MySQL {{vers}} | `log_status` may carry relay coordinates; `--slave-info` can skip extra locks | +| Standard MySQL {{vers}} | May still need `FLUSH TABLES WITH READ LOCK` with `--slave-info` when you need relay position | + +## Cloud / streaming + +Same phases; slow networks stretch Final Sync and can lengthen short locks. [Streaming backup](take-streaming-backup.md), [xbcloud overview](xbcloud-binary-overview.md). + +## See also + +* [Index of files created by Percona XtraBackup](xtrabackup-files.md) + +* [Restore full, incremental, and compressed backups](restore-a-backup.md) diff --git a/docs/how-xtrabackup-works.md b/docs/how-xtrabackup-works.md deleted file mode 100644 index aa2250fd5..000000000 --- a/docs/how-xtrabackup-works.md +++ /dev/null @@ -1,114 +0,0 @@ -# How Percona XtraBackup works - -Percona XtraBackup is based on InnoDB’s crash-recovery functionality. -It copies your InnoDB data files, which results in data that is internally -inconsistent; but then it performs crash recovery on the files to make them a -consistent, usable database again. - -This works because InnoDB maintains a redo log, also called the transaction -log. This contains a record of every change to InnoDB data. When InnoDB -starts, it inspects the data files and the transaction log, and performs two -steps. It applies committed transaction log entries to the data files, and it -performs an undo operation on any transactions that modified data but did not -commit. - -The `--register-redo-log-consumer` parameter is disabled by default. When enabled, this parameter lets Percona XtraBackup register as a redo log consumer at the start of the backup. The server does not remove a redo log that Percona XtraBackup (the consumer) has not yet copied. The consumer reads the redo log and manually advances the log sequence number (LSN). The server blocks the writes during the process. Based on the redo log consumption, the server determines when it can purge the log. - -Percona XtraBackup remembers the LSN when it starts, and then copies the data files. The operation takes time, and the files may change, then LSN reflects the state of the database at different points in time. Percona XtraBackup also runs a background process that watches the transaction log files, and copies any changes. Percona XtraBackup does this continually. The transaction logs are written in a round-robin fashion, and can be reused. - -Percona XtraBackup uses [Backup locks :octicons-link-external-16:](https://docs.percona.com/percona-server/innovation-release/backup-locks.html) where available as a lightweight alternative to `FLUSH TABLES WITH READ LOCK`. MySQL {{vers}} allows acquiring an instance level backup lock via the `LOCK INSTANCE FOR BACKUP` statement. - -Locking is only done for MyISAM and other non-InnoDB tables -after Percona XtraBackup finishes backing up all InnoDB/XtraDB data and -logs. Percona XtraBackup uses this automatically to copy non-InnoDB data to -avoid blocking DML queries that modify InnoDB tables. - -!!! important - - The `BACKUP_ADMIN` privilege is required to query the - `performance_schema_log_status` for either `LOCK - INSTANCE FOR BACKUP` or `LOCK TABLES FOR BACKUP`. - -xtrabackup tries to avoid backup locks and `FLUSH TABLES WITH READ LOCK` -when the instance contains only InnoDB tables. In this case, xtrabackup -obtains binary log coordinates from `performance_schema.log_status`. `FLUSH -TABLES WITH READ LOCK` is still required in MySQL {{vers}} when xtrabackup is -started with the `--slave-info`. The `log_status` table in Percona -Server for MySQL {{vers}} is extended to include the relay log coordinates, so no locks are -needed even with the `--slave-info` option. - -!!! admonition "See also" - - [MySQL Documentation: LOCK INSTANCE FOR BACKUP :octicons-link-external-16:](https://dev.mysql.com/doc/refman/{{vers}}/en/lock-instance-for-backup.html) - -When backup locks are supported by the server, xtrabackup first copies -InnoDB data, runs the `LOCK TABLES FOR BACKUP` and then copies the MyISAM -tables. Once this is done, the backup of the files will -begin. It will backup .frm, .MRG, .MYD, .MYI, .CSM, -.CSV, `.sdi` and `.par` files. - -After that xtrabackup will use `LOCK BINLOG FOR BACKUP` to block all -operations that might change either binary log position or -`Exec_Source_Log_Pos` or `Exec_Gtid_Set` (i.e. source binary log coordinates -corresponding to the current SQL thread state on a replication replica) as -reported by `SHOW BINARY LOG STATUS` or `SHOW REPLICA STATUS`. xtrabackup will then finish copying -the REDO log files and fetch the binary log coordinates. After this is completed -xtrabackup will unlock the binary log and tables. - -Finally, the binary log position will be printed to `STDERR` and xtrabackup -will exit returning 0 if all went OK. - -Note that the `STDERR` of xtrabackup is not written in any file. You will -have to redirect it to a file, for example, `xtrabackup OPTIONS 2> backupout.log`. - -It will also create the following files in the -directory of the backup. - -## Prepare phase - -This phase involves two primary operations: applying the redo log and the undo log. - -### Redo Log Application (Physical Operation) - -XtraBackup directly applies changes recorded in the redo log to specific page offsets within the tablespace (IBD file). This is a physical operation, meaning it works at the page level, without regard for rows or transactions. - -It's important to understand that the redo log might contain uncommitted transactions, as the server can flush or write these to the log. Therefore, the redo log application doesn't inherently guarantee transactional consistency. - -### Undo Log Application (Logical Operation) - -Following the redo log application, XtraBackup uses the undo log to logically roll back changes from any uncommitted transactions present in the redo log. -Undo log records are of two types: `INSERT` and `UPDATE`. Each record contains a `table_id`, which XtraBackup uses to locate the table definition. - -To perform the rollback, XtraBackup initializes the InnoDB engine and data dictionary, then uses Serialized Dictionary Information (SDI) from the tablespace (a JSON representation of the table) to parse index pages and apply undo operations. - -Tables are loaded as evictable, and XtraBackup scans data dictionary indexes to relate `table_id` to tablespace, which is used during rollback. User tables are loaded only when needed for rollback. This design significantly reduces memory and I/O usage, speeds up the `--prepare` phase, and improves Percona XtraDB Cluster SST performance. - -### Achieving Consistency: Redo, Undo, and MyISAM - -The `--prepare` phase ensures that InnoDB tables are rolled forward to the point where the backup completed, not rolled back to where it began. This point aligns with the time a `FLUSH TABLES WITH READ LOCK` was taken, which is crucial for maintaining consistency with MyISAM tables. - -Therefore, after the `--prepare` phase, both InnoDB and MyISAM tables are eventually consistent with each other. - -## Restore a backup - -To restore a backup with xtrabackup you can use the `--copy-back` or -`--move-back` options. - -xtrabackup will read from the `my.cnf` the variables datadir, -innodb_data_home_dir, innodb_data_file_path, -innodb_log_group_home_dir and check that the directories exist. - -It will copy the MyISAM tables, indexes, etc. (.MRG, .MYD, -.MYI, .CSM, .CSV, `.sdi`, -and `par` files) first, InnoDB tables and indexes next and the log files at -last. It will preserve file’s attributes when copying them, you may have to -change the files’ ownership to `mysql` before starting the database server, as -they will be owned by the user who created the backup. - -Alternatively, the `--move-back` option may be used to -restore a backup. This option is similar to `--copy-back` -with the only difference that instead of copying files it moves them to their -target locations. As this option removes backup files, it must be used with -caution. It is useful in cases when there is not enough free disk space to hold -both data files and their backup copies. - diff --git a/docs/index-contents.md b/docs/index-contents.md index 9199be759..6e754523c 100644 --- a/docs/index-contents.md +++ b/docs/index-contents.md @@ -21,6 +21,7 @@ - [Dump schema for all tables](dump-schema.md) - [Encrypt backups](encrypt-backups.md) - [Encrypted InnoDB tablespace backups](encrypted-innodb-tablespace-backups.md) + - [How Percona XtraBackup works: what happens underneath](how-xtrabackup-works-explanation.md) - [Exponential backoff](xbcloud-exbackoff.md) - [FIFO data sink](xbcloud-binary-fifo-datasink.md) - [Files in the DEB package](apt-files.md) @@ -29,7 +30,7 @@ - [Frequently asked questions](faq.md) - [Get help from Percona](get-help.md) - [Glossary](glossary.md) - - [How Percona XtraBackup works](how-xtrabackup-works.md) + - [How-to: backup lifecycle operations](how-xtrabackup-works-how-to.md) - [How to create a new (or repair a broken) GTID-based replica](create-gtid-replica.md) - [How to set up a replica for replication in 6 simple steps with Percona XtraBackup](set-up-replication.md) - [Improved log statements](log-enhancements.md) @@ -55,6 +56,7 @@ - [Prepare an individual partitions backup](prepare-individual-partitions-backup.md) - [Quickstart Overview](quickstart-overview.md) - [Reduced backup lock time](reduction-in-locks.md) + - [Reference: backup lifecycle](how-xtrabackup-works-reference.md) - [Restore a partial backup](restore-partial-backup.md) - [Restore full, incremental, and compressed backups](restore-a-backup.md) - [Restore single tables between databases](restore-individual-tables.md) diff --git a/docs/reduction-in-locks.md b/docs/reduction-in-locks.md index 2a7b08d12..458c72c5c 100644 --- a/docs/reduction-in-locks.md +++ b/docs/reduction-in-locks.md @@ -115,5 +115,5 @@ The following operations are performed to ensure data remains intact and consist 5. Replace the file that matches the name without the `.new` extension using the `.new` extension. 6. Replace the `meta` and `delta` files that match the name without the `.new` in the name using `.new.meta` and `.new.delta`. -After `Phase 3` is completed, the regular [crash recovery](./how-xtrabackup-works.md) starts. +After `Phase 3` is completed, the regular [crash recovery](./how-xtrabackup-works-explanation.md) starts. diff --git a/docs/server-backup-version-comparison.md b/docs/server-backup-version-comparison.md index 2bc07617b..3edceac40 100644 --- a/docs/server-backup-version-comparison.md +++ b/docs/server-backup-version-comparison.md @@ -7,7 +7,7 @@ XtraBackup version that is equal to your source server major version. This means !!! admonition "See also" - [How XtraBackup works](how-xtrabackup-works.md) + [How Percona XtraBackup works: what happens underneath](how-xtrabackup-works-explanation.md) The `--no-server-version-check` option performs the following test. Before the backup/prepare starts, XtraBackup compares the source server version to diff --git a/mkdocs-base.yml b/mkdocs-base.yml index 22338f531..9b6b20feb 100644 --- a/mkdocs-base.yml +++ b/mkdocs-base.yml @@ -184,7 +184,9 @@ nav: - How Percona XtraBackup works: - about-xtrabackup.md - features.md - - how-xtrabackup-works.md + - 'What happens underneath': how-xtrabackup-works-explanation.md + - 'How-to: backup lifecycle operations': how-xtrabackup-works-how-to.md + - 'Reference: backup lifecycle': how-xtrabackup-works-reference.md - differences-logical-backup.md - backup-overview.md - backup-strategy.md