PostgreSQL 备份与恢复（第三章）

pg_probackup备份工具

1.1 pg_probackup备份简介

pg_probackup是管理PG数据库集群备份和恢复的工具。用于执行周期性备份PG实例。兼容PG9.5，,96,10,11,12,13。和其他备份解决方案相比，提供下面优势，帮助部署不同备份策略和处理大量数据。

1.2 pg_probackup功能与限制

1、增量备份。页级别的增量备份，节省磁盘空间，加速备份和转储。有3种不同增量模式，根据数据流部署不同备份策略。
2、增量转储。页级别的转储，在目标目录中重用有效未更改的页面来加速转储。
3、合并。帮助部署”incrementally updated backups”策略，无须进行完备备份。
4、验证。自动数据一致性校验和按需备份校验，无须实际数据恢复。
5、校验。使用checkdb命令按需验证PG实例
6、保留。根据保留策略管理存档和备份。可以根据恢复实际或要保留的备份数配置保留策略。还可以为特定备份制定生存时间，过期的备份可以合并或者删除。
7、并行化。在多个并行线程上进行备份、恢复、合并、删除、验证和检验
8、压缩。以压缩状态存储备份数据以节省磁盘空间
9、消除重复数据。通过不复制未更改的非数据文件如_vm或者_fsm来节省磁盘空间
10、远程操作。备份位于远程系统上的PG实例或远程恢复备份
11、从备机上进行备份。通过从备机上进行备份来避免主机上的额外负载。
12、外部目录。备份位于数据目录PGDATA之外的文件和目录，如脚本、配置文件、日志或SQL转储文件。
13、备份Catalog。以纯文本或JSON格式获取备份列表和相应的元信息
14、归档catalog。以纯文本或JSON格式获取所有WAL时间线和相应元信息的列表
15、部分还原。仅还原制定的数据库或从转储中排出指定数据库
复制

管理备份数据:pg_probackup创建一个备份目录。这个目录存储所有备份文件包括额外的元信息，以及WAL归档。可以将不同实例的备份存储在单个目录的不同子目录中。通过该工具可以进行完整备份和增量备份。

1、全量备份，包含从头恢复数据库集群需要的所有数据文件
2、增量备份。只存储自上次以来更改的数据。允许减小备份大小并加快备份操作。支持以下增量备份模式：
1）PAGE备份。扫描自上次备份以来归档的所有WAL文件。新创建的备份仅包含WAL记录到的页面。如果这些文件大小与数据库集群文件总大小相当，则加速比较小，但备份占用空间仍然较小。
2）DELTA备份。读取PGDATA目录下所有数据文件，仅拷贝自上次备份以来的脏页。无须持续归档。此外，该模式还可以施加与全量备份相同的只读IO压力。
3）PTRACK备份。会动态追踪页面改动。无须持续归档。每次更新数据页时，都会在PTRACK位图标记。由于一个页面只需要PTRACK位图中的一个bit，这个位图非常小。跟踪意味着会有开销，但会显著加速增量备份。
无论选择哪种备份类型，都支持以下WAL传递策略：
1）自主备份，通过复制协议传输将集群恢复到备份时的一致性状态需要的所有WAL文件。即使没有设置连续归档，需要的WAL段也会包含在备份中。
2）归档备份，依赖于连续归档。
复制

限制：备份和转储的PG必须block_size和wal_block_size兼容且有相同主版本号；Windows下不支持ssh远程备份；ssh远程备份时，pg_probackup的版本必须相同。
复制

1.3 pg_probackup安装与使用

wget https://github.com/postgrespro/pg_probackup/archive/2.4.15.tar.gz
tar -zxvf 2.4.15.tar.gz 
cd pg_probackup-2.2.7/
make USE_PGXS=1 PG_CONFIG=/data/postgresql/bin/pg_config top_srcdir=/home/postgres/postgresql-13.3
make USE_PGXS=1 PG_CONFIG=/data/postgresql/bin/pg_config top_srcdir=/home/postgres/postgresql-13.3 install
复制

一旦安装了pg_probackup，请完成以下设置。

1.4 pg_probackup初始化备份目录

所谓初始化：就是创建一个备份目录；用于存放备份文件和归档日志文件

初始化备份目录，请运行以下命令：

pg_probackup init -B backup_dir
-- 示例
pg_probackup init -B /data/postgres/probackup
复制

pg_probackup创建备份目录，其中包含以下子目录：

– wal/-wal文件目录。
– backups/-备份文件的目录。
备份目录初始化了就可以添加备份实例

1.5 向备份目录添加新的备份Instance

pg_probackup可以在一个备份目录中存储多个数据库集群的备份。要设置所需的子目录，必须将备份实例添加到要备份的每个数据库群集的备份目录中。

要添加新的备份实例，请运行以下命令：

pg_probackup add-instance -B backup_dir -D data_dir --instance instance_name [remote_options]
-- 本地添加备份instance 示例
pg_probackup add-instance -B /data/postgres/probackup -D /data/postgres/data --instance local_6000
复制

– data_dir是要备份的群集的数据目录。要设置和使用pg_probackup，需要对此目录的写访问权限。
– instance_name是将存储此群集的WAL和备份文件的子目录的名称。
– remote_options是可选参数，仅当data_dir位于远程系统上时，才需要指定这些参数。后续在详细讲解，参考
pg_probackup在备份目录的 backups 和 wal 目录下创建 instance_name 子目录。backups/instance_name 目录包含 pg_probackup.conf 控制此备份实例的pg_probackup设置的配置文件。如果使用remote_选项运行此命令，则指定的参数将添加到pg_probackup.conf。

1.6 配置PostgreSQL集群

如果使用非postgres用户需要配置以下权限

For PostgreSQL 10 or higher:
$ psql -p 5432 -U postgres
BEGIN;
CREATE ROLE backup WITH LOGIN REPLICATION;
GRANT USAGE ON SCHEMA pg_catalog TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.current_setting(text) TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.pg_is_in_recovery() TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.pg_start_backup(text, boolean, boolean) TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.pg_stop_backup(boolean, boolean) TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.pg_create_restore_point(text) TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.pg_switch_wal() TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.pg_last_wal_replay_lsn() TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.txid_current() TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.txid_current_snapshot() TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.txid_snapshot_xmax(txid_snapshot) TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.pg_control_checkpoint() TO backup;
COMMIT;
复制

1.7 配置postgresql.conf

– 确保wal_level参数高于minimal。
– 如果要在primary上配置存档，则存档模式必须设置为“on”或“always”。要在standby下执行存档，请将此参数设置为“always"
– 设置archive_命令参数，如下所示：

archive_command = 'install_dir/pg_probackup archive-push -B backup_dir --instance instance_name --wal-file-name=%f [remote_options]'
-- 示例
/opt/pg13/bin/pg_probackup archive-push -B /data/postgres/probackup --instance local_6000 --wal-file-path=%p
-- wal-file-name=%f
复制

1.8 创建全量备份

创建备份，命令

pg_probackup backup -B backup_dir --instance instance_name -b backup_mode
复制

示例

pg_probackup backup -B /data/postgres/probackup --instance local_6000 -b full
复制

1.9 查看备份

pg_probackup show -B /data/postgres/probackup
pg_probackup show -B /data/postgres/probackup --instance local_6000 -i QGQMKH
pg_probackup show -B /data/postgres/probackup --instance local_6000 --archive
复制

1.10 基于PITR 备份与恢复

-- 备份
pg_probackup backup -B /data/postgres/probackup --instance local_6000 -b full
--恢复之前需要先停库；在清理PGDATA目录以及外在的表空间目录
pg_ctl stop;
rm -rf $PGDATA/*
--恢复到新的PGDATA目录
pg_probackup restore -B /data/postgres/probackup --instance local_6000 -D /data/postgres/data6000 --recovery-target-time='2021-07-11 11:05:17'
 11:05:17'
-- 恢复到原目录
pg_probackup restore -B /data/postgres/probackup --instance local_6000  --recovery-target-time='2021-07-11 11:05:17'
复制

1.11 增量备份

-- 增量备份DELTA
pg_probackup backup -B /data/postgres/probackup --instance local_6000 -b delta
-- 增量备份page
pg_probackup backup -B /data/postgres/probackup --instance local_6000 -b page
-- https://postgrespro.com/docs/postgrespro/13/app-pgprobackup
复制

1.12 远程备份

--set up ssh
ssh-copy-id postgres@node2
--备份库 添加备份实例
pg_probackup add-instance -B /data/postgres/probackup -D /data/postgres/data --instance pg207_6000 --remote-proto=ssh --remote-host=192.168.127.126 --remote-port=22  --remote-user=postgres --remote-path=/opt/pg13/bin  --ssh-options='-o ServerAliveCountMax=5 -o ServerAliveInterval=60'
--案例库 配置的参数
--备份需要用到postgres超级用户；所以在pg_hba.conf;
host    all            postgres       192.168.127.129/24        trust 
-- postgresql.conf
wal_level = replica
archive_mode = on
archive_command = '/opt/pg13/bin/pg_probackup archive-push -B /data/postgres/probackup --instance pg207_6000 --wal-file-path=%p --wal-file-name=%f --remote-proto=ssh --remote-host=192.168.127.129 --remote-port=22  --remote-user=postgres --remote-path=/opt/pg12/bin'
-- 备份库； 执行远程备份
pg_probackup backup -B /data/postgres/probackup --instance pg207_6000 -b full --remote-proto=ssh --remote-host=192.168.127.126 --remote-port=22  --remote-user=postgres --remote-path=/opt/pg12/bin  --ssh-options='-o ServerAliveCountMax=5 -o ServerAliveInterval=60'
--Take a FULL backup:
pg_probackup-11 backup -B /mnt/backups --instance 'pg-11' -b FULL --stream --remote-host=postgres_host --remote-user=postgres -U backup -d backupdb
--Take an incremental backup in the DELTA mode:
pg_probackup-11 backup -B /mnt/backups --instance 'pg-11' -b delta --stream --remote-host=postgres_host --remote-user=postgres -U backup -d backupdb
--
pg_probackup-11 set-config -B /mnt/backups --instance 'pg-11' --remote-host=postgres_host --remote-user=postgres -U backup -d backupdb
pg_probackup-11 backup -B /mnt/backups --instance 'pg-11' -b delta --stream
--pg_probackup-11 show-config -B /mnt/backups --instance 'pg-11'
--pg_probackup-11 show -B /mnt/backups --instance 'pg-11'
-- 备份库；查看备份
pg_probackup show -B /data/postgres/probackup --instance pg207_6000
复制

1.13 远程恢复

-- 恢复
pg_probackup restore -B /data/postgres/probackup --instance local_6000 --recovery-target-time='2021-07-11 14:54:55'
复制

1.14 delta模式和page模式的备份性能差异

测试目的：
本次测试pg_probackup工具的delta模式和page模式的备份性能差异
测试准备:
准备10亿条数据(数据大小146G)：

pgbench -i -s 10000 -U backup -h 127.0.0.1 pgbench
pgbench=# \dt
             List of relations
 Schema |       Name       | Type  | Owner  
--------+------------------+-------+--------
 public | pgbench_accounts | table | backup
 public | pgbench_branches | table | backup
 public | pgbench_history  | table | backup
 public | pgbench_tellers  | table | backup
(4 rows)

pgbench=# select count(*) from pgbench_accounts;
   count    
------------
 1000000000
(1 row)
磁盘占用：
[postgres@ecos75]$ df -h
Filesystem                Size  Used Avail Use% Mounted on
devtmpfs                  7.5G     0  7.5G   0% /dev
tmpfs                     7.5G   12K  7.5G   1% /dev/shm
tmpfs                     7.5G  9.2M  7.5G   1% /run
tmpfs                     7.5G     0  7.5G   0% /sys/fs/cgroup
/dev/mapper/vg00-lv_root   20G  9.9G  8.8G  53% /
/dev/sda1                 501M  335M  130M  73% /boot
/dev/sda2                 524M   12K  524M   1% /boot/efi
/dev/sdb1                 296G  147G  134G  53% /postgresql/data
/dev/sdc1                 296G   65M  281G   1% /postgresql/backup
tmpfs                     1.5G     0  1.5G   0% /run/user/0
/dev/sdd1                 296G  122G  160G  44% /postgresql/archive
复制

全量备份（DELTA）

pg_probackup backup -B /postgresql/backup/ --instance pgbak -j 4 -U probackup --stream -b full
[postgres@ecos75]$ pg_probackup backup -B /postgresql/backup/ --instance pgbak -j 4 -U probackup --stream -b full --temp-slot
INFO: Backup start, pg_probackup version: 2.4.15, instance: pgbak, backup ID: QLOIL1, backup mode: FULL, wal mode: STREAM, remote: false, compress-algorithm: none, compress-level: 1
WARNING: This PostgreSQL instance was initialized without data block checksums. pg_probackup have no way to detect data block corruption without them. Reinitialize PGDATA with option '--data-checksums'.
INFO: PGDATA size: 146GB
INFO: Start transferring data files
INFO: Data files are transferred, time elapsed: 33m:19s
2021-07-11 16:12:20.388 CST [24069] LOG:  restore point "pg_probackup, backup_id QLOIL1" created at 23/C0000178
2021-07-11 16:12:20.388 CST [24069] STATEMENT:  SELECT pg_catalog.pg_create_restore_point($1)
INFO: wait for pg_stop_backup()
INFO: pg_stop backup() successfully executed
INFO: Syncing backup files to disk
INFO: Backup files are synced, time elapsed: 15s
INFO: Validating backup QLOIL1
INFO: Backup QLOIL1 data files are valid
INFO: Backup QLOIL1 resident size: 146GB
INFO: Backup QLOIL1 completed
时间:2021-07-11 15:39 ~ 2021-07-11 16:31
复制

ps:出现如下错误，需要配置pg lib库环境变量：

export LD_LIBRARY_PATH=/usr/local/postgresql-13.3/lib
ERROR: could not connect to database postgres: could not connect to server: No such file or directory
  Is the server running locally and accepting
  connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
复制

增量数据准备：
插入一亿条增量数据（约15G数据）

pgbench -i -s 1000 -U probackup -h 127.0.0.1 pgbench_inc
磁盘占用：
[postgres@ecos75]$ df -h
Filesystem                Size  Used Avail Use% Mounted on
devtmpfs                  7.5G     0  7.5G   0% /dev
tmpfs                     7.5G   12K  7.5G   1% /dev/shm
tmpfs                     7.5G  9.1M  7.5G   1% /run
tmpfs                     7.5G     0  7.5G   0% /sys/fs/cgroup
/dev/mapper/vg00-lv_root   20G  9.9G  8.8G  53% /
/dev/sda1                 501M  335M  130M  73% /boot
/dev/sda2                 524M   12K  524M   1% /boot/efi
tmpfs                     1.5G     0  1.5G   0% /run/user/0
/dev/sdb1                 296G  162G  119G  58% /postgresql/data
/dev/sdc1                 296G  147G  134G  53% /postgresql/backup
/dev/sdd1                 296G   13G  268G   5% /postgresql/archive
复制

增量备份（DELTA）

pg_probackup backup -B /postgresql/backup/ --instance pgbak -j 4 -U probackup --stream -b delta --temp-slot
[postgres@ecos75]$ pg_probackup backup -B /postgresql/backup/ --instance pgbak -j 4 -U probackup --stream -b delta --temp-slot
INFO: Backup start, pg_probackup version: 2.4.15, instance: pgbak, backup ID: QLOMCV, backup mode: DELTA, wal mode: STREAM, remote: false, compress-algorithm: none, compress-level: 1
WARNING: This PostgreSQL instance was initialized without data block checksums. pg_probackup have no way to detect data block corruption without them. Reinitialize PGDATA with option '--data-checksums'.
INFO: Parent backup: QLOIL1
INFO: PGDATA size: 161GB
INFO: Start transferring data files
INFO: Data files are transferred, time elapsed: 31m:29s
2021-07-11 17:32:00.310 CST [30184] LOG:  restore point "pg_probackup, backup_id QLOMCV" created at 26/C1000178
2021-07-11 17:32:00.310 CST [30184] STATEMENT:  SELECT pg_catalog.pg_create_restore_point($1)
INFO: wait for pg_stop_backup()
INFO: pg_stop backup() successfully executed
INFO: Syncing backup files to disk
INFO: Backup files are synced, time elapsed: 0
INFO: Validating backup QLOMCV
INFO: Backup QLOMCV data files are valid
INFO: Backup QLOMCV resident size: 15GB
INFO: Backup QLOMCV completed
时间：2021-07-11 17:00 ~ 2021-07-11 17:35
复制

全量备份（PAGE）

修改postgresql.conf
vi postgresql.conf
archive_command='pg_probackup archive-push -B /postgresql/backup --instance pgbak --wal-file-name=%f'
删除增量库pgbench_inc
postgres=# drop database pgbench_inc;
pg_probackup backup -B /postgresql/backup/ --instance pgbak -j 4 -U probackup -b full
[postgres@ecos75]$ pg_probackup backup -B /postgresql/backup/ --instance pgbak -j 4 -U probackup -b full
INFO: Backup start, pg_probackup version: 2.4.15, instance: pgbak, backup ID: QLILZE, backup mode: FULL, wal mode: ARCHIVE, remote: false, compress-algorithm: none, compress-level: 1
WARNING: This PostgreSQL instance was initialized without data block checksums. pg_probackup have no way to detect data block corruption without them. Reinitialize PGDATA with option '--data-checksums'.
INFO: Wait for WAL segment /postgresql/backup/wal/pgbak/0000000100000020000000AE to be archived
INFO: PGDATA size: 146GB
INFO: Start transferring data files
INFO: Data files are transferred, time elapsed: 38m:52s
INFO: wait for pg_stop_backup()
INFO: pg_stop backup() successfully executedv
INFO: Syncing backup files to disk
INFO: Backup files are synced, time elapsed: 14s
INFO: Validating backup QLILZE
INFO: Backup QLILZE data files are valid
INFO: Backup QLILZE resident size: 146GB
INFO: Backup QLILZE completed
时间：2021-07-12 08:57 ~ 2021-07-12 09:54
复制

增量数据准备:
插入一亿条增量数据（约15G数据）

pgbench -i -s 1000 -U probackup -h 127.0.0.1 pgbench_inc
复制

磁盘占用：

[postgres@ecos75]$ df -h
Filesystem                Size  Used Avail Use% Mounted on
devtmpfs                  7.5G     0  7.5G   0% /dev
tmpfs                     7.5G   12K  7.5G   1% /dev/shm
tmpfs                     7.5G   18M  7.5G   1% /run
tmpfs                     7.5G     0  7.5G   0% /sys/fs/cgroup
/dev/mapper/vg00-lv_root   20G  9.9G  8.8G  53% /
/dev/sda1                 501M  335M  130M  73% /boot
/dev/sda2                 524M   12K  524M   1% /boot/efi
/dev/sdb1                 296G  162G  119G  58% /postgresql/data
/dev/sdc1                 296G  159G  122G  57% /postgresql/backup
tmpfs                     1.5G     0  1.5G   0% /run/user/0
/dev/sdd1                 296G  122G  159G  44% /postgresql/archive
复制

增量备份（PAGE）

pg_probackup-12 backup -B /postgresql/backup/ --instance pgbak -b PAGE -U probackup
[postgres@ecos75]$ pg_probackup backup -B /postgresql/backup/ --instance pgbak -b page -j 4 -U probackup
INFO: Backup start, pg_probackup version: 2.4.15, instance: pgbak, backup ID: QLPZ42, backup mode: PAGE, wal mode: ARCHIVE, remote: false, compress-algorithm: none, compress-level: 1
WARNING: This PostgreSQL instance was initialized without data block checksums. pg_probackup have no way to detect data block corruption without them. Reinitialize PGDATA with option '--data-checksums'.
INFO: Wait for WAL segment /postgresql/backup/wal/pgbak/0000000100000029000000C8 to be archived
INFO: Parent backup: QLPUN6
INFO: PGDATA size: 161GB
INFO: Extracting pagemap of changed blocks
INFO: Pagemap successfully extracted, time elapsed: 142 sec
INFO: Start transferring data files
INFO: Data files are transferred, time elapsed: 5m:16s
INFO: wait for pg_stop_backup()
INFO: pg_stop backup() successfully executed
INFO: Syncing backup files to disk
INFO: Backup files are synced, time elapsed: 8s
INFO: Validating backup QLPZ42
INFO: Backup QLPZ42 data files are valid
INFO: Backup QLPZ42 resident size: 15GB
INFO: Backup QLPZ42 completed
时间：2021-07-12 10:33 ~ 2021-07-12 10:44
复制

测试结果：
本次只提供了测试流程和其中一次测试结果：

通过本次测试发现，delta模式与page模式全量备份差异不明显，增量备份方面，page模式比delta模式备份效率高3倍（在这个数据量级下，数据量越大，差异或许更大）
复制