PolarDB for PostgreSQL 三节点开源版 3主机集群部署实践

内网ip:

192.168.1.103  
192.168.1.120  
192.168.1.121

环境依赖

cat /etc/redhat-release
CentOS Linux release 7.9.2009 (Core)

部署步骤

1、安装依赖包(所有节点都执行)
yum install -y gcc bison flex libzstd-devel libzstd zstd cmake openssl-devel protobuf-devel readline-devel libxml2-devel libxslt-devel zlib-devel bzip2-devel lz4-devel snappy-devel python-devel unzip

2、添加OS用户, 用于部署PolarDB数据库集群（所有节点都执行）
useradd polardb    

3、设置用户密码（所有节点都执行）
passwd polardb

4、下载PolarDB for PostgreSQL源码（所有节点都执行）
su - polardb  
  
wget https://github.com/alibaba/PolarDB-for-PostgreSQL/archive/refs/heads/master.zip  
  
unzip master.zip  

5、配置PolarDB OS用户主机之间的ssh互认, 配置ssh互认后, 方便集群管理, 这也是Greenplum使用的方法.
all node:

产生ssh key
su - polardb  
ssh-keygen  
chmod 700 ~/.ssh
chmod 400 ~/.ssh/id_rsa*

all node:

配置互相认证

su - digoal  
ssh-copy-id -f polardb@192.168.1.103  
ssh-copy-id -f polardb@192.168.1.120 
ssh-copy-id -f polardb@192.168.1.121 
  
输入目标主机polardb用户登陆密码, 完成互认证

all node:

验证是否不需要密码, 返回日期表示已经完成互认

su - polardb  
ssh 'polardb@192.168.1.103' date  
ssh 'polardb@192.168.1.120' date  
ssh 'polardb@192.168.1.121' date    

6、配置环境变量

all node:

su - polardb  
  
vi ~/.bashrc  
  
export POLARDBHOME="$HOME/polardb"  
export PATH="$POLARDBHOME/bin:$PATH"  
export LD_LIBRARY_PATH="$POLARDBHOME/lib:$LD_LIBRARY_PATH"  
export PGUSER=polardb  
export PGDATABASE=postgres  
export PGHOST=/tmp  
export PGPORT=10001  
应用环境变量生效

su - polardb 
  
. ~/.bashrc  

7、编译安装PolarDB for PostgreSQL二进制软件.

all node:

su - polardb  
cd ~/PolarDB-for-PostgreSQL-master  
设置安装目录

export PG_INSTALL=$HOME/polardb  
其他部署详情, 可以看一下build.sh脚本的内容.

编译安装二进制软件:

sh build.sh debug   ## 开发环境 
或  
sh build.sh deploy  ## 生产环境
8、配置PolarDB 3主机集群.

创建配置文件存放目录

all node:

su - digoal  
mkdir $POLARDBHOME/etc  
创建存放PolarDB集群数据文件的目录

su - root  
mkdir -p /data01/polardb/data  
chown -R polardb:polardb /data01/polardb  
chmod 700 /data01/polardb  
生成集群部署配置文件模板(这个只需要在master主机执行即可, 192.168.1.103)

master node:

su - polardb  
  
touch $POLARDBHOME/etc/polardb_paxos.conf  
  
pgxc_ctl -v -c $POLARDBHOME/etc/polardb_paxos.conf prepare standalone 

修改配置文件内容, 匹配我们的三主机环境
#!/usr/bin/env bash
#
# polardb Configuration file for pgxc_ctl utility. 
#
# Configuration file can be specified as -c option from pgxc_ctl command.   Default is
# $PGXC_CTL_HOME/pgxc_ctl.org.
#
# This is bash script so you can make any addition for your convenience to configure
# your polardb.
#
#========================================================================================
#
#
# pgxcInstallDir variable is needed if you invoke "deploy" command from pgxc_ctl utility.
# If don't you don't need this variable.
pgxcInstallDir=$HOME/polardb
#---- OVERALL -----------------------------------------------------------------------------
#
pgxcOwner=polardb		# owner of the Postgres-XC databaseo cluster.  Here, we use this
						# both as linus user and database user.  This must be
						# the super user of each coordinator and datanode.
pgxcUser=polardb		# OS user of Postgres-XC owner

tmpDir=/tmp					# temporary dir used in XC servers
localTmpDir=$tmpDir			# temporary dir used here locally

configBackup=n					# If you want config file backup, specify y to this value.
configBackupHost=pgxc-linker	# host to backup config file
configBackupDir=$HOME/pgxc		# Backup directory
configBackupFile=pgxc_ctl.bak	# Backup file name --> Need to synchronize when original changed.
standAlone=n

dataDirRoot=/data01/polardb/data

#---- Datanodes -------------------------------------------------------------------------------------------------------

#---- Shortcuts --------------
datanodeMasterDir=$dataDirRoot/dn_master
datanodeSlaveDir=$dataDirRoot/dn_slave
datanodeLearnerDir=$dataDirRoot/dn_learner
datanodeArchLogDir=$dataDirRoot/datanode_archlog

#---- Overall ---------------
primaryDatanode=datanode_1				# Primary Node.
datanodeNames=(datanode_1)
datanodePorts=(10001)	# Master and slave use the same port!
datanodePoolerPorts=(10011)	# Master and slave use the same port!
datanodePgHbaEntries=(::1/128)	# Assumes that all the coordinator (master/slave) accepts
										# the same connection
										# This list sets up pg_hba.conf for $pgxcOwner user.
										# If you'd like to setup other entries, supply them
										# through extra configuration files specified below.
datanodePgHbaEntries=(192.168.1.103/32 192.168.1.120/32 192.168.1.121/32)	# Same as above but for IPv4 connections

#---- Master ----------------
datanodeMasterServers=(192.168.1.103)	# none means this master is not available.
													# This means that there should be the master but is down.
													# The cluster is not operational until the master is
													# recovered and ready to run.	
datanodeMasterDirs=($datanodeMasterDir)
datanodeMaxWalSender=5								# max_wal_senders: needed to configure slave. If zero value is 
													# specified, it is expected this parameter is explicitly supplied
													# by external configuration files.
													# If you don't configure slaves, leave this value zero.
datanodeMaxWALSenders=($datanodeMaxWalSender)
						# max_wal_senders configuration for each datanode

#---- Slave -----------------
datanodeSlave=y			# Specify y if you configure at least one coordiantor slave.  Otherwise, the following
						# configuration parameters will be set to empty values.
						# If no effective server names are found (that is, every servers are specified as none),
						# then datanodeSlave value will be set to n and all the following values will be set to
						# empty values.
datanodeSlaveServers=(192.168.1.120)	# value none means this slave is not available
datanodeSlavePorts=(10001)	# Master and slave use the same port!
datanodeSlavePoolerPorts=(10011)	# Master and slave use the same port!
datanodeSlaveSync=y		# If datanode slave is connected in synchronized mode
datanodeSlaveDirs=($datanodeSlaveDir)
datanodeArchLogDirs=( $datanodeArchLogDir)
datanodeRepNum=2  #  no HA setting 0, streaming HA and active-active logcial replication setting 1 replication,  paxos HA setting 2 replication.	
datanodeSlaveType=(3) # 1 is streaming HA, 2 is active-active logcial replication, 3 paxos HA.

#---- Learner -----------------
datanodeLearnerServers=(192.168.1.121)	# value none means this learner is not available
datanodeLearnerPorts=(10001)	# learner port!
#datanodeSlavePoolerPorts=(11011)	# learner pooler port!
datanodeLearnerSync=y		# If datanode learner is connected in synchronized mode
datanodeLearnerDirs=($datanodeLearnerDir)

# ---- Configuration files ---
# You may supply your bash script to setup extra config lines and extra pg_hba.conf entries here.
# These files will go to corresponding files for the master.
# Or you may supply these files manually.
datanodeExtraConfig=datanodeExtraConfig	
cat > $datanodeExtraConfig <<EOF
#================================================
# Added to all the datanode postgresql.conf
# Original: $datanodeExtraConfig
log_destination = 'csvlog'  
unix_socket_directories = '., /tmp'  
logging_collector = on  
log_directory = 'log'  
listen_addresses = '0.0.0.0'  
max_connections = 100  
hot_standby = on  
synchronous_commit = on  
max_worker_processes = 30  
cron.database_name = 'postgres'  
tcp_keepalives_idle = 30  
tcp_keepalives_interval = 10    
tcp_keepalives_count = 6                           
shared_buffers = 128MB  
maintenance_work_mem = 16MB  
bgwriter_delay = 10ms  
bgwriter_lru_maxpages = 1000  
bgwriter_lru_multiplier = 5.0  
effective_io_concurrency = 0  
parallel_leader_participation = off  
max_wal_size = 2GB  
min_wal_size = 1GB  
wal_keep_segments = 4096 
wal_sender_timeout = 5s  
random_page_cost = 1.1  
effective_cache_size = 1GB  
log_truncate_on_rotation = on  
log_min_duration_statement = 3s  
log_checkpoints = on  
log_lock_waits = on  
log_statement = 'ddl'  
log_autovacuum_min_duration = 0  
autovacuum_freeze_max_age = 800000000  
autovacuum_multixact_freeze_max_age = 900000000  
autovacuum_vacuum_cost_delay = 0ms  
vacuum_freeze_min_age = 700000000  
vacuum_freeze_table_age = 850000000  
vacuum_multixact_freeze_min_age = 700000000  
vacuum_multixact_freeze_table_age = 850000000  
statement_timeout = 0                    # in milliseconds, 0 is disabled  
lock_timeout = 0                         # in milliseconds, 0 is disabled  
idle_in_transaction_session_timeout = 0  # in milliseconds, 0 is disabled  
shared_preload_libraries = 'pg_cron'  
max_parallel_replay_workers = 0
EOF
# Additional Configuration file for specific datanode master.
# You can define each setting by similar means as above.
datanodeSpecificExtraConfig=(none)
datanodeSpecificExtraPgHba=(none)

9、初始化三节点集群
master node:

pgxc_ctl -c $POLARDBHOME/etc/polardb_paxos.conf clean all  
pgxc_ctl -c $POLARDBHOME/etc/polardb_paxos.conf init all