PostgreSQL 15 preview - Add support for LZ4 compression in pg_receivewal --compression-method=lz4

作者

digoal

日期

2021-11-07

标签

PostgreSQL , 实时归档 , pg_receivewal , 压缩

https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=babbbb595d2322da095a1e6703171b3f1f2815cb

Add support for LZ4 compression in pg_receivewal  
author  Michael Paquier <michael@paquier.xyz>     
Fri, 5 Nov 2021 02:33:25 +0000 (11:33 +0900)  
committer   Michael Paquier <michael@paquier.xyz>     
Fri, 5 Nov 2021 02:33:25 +0000 (11:33 +0900)  
commit  babbbb595d2322da095a1e6703171b3f1f2815cb  
tree    2822626ff97ea4f4f88f6776354e79e40a07925f    tree  
parent  5cd7eb1f1c32e1b95894f28b277b4e4b89add772    commit | diff  
Add support for LZ4 compression in pg_receivewal  
pg_receivewal gains a new option, --compression-method=lz4, available  
when the code is compiled with --with-lz4.  Similarly to gzip, this  
gives the possibility to compress archived WAL segments with LZ4.  This  
option is not compatible with --compress.  
The implementation uses LZ4 frames, and is compatible with simple lz4  
commands.  Like gzip, using --synchronous ensures that any data will be  
flushed to disk within the current .partial segment, so as it is  
possible to retrieve as much WAL data as possible even from a  
non-completed segment (this requires completing the partial file with  
zeros up to the WAL segment size supported by the backend after  
decompression, but this is the same as gzip).  
The calculation of the streaming start LSN is able to transparently find  
and check LZ4-compressed segments.  Contrary to gzip where the  
uncompressed size is directly stored in the object read, the LZ4 chunk  
protocol does not store the uncompressed data by default.  There is  
contentSize that can be used with LZ4 frames by that would not help if  
using an archive that includes segments compressed with the defaults of  
a "lz4" command, where this is not stored.  So, this commit has taken  
the most extensible approach by decompressing the already-archived  
segment to check its uncompressed size, through a blank output buffer in  
chunks of 64kB (no actual performance difference noticed with 8kB, 16kB  
or 32kB, and the operation in itself is actually fast).  
Tests have been added to verify the creation and correctness of the  
generated LZ4 files.  The latter is achieved by the use of command  
"lz4", if found in the environment.  
The tar-based WAL method in walmethods.c, used now only by  
pg_basebackup, does not know yet about LZ4.  Its code could be extended  
for this purpose.  
Author: Georgios Kokolatos  
Reviewed-by: Michael Paquier, Jian Guo, Magnus Hagander, Dilip Kumar  
Discussion: https://postgr.es/m/ZCm1J5vfyQ2E6dYvXz8si39HQ2gwxSZ3IpYaVgYa3lUwY88SLapx9EEnOf5uEwrddhx2twG7zYKjVeuP5MwZXCNPybtsGouDsAD1o2L_I5E=@pm.me  
复制

 266      <varlistentry>  
 267       <term><option>--compression-method=<replaceable class="parameter">method</replaceable></option></term>  
 268       <listitem>  
 269        <para>  
 270         Enables compression of write-ahead logs using the specified method.  
 271         Supported values <literal>gzip</literal>, <literal>lz4</literal>  
 272         (if <productname>PostgreSQL</productname> was compiled with  
 273         <option>--with-lz4</option>), and <literal>none</literal>.  
 274        </para>  
 275   
 276        <para>  
 277         The suffix <filename>.gz</filename> will automatically be added to  
 278         all filenames when using <literal>gzip</literal>, and the suffix  
 279         <filename>.lz4</filename> is added when using <literal>lz4</literal>.  
 280        </para>  
 281       </listitem>  
 282      </varlistentry>  
复制

期望 PostgreSQL 增加什么功能?

类似Oracle RAC架构的PostgreSQL已开源: 阿里云PolarDB for PostgreSQL云原生分布式开源数据库!

PostgreSQL 解决方案集合

德哥 / digoal's github - 公益是一辈子的事.