PostgreSQL 15 preview - log destination 即将支持jsonlog格式

作者

digoal

日期

2022-01-12

标签

PostgreSQL , log , csvlog , jsonlog

https://www.postgresql.org/docs/devel/runtime-config-logging.html#RUNTIME-CONFIG-LOGGING-WHERE

Including jsonlog in the log_destination list provides a convenient way to import log files into many different programs. This option emits log lines in (JSON) format.
String fields with null values are excluded from output. Additional fields may be added in the future. User applications that process jsonlog output should ignore unknown fields.
Each log line is serialized as a JSON object as of the following set of keys with their values.
复制

https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=dc686681e0799b12c40f44f85fc5bfd7fed4e57f

Introduce log_destination=jsonlog
author  Michael Paquier <michael@paquier.xyz>   
Mon, 17 Jan 2022 01:16:53 +0000 (10:16 +0900)
committer   Michael Paquier <michael@paquier.xyz>   
Mon, 17 Jan 2022 01:16:53 +0000 (10:16 +0900)
commit  dc686681e0799b12c40f44f85fc5bfd7fed4e57f
tree    66654ff9d34f4153810245c2152a6d2ddfbeb761    tree
parent  6478896675660402171c97d6307e6e1519250025    commit | diff
Introduce log_destination=jsonlog
"jsonlog" is a new value that can be added to log_destination to provide
logs in the JSON format, with its output written to a file, making it
the third type of destination of this kind, after "stderr" and
"csvlog".  The format is convenient to feed logs to other applications.
There is also a plugin external to core that provided this feature using
the hook in elog.c, but this had to overwrite the output of "stderr" to
work, so being able to do both at the same time was not possible.  The
files generated by this log format are suffixed with ".json", and use
the same rotation policies as the other two formats depending on the
backend configuration.
This takes advantage of the refactoring work done previously in ac7c807,
bed6ed3, 8b76f89 and 2d77d83 for the backend parts, and 72b76f7 for the
TAP tests, making the addition of any new file-based format rather
straight-forward.
The documentation is updated to list all the keys and the values that
can exist in this new format.  pg_current_logfile() also required a
refresh for the new option.
Author: Sehrope Sarkuni, Michael Paquier
Reviewed-by: Nathan Bossart, Justin Pryzby
Discussion: https://postgr.es/m/CAH7T-aqswBM6JWe4pDehi1uOiufqe06DJWaU5=X7dDLyqUExHg@mail.gmail.com
复制

https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=ac7c80758a7ad6369aefce7406f08c689144ec4b

Refactor set of routines specific to elog.c  
author  Michael Paquier <michael@paquier.xyz>     
Wed, 12 Jan 2022 05:16:59 +0000 (14:16 +0900)  
committer   Michael Paquier <michael@paquier.xyz>     
Wed, 12 Jan 2022 05:16:59 +0000 (14:16 +0900)  
commit  ac7c80758a7ad6369aefce7406f08c689144ec4b  
tree    2a42b18e86912ceb5137eaf6033bf443b3a37bc9    tree  
parent  9a3d8e1886670ca3532687c5336d97965a79a084    commit | diff  
Refactor set of routines specific to elog.c  
This refactors the following routines and facilities coming from  
elog.c, to ease their use across multiple log destinations:  
- Start timestamp, including its reset, to store when a process has been  
started.  
- The log timestamp, associated to an entry (the same timestamp is used  
when logging across multiple destinations).  
- Routine deciding if a query can be logged or not.  
- The backend type names, depending on the process that logs any  
information (postmaster, bgworker name or just GetBackendTypeDesc() with  
a regular backend).  
- Write of logs using the logging piped protocol, with the log collector  
enabled.  
- Error severity converted to a string.  
These refactored routines will be used for some follow-up changes  
to move all the csvlog logic into its own file and to potentially add  
JSON as log destination, reducing the overall size of elog.c as the end  
result.  
Author: Michael Paquier, Sehrope Sarkuni  
Reviewed-by: Nathan Bossart  
Discussion: https://postgr.es/m/CAH7T-aqswBM6JWe4pDehi1uOiufqe06DJWaU5=X7dDLyqUExHg@mail.gmail.com  
复制

Hi,  
This patch adds a new log_destination, "jsonlog", that writes log entries  
as lines of JSON. It was originally started by David Fetter using  
the jsonlog module by Michael Paquier (  
https://github.com/michaelpq/pg_plugins/blob/master/jsonlog/jsonlog.c) as a  
basis for how to serialize the log messages. Thanks to both of them because  
this wouldn't be possible without that starting point.  
The first commit splits out the destination in log pipe messages into its  
own field. Previously it would piggyback on the "is_last" field. This adds  
an int to the message size but makes the rest of the code easier to follow.  
The second commit adds a TAP test for log_destination "csvlog". This was  
done to both confirm that the previous change didn't break anything and as  
a skeleton for the test in the next commit.  
The third commit adds the new log_destination "jsonlog". The output format  
is one line per entry with the top level output being a JSON object keyed  
with the log fields. Newlines in the output fields are escaped as \n so the  
output file has exactly one line per log entry. It also includes a new test  
for verifying the JSON output with some basic regex checks (similar to the  
csvlog test).  
Here's a sample of what the log entries look like:  
{"timestamp":"2021-08-31 10:15:25.129  
EDT","user":"sehrope","dbname":"postgres","pid":12012,"remote_host":"[local]","session_id":"612e397d.2eec","line_num":1,"ps":"idle","session_start":"2021-08-31  
10:15:25  
EDT","vxid":"3/2","txid":"0","error_severity":"LOG","application_name":"  
006_jsonlog.pl","message":"statement: SELECT 1/0"}  
It builds and passes "make check-world" on Linux. It also includes code to  
handle Windows as well but I have not actually tried building it there.  
Regards,  
-- Sehrope Sarkuni  
Founder & CEO | JackDB, Inc. | https://www.jackdb.com/  
复制

期望 PostgreSQL 增加什么功能?

PolarDB for PostgreSQL云原生分布式开源数据库

PostgreSQL 解决方案集合

德哥 / digoal's github - 公益是一辈子的事.