暂无图片
暂无图片
暂无图片
暂无图片
暂无图片

0071.O 1.5.2oceanbase日志循环配置-日志空间满处理过程

rundba 2021-07-22
721

oceanbase日常使用中,默认情况下日志产生较快,日志目录较小时,如果不做任何处理,将会很快导致日志目录撑满,从而导致服务异常。

0.ENV

oceanbase企业版、oceanbase社区版

当前采用单机3个ob server部署,生成环境不建议该部署方式。

1. 现象

oceanbase数据库主机,/home目录使用率100%。

    [admin@ob3 ~]$ df -Th home
    Filesystem Type Size Used Avail Use% Mounted on
    /dev/mapper/centos-home ext4 30G 30G 0G 100% home
    复制

    2. 问题分析

    当前测试环境,因为我们实验环境留给log的空间比较小,/home共30G,通过目录大小查看,observer3 log目录使用13G,占用空间较大。

      [admin@ob3 ~]$ du -sh *
      4.0K 1host-3ob-cluster.yaml
      28G oceanbase
      [admin@ob3 ~]$ cd oceanbase/
      [admin@ob3 oceanbase]$ ls
      1 2 3
      [admin@ob3 oceanbase]$ du -sh *
      7.9G 1
      7.5G 2
      13G 3
      [admin@ob3 oceanbase]$ du -sh 3/*
      4.0K 3/admin
      12K 3/etc
      12K 3/etc2
      12K 3/etc3
      13G 3/log
      8.0K 3/run
      0 3/store
      复制

      3. 问题处理

      以sys租户登录,启动日志文件循环,并且只保留10个日志文件。

      打开日志循环,设置为true。该值默认为false,即关闭日志循环。

        obclient> alter system set enable_syslog_recycle=true;
        Query OK, 0 rows affected (0.09 sec)
        复制

        查看日志循环已经打开,value一栏3个zone均为True。

          MySQL [(none)]> show parameters like '%enable_syslog_recycle%';  
          +-------+----------+-----------+----------+-----------------------+-----------+-------+---------------------------------------------------------------------------------------------+----------+---------+---------+-------------------+
          | zone | svr_type | svr_ip | svr_port | name | data_type | value | info | section | scope | source | edit_level |
          +-------+----------+-----------+----------+-----------------------+-----------+-------+---------------------------------------------------------------------------------------------+----------+---------+---------+-------------------+
          | zone3 | observer | 127.0.0.1 | 4882 | enable_syslog_recycle | NULL | True | specifies whether log file recycling is turned on. Value: True:turned on; False: turned off | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
          | zone2 | observer | 127.0.0.1 | 3882 | enable_syslog_recycle | NULL | True | specifies whether log file recycling is turned on. Value: True:turned on; False: turned off | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
          | zone1 | observer | 127.0.0.1 | 2882 | enable_syslog_recycle | NULL | True | specifies whether log file recycling is turned on. Value: True:turned on; False: turned off | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
          +-------+----------+-----------+----------+-----------------------+-----------+-------+---------------------------------------------------------------------------------------------+----------+---------+---------+-------------------+
          3 rows in set (0.016 sec)
          复制

          设置日志数目为10,即只保留10个日志文件,1个日志文件最大为256M。

            obclient> alter system set max_syslog_file_count=10;
            Query OK, 0 rows affected (0.08 sec)
            复制

            查看日志数目为10,value一栏3个zone均为10。

              MySQL [(none)]> show parameters like '%max_syslog_file_count%';
              +-------+----------+-----------+----------+-----------------------+-----------+-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+---------+---------+-------------------+
              | zone | svr_type | svr_ip | svr_port | name | data_type | value | info | section | scope | source | edit_level |
              +-------+----------+-----------+----------+-----------------------+-----------+-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+---------+---------+-------------------+
              | zone2 | observer | 127.0.0.1 | 3882 | max_syslog_file_count | NULL | 10 | specifies the maximum number of the log files that can co-exist before the log file recycling kicks in. Each log file can occupy at most 256MB disk space. When this value is set to 0, no log file will be removed. Range: [0, +∞) in integer | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
              | zone3 | observer | 127.0.0.1 | 4882 | max_syslog_file_count | NULL | 10 | specifies the maximum number of the log files that can co-exist before the log file recycling kicks in. Each log file can occupy at most 256MB disk space. When this value is set to 0, no log file will be removed. Range: [0, +∞) in integer | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
              | zone1 | observer | 127.0.0.1 | 2882 | max_syslog_file_count | NULL | 10 | specifies the maximum number of the log files that can co-exist before the log file recycling kicks in. Each log file can occupy at most 256MB disk space. When this value is set to 0, no log file will be removed. Range: [0, +∞) in integer | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |
              +-------+----------+-----------+----------+-----------------------+-----------+-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+---------+---------+-------------------+
              3 rows in set (0.016 sec)
              复制

              设置完成后,日志会自动被清理,此时空间会被释放。

                [admin@ob3 ~]$ df -Th home
                Filesystem Type Size Used Avail Use% Mounted on
                /dev/mapper/centos-home ext4 30G 16G 13G 56% home
                复制

                查看observer日志:

                  [admin@ob3 ~]$ cd oceanbase/3/log
                  [admin@ob3 log]$ du -sh *
                  154M election.log
                  257M election.log.20210717135834
                  257M election.log.20210717142732
                  129M election.log.wf
                  196M election.log.wf.20210717135834
                  225M election.log.wf.20210717142732
                  210M observer.log
                  257M observer.log.20210717094420
                  257M observer.log.20210717103300
                  257M observer.log.20210717111741
                  257M observer.log.20210717121149
                  257M observer.log.20210717125650
                  257M observer.log.20210717132204
                  257M observer.log.20210717135635
                  257M observer.log.20210717140656
                  257M observer.log.20210717142621
                  257M observer.log.20210717143958
                  50M observer.log.wf
                  11M observer.log.wf.20210717094420
                  11M observer.log.wf.20210717103300
                  21M observer.log.wf.20210717111741
                  13M observer.log.wf.20210717121149
                  38M observer.log.wf.20210717125650
                  55M observer.log.wf.20210717132204
                  45M observer.log.wf.20210717135635
                  92M observer.log.wf.20210717140656
                  68M observer.log.wf.20210717142621
                  78M observer.log.wf.20210717143958
                  131M rootservice.log
                  257M rootservice.log.20210716232113
                  257M rootservice.log.20210717034416
                  257M rootservice.log.20210717075924
                  257M rootservice.log.20210717120926
                  257M rootservice.log.20210717142034
                  94M rootservice.log.wf
                  7.7M rootservice.log.wf.20210716232113
                  1.9M rootservice.log.wf.20210717034416
                  2.3M rootservice.log.wf.20210717075924
                  9.4M rootservice.log.wf.20210717120926
                  135M rootservice.log.wf.20210717142034
                  复制


                  4. 小结

                  log增长过快而导致日志目录/home撑满,通过启动日志文件循环,并且只保留10个日志文件,自动清理日志,建议生产环境安装完成后,设置参数,消除隐患。也可设置日志级别为ERROR和WARN,减少日志输出,本文未做描述。


                  更多oceanbase日志参考:

                  https://www.oceanbase.com/docs/oceanbase-database/oceanbase-database-1-4/V1.4/oblog


                  -  完 -

                  不足之处,还望抛砖。

                  作者:王坤,微信公众号:rundba,欢迎转载,转载请注明出处。

                  如需公众号转发,请联系wx: landnow。


                  文章转载自rundba,如果涉嫌侵权,请发送邮件至:contact@modb.pro进行举报,并提供相关证据,一经查实,墨天轮将立刻删除相关内容。

                  评论