暂无图片
暂无图片
暂无图片
暂无图片
暂无图片

[数据库-ORACLE]oracle 11.2.0.3系统表空间用尽处理过程

半个IT人 2017-08-20
590

问题描述: 

      某客户反馈在登录数据库时提示如下错误: 
1、通过用户test登录: 
[oracle@test ~]$ sqlplus test/test@test
SQL*Plus: Release 11.2.0.3.0 Production on Thu Aug 3 19:45:35 2017 

Copyright (c) 1982, 2011, Oracle. All rights reserved. 

Copyright (c) 1982, 2011, Oracle. All rights reserved. 

ERROR: 
ORA-00604: error occurred at recursive SQL level 1 
ORA-01653: unable to extend table SYS.AUD$ by 8192 in tablespace SYSTEM 
ORA-02002: error while writing to audit trail 
ORA-00604: error occurred at recursive SQL level 1 
ORA-01653: unable to extend table SYS.AUD$ by 8192 in tablespace SYSTEM 

Enter user-name: 
ERROR: 
ORA-01017: invalid username/password; logon denied 

Enter user-name: 
ERROR: 
ORA-01017: invalid username/password; logon denied 

SP2-0157: unable to CONNECT to ORACLE after 3 attempts, exiting SQL*Plus 

2、通过sys用户登录: 
[oracle@test ~]$ sqlplus as sysdba 
SQL*Plus: Release 11.2.0.3.0 Production on Thu Aug 3 19:18:47 2017 

Copyright (c) 1982, 2011, Oracle. All rights reserved. 

ERROR: 
ORA-00020: maximum number of processes (1000) exceeded 

Enter user-name: 
ERROR: 
ORA-01017: invalid username/password; logon denied 

Enter user-name: 
ERROR: 
ORA-01017: invalid username/password; logon denied 

SP2-0157: unable to CONNECT to ORACLE after 3 attempts, exiting SQL*Plus 
现场环境: 

      现场为linux操作系统:RHEL 6.5 64 bit,数据库为:oracle 11.2.0.3单机版;

处理方法:

      尽管从现场反馈的第一个描述中就能够得知问题原因,我们仍然需要分析一下问题:
1、查看监听日志:
listener.ora中有大里的日志: 
21-FEB-2016 22:06:45 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=oracle))(SERVICE_NAME=testdb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=192.0.2.4)(PORT=25375)) * establish * testdb * 12516 
TNS-12516: TNS:listener could not find available handler with matching protocol stack 
21-FEB-2016 22:06:45 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=oracle))(SERVICE_NAME=testdb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=192.0.2.4)(PORT=25377)) * establish * testdb * 12516 
TNS-12516: TNS:listener could not find available handler with matching protocol stack 
21-FEB-2016 22:06:45 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=oracle))(SERVICE_NAME=testdb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=192.0.2.4)(PORT=25378)) * establish * testdb * 12516 
TNS-12516: TNS:listener could not find available handler with matching protocol stack 
21-FEB-2016 22:06:51 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=oracle))(SERVICE_NAME=testdb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=192.0.2.4)(PORT=25385)) * establish * testdb * 12516 
TNS-12516: TNS:listener could not find available handler with matching protocol stack 
Sun Feb 21 22:07:03 2016 
21-FEB-2016 22:07:03 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=oracle))(SERVICE_NAME=testdb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=192.0.2.4)(PORT=25397)) * establish * testdb * 12516 
TNS-12516: TNS:listener could not find available handler with matching protocol stack 
Sun Feb 21 22:07:15 2016 
21-FEB-2016 22:07:15 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=oracle))(SERVICE_NAME=testdb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=192.0.2.4)(PORT=25412)) * establish * testdb * 12516 
TNS-12516: TNS:listener could not find available handler with matching protocol stack 
21-FEB-2016 22:07:15 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=oracle))(SERVICE_NAME=testdb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=192.0.2.4)(PORT=25413)) * establish * testdb * 12516 
TNS-12516: TNS:listener could not find available handler with matching protocol stack 
21-FEB-2016 22:07:15 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=oracle))(SERVICE_NAME=testdb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=192.0.2.4)(PORT=25415)) * establish * testdb * 12516 
TNS-12516: TNS:listener could not find available handler with matching protocol stack 

这表明客户端在进行连接时,由于无法正常连接数据库,其在不停地进行尝试连接,通过现场描述的第二条也可以看出,数据库的会话数据已被占满;

我们也可以查看一下系统上的tcp连接:

[oracle@test ~]$ ps -ef|grep LOCAL |wc -l 
975 

可以看到有975个tcp连接,再加上数据库自身的后台进程,实际上已经超过了数据库的process值:1000;


2、查看数据库日志:
alert日志中存在的如下信息更能说明问题,system表空间满了: 
Thu Aug 03 19:50:44 2017 
ORA-1653: unable to extend table SYS.AUD$ by 8192 in tablespace SYSTEM 
Thu Aug 03 19:50:44 2017 
ORA-1653: unable to extend table SYS.AUD$ by 8192 in tablespace SYSTEM 
Thu Aug 03 19:50:49 2017 
ORA-1653: unable to extend table SYS.AUD$ by 8192 in tablespace SYSTEM 
Thu Aug 03 19:50:52 2017 
ORA-1653: unable to extend table SYS.AUD$ by 8192 in tablespace SYSTEM 

3、通过手工kill掉部分tcp连接,可以顺利登录数据库,此时查看表空间使用率:


可以看到确实SYSTEM表空间已达到99.39%,而根据现场反馈1中得知,数据库未关闭审计,系统运行时间一长,审计日志占满了SYSTEM表空间;


4、问题解决:

找到问题原因,问题也就迎刃而解了,关闭数据库的审计,同时删除审计表即可,再操作重启数据库实例前,记得要增加一些SYSTEM表空间大小,以免实例无法启动;


引申:

我们在进行项目初期总成,亦或者在现场需要对数据库服务端进行重装时,一定要进行检查,避免未进行审计关闭或者密码策略修改等配置,从而增加了后期不必要的维护量;

切记,切记,切记!!!(重要的事情说三遍


文章转载自半个IT人,如果涉嫌侵权,请发送邮件至:contact@modb.pro进行举报,并提供相关证据,一经查实,墨天轮将立刻删除相关内容。

评论