问题描述:
某客户反馈在登录数据库时提示如下错误:
1、通过用户test登录:
[oracle@test ~]$ sqlplus test/test@test
SQL*Plus: Release 11.2.0.3.0 Production on Thu Aug 3 19:45:35 2017
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Copyright (c) 1982, 2011, Oracle. All rights reserved.
ERROR:
ORA-00604: error occurred at recursive SQL level 1
ORA-01653: unable to extend table SYS.AUD$ by 8192 in tablespace SYSTEM
ORA-02002: error while writing to audit trail
ORA-00604: error occurred at recursive SQL level 1
ORA-01653: unable to extend table SYS.AUD$ by 8192 in tablespace SYSTEM
Enter user-name:
ERROR:
ORA-01017: invalid username/password; logon denied
Enter user-name:
ERROR:
ORA-01017: invalid username/password; logon denied
SP2-0157: unable to CONNECT to ORACLE after 3 attempts, exiting SQL*Plus
2、通过sys用户登录:
[oracle@test ~]$ sqlplus as sysdba
SQL*Plus: Release 11.2.0.3.0 Production on Thu Aug 3 19:18:47 2017
Copyright (c) 1982, 2011, Oracle. All rights reserved.
ERROR:
ORA-00020: maximum number of processes (1000) exceeded
Enter user-name:
ERROR:
ORA-01017: invalid username/password; logon denied
Enter user-name:
ERROR:
ORA-01017: invalid username/password; logon denied
SP2-0157: unable to CONNECT to ORACLE after 3 attempts, exiting SQL*Plus
现场环境:
现场为linux操作系统:RHEL 6.5 64 bit,数据库为:oracle 11.2.0.3单机版;
处理方法:
尽管从现场反馈的第一个描述中就能够得知问题原因,我们仍然需要分析一下问题:
1、查看监听日志:
listener.ora中有大里的日志:
21-FEB-2016 22:06:45 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=oracle))(SERVICE_NAME=testdb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=192.0.2.4)(PORT=25375)) * establish * testdb * 12516
TNS-12516: TNS:listener could not find available handler with matching protocol stack
21-FEB-2016 22:06:45 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=oracle))(SERVICE_NAME=testdb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=192.0.2.4)(PORT=25377)) * establish * testdb * 12516
TNS-12516: TNS:listener could not find available handler with matching protocol stack
21-FEB-2016 22:06:45 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=oracle))(SERVICE_NAME=testdb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=192.0.2.4)(PORT=25378)) * establish * testdb * 12516
TNS-12516: TNS:listener could not find available handler with matching protocol stack
21-FEB-2016 22:06:51 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=oracle))(SERVICE_NAME=testdb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=192.0.2.4)(PORT=25385)) * establish * testdb * 12516
TNS-12516: TNS:listener could not find available handler with matching protocol stack
Sun Feb 21 22:07:03 2016
21-FEB-2016 22:07:03 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=oracle))(SERVICE_NAME=testdb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=192.0.2.4)(PORT=25397)) * establish * testdb * 12516
TNS-12516: TNS:listener could not find available handler with matching protocol stack
Sun Feb 21 22:07:15 2016
21-FEB-2016 22:07:15 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=oracle))(SERVICE_NAME=testdb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=192.0.2.4)(PORT=25412)) * establish * testdb * 12516
TNS-12516: TNS:listener could not find available handler with matching protocol stack
21-FEB-2016 22:07:15 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=oracle))(SERVICE_NAME=testdb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=192.0.2.4)(PORT=25413)) * establish * testdb * 12516
TNS-12516: TNS:listener could not find available handler with matching protocol stack
21-FEB-2016 22:07:15 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=oracle))(SERVICE_NAME=testdb)) * (ADDRESS=(PROTOCOL=tcp)(HOST=192.0.2.4)(PORT=25415)) * establish * testdb * 12516
TNS-12516: TNS:listener could not find available handler with matching protocol stack
这表明客户端在进行连接时,由于无法正常连接数据库,其在不停地进行尝试连接,通过现场描述的第二条也可以看出,数据库的会话数据已被占满;
我们也可以查看一下系统上的tcp连接:
[oracle@test ~]$ ps -ef|grep LOCAL |wc -l
975
可以看到有975个tcp连接,再加上数据库自身的后台进程,实际上已经超过了数据库的process值:1000;
2、查看数据库日志:
alert日志中存在的如下信息更能说明问题,system表空间满了:
Thu Aug 03 19:50:44 2017
ORA-1653: unable to extend table SYS.AUD$ by 8192 in tablespace SYSTEM
Thu Aug 03 19:50:44 2017
ORA-1653: unable to extend table SYS.AUD$ by 8192 in tablespace SYSTEM
Thu Aug 03 19:50:49 2017
ORA-1653: unable to extend table SYS.AUD$ by 8192 in tablespace SYSTEM
Thu Aug 03 19:50:52 2017
ORA-1653: unable to extend table SYS.AUD$ by 8192 in tablespace SYSTEM
3、通过手工kill掉部分tcp连接,可以顺利登录数据库,此时查看表空间使用率:
可以看到确实SYSTEM表空间已达到99.39%,而根据现场反馈1中得知,数据库未关闭审计,系统运行时间一长,审计日志占满了SYSTEM表空间;
4、问题解决:
找到问题原因,问题也就迎刃而解了,关闭数据库的审计,同时删除审计表即可,再操作重启数据库实例前,记得要增加一些SYSTEM表空间大小,以免实例无法启动;
引申:
我们在进行项目初期总成,亦或者在现场需要对数据库服务端进行重装时,一定要进行检查,避免未进行审计关闭或者密码策略修改等配置,从而增加了后期不必要的维护量;
切记,切记,切记!!!(重要的事情说三遍)