HP-UX安装Oracle 11.2.0.3单机执行任何命令均报错Memory Fault(coredump)分析解决

1. 故障现象

顺利无报错安装Oracle软件之后，执行netca、dbca等任何Oracle相关命令均报错Memory fault(coredump)

$ sqlplus / as sysdba
Memory fault(coredump)

$ expdp
Memory fault(coredump)

$ rman target /
Memory fault(coredump)

复制

2. 分析

2.1 检查参数

检查操作系统参数以及ulimit限制，发现该报错与参数与ulimit无关。

2.2 更换Oracle版本

通过安装11.2.0.4版本，报错依旧，依然是可以正常安装软件，执行命令报相同错误。

2.3 tusc追踪

tusc -Dfaepo /tmp/sqlplus.out -T %T sqlplus / as sysdba
# 节选部分关键内容
# 在判断当前时间以及访问时区文件之后就开始mmap分配内存
gettimeofday(0x9fffffffffffc6b0, NULL) ....................................................... = 0
stat("/usr/lib/tztab", 0x9fffffffffffc5d0) ................................................... = 0
open("/usr/lib/tztab", O_RDONLY|0x800, 01210) ................................................ = 9
mmap(NULL, 18261, PROT_READ, MAP_PRIVATE, 9, 0) .............................................. = 0x9fffffff7f568000
close(9) ..................................................................................... = 0
gettimeofday(0x9fffffffffffc6b0, NULL) ....................................................... = 0
gettimeofday(0x9fffffffffffc580, NULL) ....................................................... = 0
lseek(8, 1536, SEEK_SET) ..................................................................... = 1536
read(8, "\0\bbb81\0\0\08 bb82\0\0\086bb83".., 512) ........................................... = 512
mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) .................... = 0x9fffffff7f564000
open("/oracle/app/oracle/product/11.2.0/dbhome_1/lib/libclntsh.so.11.1", O_RDONLY|0x800, 0163060) = 9
# 分配内存
mmap(NULL, 8192, PROT_READ, MAP_PRIVATE, 9, 121794560) ....................................... = 0x9fffffff7f562000
# mmap分配之后其实检测信号掩码之后就开始取消分配内存munmap
sigprocmask(SIG_SETMASK, 0x9ffffffffffd2560, 0x9ffffffffffd2580) ............................. = 0
sigprocmask(SIG_SETMASK, 0x9ffffffffffd2510, 0x9ffffffffffd2530) ............................. = 0
sigprocmask(SIG_SETMASK, 0x9ffffffffffd2530, 0x9ffffffffffd2510) ............................. = 0
sigsetreturn(NULL, 0x6211988, 48640) ......................................................... = 0
sigprocmask(SIG_SETMASK, 0x9ffffffffffd2510, 0x9ffffffffffd2530) ............................. = 0
sigprocmask(SIG_SETMASK, 0x9ffffffffffd2530, 0x9ffffffffffd2510) ............................. = 0
sigaction(SIGBUS, NULL, 0x9fffffff7f7d42b0) .................................................. = 0
# 省略相似输出
sigaction(SIGSEGV, 0x9fffffff7f7d42e0, NULL) ................................................. = 0
sigprocmask(SIG_SETMASK, 0x9ffffffffffd2810, NULL) ........................................... = 0
munmap(0x9fffffff7f330000, 28672) ............................................................ = 0
munmap(0x9fffffff7f338000, 36864) ............................................................ = 0
munmap(0x9fffffff7f344000, 8192) ............................................................. = 0
close(12) .................................................................................... = 0
munmap(0x9fffffff7f346000, 8192) ............................................................. = 0
munmap(0x9fffffff7f560000, 8192) ............................................................. = 0
close(11) .................................................................................... = 0
munmap(0x9fffffff7f348000, 16384) ............................................................ = 0
munmap(0x9fffffff7f34c000, 49152) ............................................................ = 0
munmap(0x9fffffff7f567000, 4096) ............................................................. = 0
close(10) .................................................................................... = 0
munmap(0x9fffffff7f358000, 733184) ........................................................... = 0
munmap(0x9fffffff7f40c000, 1388544) .......................................................... = 0
munmap(0x9fffffff7f562000, 8192) ............................................................. = 0
close(9) ..................................................................................... = 0
  Received signal 11, SIGSEGV, in user mode, [SIG_DFL], partial siginfo
    Siginfo: si_code: SEGV_ACCERR, faulting address: 0x1, si_errno: 0
     PC: 00000001000000a0.0	     	break.m 0x16000 
exit(11) [implicit] .......................................................................... WIFSIGNALED(SIGSEGV)|WCOREDUMP

# 从上面可以看出，在判断了当前系统时间之后，就开始分配内存，之后就取消分配内存，之后就报错了，生成了core文件。 
复制

3. 解决

分析到2.3，去检查了一下系统时间，基本就已经确定问题了，系统当前时间不正确，使用date命令确认一下如下：

# date
Thu Jul   2 11:21:05 MDT 1970

# 修改系统时间为正确时间，问题解决。
复制