CV备份时ORA-15028报错处理

基础环境
操作系统：AIX 6.1
数据库：Oracle RAC 11.2.0.4
备份软件：COMMVAULT 10.0.0(BUILD116)
问题概述
cv备份时报ORA-15028

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of delete command on default channel at 12/04/2023 08:27:40
ORA-15028: ASM file '+ARCH/xfdb/archivelog/2023_12_01/thread_1_seq_196506.1168.1154465965' not dropped; currently being accessed
RMAN> 
Recovery Manager complete.

归档日志无法正常删除，可能会归档目录空间增长或空间不足，引发数据库故障。
问题原因
OGG大事务或长事务在使用该归档文件导致备份时无法删除该归档文件。Oracle数据库部署了OGG，DSG等同步产品如果有大事务，长事务或者同步进程异常时，会导致归档文件无法删除。
解决方案

1、kill长事务

SQL> select inst_id,addr,start_time,start_scnb,start_date,start_scn from gv$transaction
 where START_date<=to_date('2023-12-04 12:00:00','yyyy-mm-dd hh24:mi:ss') order by START_DATE;
SQL> col program for a20
SQL> col event for a30
SQL> select inst_id,sid,serial#,status ,program,event,sql_id from gv$session where taddr='070001282BD5A440';

             INST_ID                  SID              SERIAL# STATUS   PROGRAM              EVENT                          SQL_ID
-------------------- -------------------- -------------------- -------- -------------------- ------------------------------ -------------
                   1                 1911                34435 INACTIVE plsqldev.exe         SQL*Net message from client

SQL>
SQL> col program for a20
SQL> col event for a30
SQL> select inst_id,sid,serial#,status ,program,event,sql_id from gv$session where taddr='070001282BD5A440';

             INST_ID                  SID              SERIAL# STATUS   PROGRAM              EVENT                          SQL_ID
-------------------- -------------------- -------------------- -------- -------------------- ------------------------------ ---------
                   1                 1911                34435 INACTIVE plsqldev.exe         SQL*Net message from client

SQL>

建议
1、增加数据库中长事务和大事务的监控。比如，对于超过4小时（具体告警阈值根据业务情况设定）的事务进行告警，提醒DBA进行关注和处理。
2、监控归档空间使用率。由于归档日志文件不能正常删除，归档目录空间使用率可能会逐步增加，有些业务情况下归档目录空间可能暴增，所以必须加强归档空间使用率监控。
3、使用产品对数据库日志和备份产品的日志进行管理。

cv备份日志检查
1、执行simpana status命令

 # simpana  status
--------------------------------- Instance001 ---------------------------------
[ General ]
 Version = 10(BUILD116) SP14
 CommServe = CommServer
 Home Directory = /cv/simpana/Base
 Log Directory = /cv/log/simpana/Log_Files
 Core Directory = /cv/simpana
 Temp Directory = /cv/simpana/Base/Temp
 Platform Type = 4
 Cvd Port Number = 8400
 EvMgrC Port Number = 8402
[ Package ]
 1002/CVGxBase 10(BUILD116) = File System Core
 1101/CVGxIDA 10(BUILD116) = File System
 1112/CVGxDC 10(BUILD116) = Data Classification Enabler
  - Cache Root Directory = /cv/simpana/iDataAgent/jobResults/cvdccache
 1113/CVGxFSF 10(BUILD116) = File System Filter Driver
 1204/CVGxOrIDA 10(BUILD116) = Oracle iDataAgent
[ Physical Machine/Cluster Groups ]
 Display Name = pipdb1
  - Client Hostname = pipdb1
  - **Job Results Directory = /cv/simpana/iDataAgent/jobResults**
#

2、进入备份日志目录
cd /cv/simpana/iDataAgent/jobResults/CV_JobResults/2/0/204733
3、查看备份日志

# cat backup.out
Rman Script:
[CONFIGURE CONTROLFILE AUTOBACKUP ON;
run {
allocate channel ch1 type 'sbt_tape'
PARMS="SBT_LIBRARY=/cv/simpana/Base64/libobk.a(shr.o),BLKSIZE=262144,ENV=(CV_mmsApiVsn=2,CV_channelPar=ch1,ThreadCommandLine= -cn pipdb1 -vm Instance001)"
TRACE 0;
send "BACKUP -jm 45 -a 2:590 -cl 236 -ins 96 -at 22 -j 204733 -jt 204733:5:1 -bal 1 -rcp 0 -ms 1 -logs -ma 200";
setlimit channel ch1 maxopenfiles 8;
sql "alter system archive log current";
 backup 
 filesperset = 32 
format='204733_%d_%U'
 (archivelog  all   not backed up 1 times );
 delete noprompt archivelog  until time = 'sysdate-1'  ; 
}
exit;
]
Rman Log:[
Recovery Manager: Release 11.2.0.4.0 - Production on Mon Dec 4 12:30:21 2023
Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.
RMAN> 
RMAN> 
connected to target database: PIPDB (DBID=2011412252)
using target database control file instead of recovery catalog
RMAN> 
old RMAN configuration parameters:
CONFIGURE CONTROLFILE AUTOBACKUP ON;
new RMAN configuration parameters:
CONFIGURE CONTROLFILE AUTOBACKUP ON;
new RMAN configuration parameters are successfully stored
RMAN> 2> 3> 4> 5> 6> 7> 8> 9> 10> 11> 12> 13> 
allocated channel: ch1
channel ch1: SID=1059 instance=pipdb1 device type=SBT_TAPE
channel ch1: CommVault Systems for Oracle: Version 10.0.0(BUILD116)
sent command to channel: ch1
sql statement: alter system archive log current
Starting backup at Dec 04 2023 12:30:36
current log archived
skipping archived logs of thread 1 from sequence 196613 to 196769; already backed up
skipping archived logs of thread 2 from sequence 196707 to 196860; already backed up
channel ch1: starting archived log backup set
channel ch1: specifying archived log(s) in backup set
input archived log thread=2 sequence=196861 RECID=394292 STAMP=1154689985
input archived log thread=1 sequence=196770 RECID=394291 STAMP=1154689828
input archived log thread=1 sequence=196771 RECID=394293 STAMP=1154690301
input archived log thread=2 sequence=196862 RECID=394294 STAMP=1154690546
input archived log thread=1 sequence=196772 RECID=394295 STAMP=1154690731
input archived log thread=2 sequence=196863 RECID=394296 STAMP=1154691155
input archived log thread=1 sequence=196773 RECID=394297 STAMP=1154691247
input archived log thread=2 sequence=196864 RECID=394299 STAMP=1154691793
input archived log thread=1 sequence=196774 RECID=394298 STAMP=1154691768
input archived log thread=1 sequence=196775 RECID=394300 STAMP=1154692275
input archived log thread=2 sequence=196865 RECID=394301 STAMP=1154692483
input archived log thread=1 sequence=196776 RECID=394302 STAMP=1154692921
input archived log thread=2 sequence=196866 RECID=394303 STAMP=1154693321
input archived log thread=1 sequence=196777 RECID=394304 STAMP=1154693774
input archived log thread=2 sequence=196867 RECID=394305 STAMP=1154693980
input archived log thread=1 sequence=196778 RECID=394306 STAMP=1154694631
input archived log thread=2 sequence=196868 RECID=394307 STAMP=1154694631
input archived log thread=1 sequence=196779 RECID=394309 STAMP=1154694639
input archived log thread=2 sequence=196869 RECID=394308 STAMP=1154694637
channel ch1: starting piece 1 at Dec 04 2023 12:30:48
channel ch1: finished piece 1 at Dec 04 2023 12:35:33
piece handle=204733_PIPDB_402d6ffo_1_1 tag=TAG20231204T123047 comment=API Version 2.0,MMS Version 10.0.0.116
channel ch1: backup set complete, elapsed time: 00:04:45
Finished backup at Dec 04 2023 12:35:33


Starting Control File and SPFILE Autobackup at Dec 04 2023 12:35:33
piece handle=c-2011412252-20231204-1b comment=API Version 2.0,MMS Version 10.0.0.116
Finished Control File and SPFILE Autobackup at Dec 04 2023 12:35:40


List of Archived Log Copies for database with db_unique_name XFDB
=====================================================================
Key     Thrd Seq     S Low Time            
------- ---- ------- - --------------------
393980  1    196613  A Dec 03 2023 11:48:26
        Name: +ARCH/xfdb/archivelog/2023_12_03/thread_1_seq_196613.398.1154607015 
...
archived log file name=+ARCH/xfdb/archivelog/2023_12_03/thread_2_seq_196710.832.1154608353 RECID=393988 STAMP=1154608353
deleted archived log
archived log file name=+ARCH/xfdb/archivelog/2023_12_03/thread_2_seq_196711.451.1154609721 RECID=393990 STAMP=1154609723
Deleted 11 objects
released channel: ch1
RMAN> 
Recovery Manager complete.
]
#  

-the end-