数据库管理-第102期 EXPDP遇到的问题（20230913）

之前迁移一个PDB的时候，发现源CDB安装了DV（Database Vault）组件，但是目标CDB没有安装，因此在datapatch和启动的时候报错，无法正常使用，因此迁移直接回滚了。于是想着测试下用DUMP的方式迁移，结果又遇到一大堆问题。

1 问题出现

在expdp的时候使用的是下面的命令：

expdp system@pdb_xxxx directory=ext dumpfile=xxxx_expdp%U.dmp logfile=xxxx_expdp.log schemas=USERNAME parallel=32 cluster=n compression=all

因为本机可用本地磁盘空间仅有2TB，实际数据量也为2TB左右，为了避免空间不足，因此使用了压缩。但是在执行过程中出现了以下报错：

ORA-39126: Worker unexpected fatal error in KUPW$WORKER.FETCH_XML_OBJECTS [ORA-22814: attribute or element value is larger than specified in type
ORA-06512: at "SYS.DBMS_SYS_ERROR", line 105
ORA-06512: at "SYS.KUPW$WORKER", line 12630
ORA-06512: at "SYS.DBMS_SYS_ERROR", line 105
ORA-06512: at "SYS.DBMS_METADATA", line 9571
ORA-06512: at "SYS.DBMS_METADATA", line 4250
ORA-06512: at "SYS.DBMS_METADATA", line 5008
ORA-06512: at "SYS.DBMS_METADATA", line 5327
ORA-06512: at "SYS.DBMS_METADATA", line 9552
ORA-06512: at "SYS.KUPW$WORKER", line 15119
----- PL/SQL Call Stack -----
object line object
handle number name
0x73636f2920 33543 package body SYS.KUPW$WORKER.WRITE_ERROR_INFORMATION
0x73636f2920 12651 package body SYS.KUPW$WORKER.DETERMINE_FATAL_ERROR
0x73636f2920 15452 package body SYS.KUPW$WORKER.FETCH_XML_OBJECTS
0x73636f2920 3917 package body SYS.KUPW$WORKER.UNLOAD_METADATA
0x73636f2920 13746 package body SYS.KUPW$WORKER.DISPATCH_WORK_ITEMS
0x73636f2920 2439 package body SYS.KUPW$WORKER.MAIN
0x7be2734c8 2 anonymous block
DBMS_METADATA.SET_FILTER
DBMS_METADATA.SET_FILTER
DBMS_METADATA.SET_FILTER
KUPW: In FETCH_XML_OBJECTS
KUPW: End seqno is: 278
KUPF$FILE.OPEN_CONTEXT
KUPF$FILE.OPEN_CONTEXT
DBMS_METADATA.FETCH_XML_CLOB
DBMS_METADATA.FETCH_XML_CLOB
KUPW: In procedure DETERMINE_FATAL_ERROR with ORA-22814: attribute or element value is larger than specified in type
ORA-06512: at "SYS.DBMS_METADATA", line 9571
ORA-06512: at "SYS.DBMS_METADATA", line 4250
ORA-06512: at "SYS.DBMS_METADATA", line 5008
ORA-06512: at "SYS.DBMS_METADATA", line 5327
ORA-06512: at "SYS.DBMS_METADATA", line 9552
ORA-39126: Worker unexpected fatal error in KUPW$WORKER.FETCH_XML_OBJECTS [ORA-22814: attribute or element value is larger than specified in type
ORA-06512: at "SYS.DBMS_SYS_ERROR", line 105
ORA-06512: at "SYS.KUPW$WORKER", line 12630
ORA-06512: at "SYS.DBMS_SYS_ERROR", line 105
ORA-06512: at "SYS.DBMS_METADATA", line 9571
ORA-06512: at "SYS.DBMS_METADATA", line 4250
ORA-06512: at "SYS.DBMS_METADATA", line 5008
ORA-06512: at "SYS.DBMS_METADATA", line 5327
ORA-06512: at "SYS.DBMS_METADATA", line 9552
ORA-06512: at "SYS.KUPW$WORKER", line 15119
----- PL/SQL Call Stack -----
object line object
handle number name
0x73636f2920 33543 package body SYS.KUPW$WORKER.WRITE_ERROR_INFORMATION
0x73636f2920 12651 package body SYS.KUPW$WORKER.DETERMINE_FATAL_ERROR
0x73636f2920 15452 package body SYS.KUPW$WORKER.FETCH_XML_OBJECTS
0x73636f2920 3917 package body SYS.KUPW$WORKER.UNLOAD_METADATA
0x73636f2920 13746 package body SYS.KUPW$WORKER.DISPATCH_WORK_ITEMS
0x73636f2920 2439 package body SYS.KUPW$WORKER.MAIN
0x7be2734c8 2 anonymous block
DBMS_METADATA.SET_FILTER
DBMS_METADATA.SET_FILTER
DBMS_METADATA.SET_FILTER
KUPW: In FETCH_XML_OBJECTS
KUPW: End seqno is: 278
KUPF$FILE.OPEN_CONTEXT
KUPF$FILE.OPEN_CONTEXT
DBMS_METADATA.FETCH_XML_CLOB
DBMS_METADATA.FETCH_XML_CLOB
KUPW: In procedure DETERMINE_FATAL_ERROR with ORA-22814: attribute or element value is larger than specified in type
ORA-06512: at "SYS.DBMS_METADATA", line 9571
ORA-06512: at "SYS.DBMS_METADATA", line 4250
ORA-06512: at "SYS.DBMS_METADATA", line 5008
ORA-06512: at "SYS.DBMS_METADATA", line 5327
ORA-06512: at "SYS.DBMS_METADATA", line 9552

因为DUMP不会对数据库运行造成影响，所以默认情况下数据库不会记录相关的告警和追踪日志。查问题陷入了僵局。

2 找到问题

既然DUMP本身不会生成告警和追踪日志，没法进一步分析那么就收工开启相关记录：

alter system set events='22814 trace name errorstack level 3';

再次执行DUMP仍然失败报错，但是在告警日志中出现了以下内容：

PDB_XXXX(6):Errors in file /u01/app/oracle/diag/rdbms/xxdbaas/xxdbaas2/trace/xxdbaas2_dw0c_171811.trc:
ORA-39126: Worker unexpected fatal error in KUPW$WORKER.FETCH_XML_OBJECTS [ORA-06512: at "SYS.DBMS_METADATA", line 9571
ORA-06512: at "SYS.DBMS_METADATA", line 4250
ORA-06512: at "SYS.DBMS_METADATA", line 5008
ORA-06512: at "SYS.DBMS_METADATA", line 5327
ORA-06512: at "SYS.DBMS_METADATA", line 9552
ORA-06512: at "SYS.KUPW$WORKER", line 15119
]
...
PDB_XXXX(6):Errors in file /u01/app/oracle/diag/rdbms/xxdbaas/xxdbaas2/trace/xxdbaas2_dw0a_171850.trc:
ORA-39126: Worker unexpected fatal error in KUPW$WORKER.FETCH_XML_OBJECTS [ORA-06512: at "SYS.DBMS_METADATA", line 9571
ORA-06512: at "SYS.DBMS_METADATA", line 4250
ORA-06512: at "SYS.DBMS_METADATA", line 5008
ORA-06512: at "SYS.DBMS_METADATA", line 5327
ORA-06512: at "SYS.DBMS_METADATA", line 9552
ORA-06512: at "SYS.KUPW$WORKER", line 15119
]

在对应的追踪日志中也找到了出现问题的地方：

----- Error Stack Dump -----
<error barrier> at 0x7ffd40eab450 placed dbkda.c@298
ORA-39126: Worker unexpected fatal error in KUPW$WORKER.FETCH_XML_OBJECTS [ORA-06512: at "SYS.DBMS_METADATA", line 9571
ORA-06512: at "SYS.DBMS_METADATA", line 4250
ORA-06512: at "SYS.DBMS_METADATA", line 5008
ORA-06512: at "SYS.DBMS_METADATA", line 5327
ORA-06512: at "SYS.DBMS_METADATA", line 9552
ORA-06512: at "SYS.KUPW$WORKER", line 15119
]
SCHEMA_EXPORT/JOB
<error barrier> at 0x7fb0fe339fb0 placed pfrrun.c@3799
ORA-22814: attribute or element value is larger than specified in type
ORA-06512: at "SYS.DBMS_METADATA", line 9571
ORA-06512: at "SYS.DBMS_METADATA", line 4250
ORA-06512: at "SYS.DBMS_METADATA", line 5008
ORA-06512: at "SYS.DBMS_METADATA", line 5327
ORA-06512: at "SYS.DBMS_METADATA", line 9552
<error barrier> at 0x7ffd40eb17e0 placed kpoodr.c@237
----- Current SQL Statement for this session (sql_id=4q4xc5vstbfq8) -----
BEGIN
SYS.KUPW$WORKER.MAIN('SYS_EXPORT_SCHEMA_01', 'SYSTEM', 0, 0);
END;

这里可以看到是在导出JOB的时候报的错。

3 处理

既然发现了是JOB的问题，但是通过告警日志无法排查出事哪个JOB造成的问题，因此调整了导出命令，将JOB暂时排除：

expdp system@pdb_xxxx directory=ext dumpfile=xxxx_expdp%U.dmp logfile=xxxx_expdp.log schemas=USERNAME parallel=32 cluster=n compression=all EXCLUDE=JOB

导出导入后再通过手动方式去迁移JOB。
当然最后记得关闭trace:

alter system set events='22814 trace name errorstack off';

当然最后测试完了发现超时严重，也不会采用DUMP的方式来迁移数据。

总结

本次DUMP的问题处理，其实主要是通过指定event的方式去抓取对应报错的trace信息并生成追踪日志，然后通过追踪日志去定位问题。
老规矩，知道写了些啥。