Exadata环境ORA-00603ANDORA-27515报错的分析处理

 在近期的系统巡检检查中，发现一个最近新安装上线的EXADATA一体机环境的数据库后台ALERT日志中有报错ORA-00603: ORACLE server session terminated by fatal error    ORA-27515: inadequate memlock limit or driver settings；初看起来是进程异常关闭，检查发现数据库未宕机，查看TRACE为普通的用户进程而不是数据库后台进程，业务使用正常。

报错日志如下：

继续分析此报错，结合ORACLE MOS文档 EXADATA : ORA-00603 AND ORA-27515 during EXAChk run with Autonomous Health Framework (AHF) 20.1.1. (Doc ID 2662144.1)、Instance crashes with ORA-00603, ORA-27515 (Doc ID 2700814.1)等文档，初步判断是AHF工具BUG导致。至于解决方式，官方给出的是升级AHF打补丁或者关闭AHF，我这里使用了最简单的关闭方式，步骤命令如下：（root用户下执行，2个计算节点都执行）

#exachk -autostop

#tfactl stop

MOS中描述如下：

Applies to:

Gen 1 Exadata Cloud at Customer (Oracle Exadata Database Cloud Machine) - Version N/A to N/A Information in this document applies to any platform.

Symptoms

Instance failure with following errors reported in alert log.

2020-07-14T01:11:58.695144-05:00 Archived Log entry 1275 added for T-1.S-2367 ID 0x2c5d558c LAD:1 2020-07-14T01:13:53.787461-05:00 Errors in file /u02/app/oracle/diag/rdbms/cd1sfm/cd1sfm1/trace/cd1sfm1_ora_167850.trc  (incident=770730) (PDBNAME=CDB$ROOT): ORA-00603: ORACLE server session terminated by fatal error ORA-27515: inadequate memlock limit or driver settings Incident details in: /u02/app/oracle/diag/rdbms/cd1sfm/cd1sfm1/incident/incdir_770730/cd1sfm1_ora_167850_i770730.trc ..... 2020-07-14T01:14:58.631500-05:00 opiodr aborting process unknown ospid (167850) as a result of ORA-603 2020-07-14T01:17:23.410311-05:00 Errors in file /u02/app/oracle/diag/rdbms/cd1sfm/cd1sfm1/trace/cd1sfm1_ora_172703.trc  (incident=770475) (PDBNAME=SFDAT): ORA-00603: ORACLE server session terminated by fatal error ORA-27515: inadequate memlock limit or driver settings SFDAT(3):Incident details in: /u02/app/oracle/diag/rdbms/cd1sfm/cd1sfm1/incident/incdir_770475/cd1sfm1_ora_172703_i770475.trc 2020-07-14T01:17:25.642557-05:00 SFDAT(3):opiodr aborting process unknown ospid (172703) as a result of ORA-603 2020-07-14T01:19:19.671875-05:00

Cause

 Internal  Bug 31118086 involving TFA  

Solution

AHF version 20.1.3 or above contains the fix.

Please upgrade to the latest available versions of AHF

====

Applies to:

Oracle Database - Enterprise Edition - Version 12.2.0.1 to 19.6.0.0.0 [Release 12.2 to 19] Linux x86-64

Symptoms

We see following errors in the RAC Databases :

Excerpts from database Alert log:

Errors in file /u01/app/oracle/diag/rdbms/xxxx/xxxx1/trace/xxxx1_ora_9460.trc (incident=1737933) (PDBNAME=CDB$ROOT): ORA-00603: ORACLE server session terminated by fatal error ORA-27515: inadequate memlock limit or driver settings Incident details in: /u01/app/oracle/diag/rdbms/xxxx/xxxx1/incident/incdir_1737933/xxxx1_ora_9460_i1737933.trc 2020-03-24T01:03:25.650686-04:00 opiodr aborting process unknown ospid (9460) as a result of ORA-603

oracle$ ulimit -a max locked memory (kbytes, -l) unlimited

root# ulimit -a max locked memory (kbytes, -l) 32768

Changes

 Autonomous Health Framework (AHF) 20.1.1 Installed

exachk -v EXAchk Version 20.1.1_20200317            <<<<<<<<<

Cause

ORA-27515 is hit when EXAChk is either run manually or TFA initiated.

 The issue is due to following unpublished EXAChk bug:

   Bug 31118086 - AHF 20.1.1: ORA-00603, ORA-27515 OCCURRED ONLY DURING EXACHK RUN     

Solution

Bug 31118086 is fixed in AHF 20.1.3.

Upgrade to Autonomous Health Framework (AHF) 20.1.3 version.

  = OR =

Use the following workaround:

#exachk -autostop #tfactl stop