When dealing with a Oracle Real Application Cluster (RAC) issue, a DBA generally needs to jump around multiple directories to find the relevant logs. Those days are now history as Oracle has centralized all the Clusterware (RAC) log files in a common location starting with Oracle version 12c.
Troubleshooting a RAC issue is now made easier as we can easily locate all the required log/trace files under a centralized location. We need not jump around different directories to find the Clusterware related log files. We can now free up a tiny slot of memory in our brain as it doesnít need to remember the different log locations for a Oracle Clusterware setup, which we used to do for pre 12c Oracle RAC databases.
Further, Oracle has changed the naming conventions for the Clusterware logs starting with Oracle 12.1.0.2. All the Clusterware log files are now termed as TRACE files and ends with .trc extension (except the Clusterware alert log) rather than the .log extension which was the case in prior releases.
Starting with Oracle 12c, the Clusterware log/trace files are stored in the Automatic Diagnostic Repository (ADR) under the ADR_HOME location $ADR_BASE/diag/crs/hostname/crs. We can confirm that from ADRCI command line as shown below
—//
—// Finding ADR_HOME for Clusterware logs //—
—//
myrac01 {/home/oracle}: adrci
ADRCI: Release 12.1.0.2.0 – Production on Wed Jan 13 04:01:00 2016
Copyright (c) 1982, 2014, Oracle and/or its affiliates. All rights reserved.
ADR base = “/app/oracle”
adrci> show homes
ADR Homes:
diag/tnslsnr/myrac01/mgmtlsnr
diag/rdbms/_mgmtdb/-MGMTDB
diag/crs/myrac01/crs
Letís quickly walk through the new centralized Clusterware log location to identify the related Clusterware log/trace files.
Oracle stores all the Clusterware diagnostic log/trace files under trace directory of Clusterware ADR_HOME.
—//
—// Locating Clusterware centralized log repository //—-
—//
myrac01 {/home/oracle}: ls -ld $ADR_BASE/diag/crs/hostname/crs/trace
drwxrwxr-x 2 oracle oinstall 876544 Jan 13 04:02 /app/oracle/diag/crs/myrac01/crs/trace
—//
—// moving to Clusterware centralized log repository //—-
—//
myrac01 {/home/oracle}: cd $ADR_BASE/diag/crs/hostname/crs/trace
Clusterware alert log file
—//
—// Clusterware alert log file //—-
—//
myrac01 {/app/oracle/diag/crs/myrac01/crs/trace}: ls -lrt alert.log
-rw-rw—- 1 oracle oinstall 1024107 Jan 13 04:04 alert.log
Cluster Ready Services (CRS) daemon/agent trace files
—//
—// crs daemon and agent trace files //—-
—//
myrac01 {/app/oracle/diag/crs/myrac01/crs/trace}: ls -lrt crs* | tail -6
-rw-rw—- 1 root oinstall 1340371 Jan 13 04:03 crsd.trm
-rw-rw—- 1 root oinstall 7238701 Jan 13 04:03 crsd.trc
-rw-rw—- 1 oracle oinstall 739563 Jan 13 04:04 crsd_oraagent_oracle.trm
-rw-rw—- 1 oracle oinstall 4371314 Jan 13 04:04 crsd_oraagent_oracle.trc
-rw-rw—- 1 root oinstall 650733 Jan 13 04:04 crsd_orarootagent_root.trm
-rw-rw—- 1 root oinstall 3565651 Jan 13 04:04 crsd_orarootagent_root.trc
Cluster Synchronization Services (CSS) daemon trace files
—//
—// css daemon trace files //—-
—//
myrac01 {/app/oracle/diag/crs/myrac01/crs/trace}: ls -lrt ocssd* | tail -2
-rw-rw—- 1 oracle oinstall 443750 Jan 13 04:04 ocssd.trm
-rw-rw—- 1 oracle oinstall 2868877 Jan 13 04:04 ocssd.trc
High Availability Service (HAS) daemon/agent trace files
—//
—// has daemon and agent trace files //—-
—//
myrac01 {/app/oracle/diag/crs/myrac01/crs/trace}: ls -lrt ohasd* | tail -6
-rw-rw—- 1 root oinstall 783540 Jan 13 04:04 ohasd_orarootagent_root.trm
-rw-rw—- 1 root oinstall 4805435 Jan 13 04:04 ohasd_orarootagent_root.trc
-rw-rw—- 1 root oinstall 211308 Jan 13 04:04 ohasd.trm
-rw-rw—- 1 root oinstall 1225033 Jan 13 04:04 ohasd.trc
-rw-rw—- 1 oracle oinstall 1382492 Jan 13 04:04 ohasd_oraagent_oracle.trm
-rw-rw—- 1 oracle oinstall 6877368 Jan 13 04:04 ohasd_oraagent_oracle.trc
Event Management (EVM) daemon trace files.
—//
—// evm daemon trace files //—-
—//
myrac01 {/app/oracle/diag/crs/myrac01/crs/trace}: ls -lrt evm* | tail -2
-rw-rw—- 1 oracle oinstall 1562665 Jan 13 04:04 evmd.trm
-rw-rw—- 1 oracle oinstall 10052660 Jan 13 04:04 evmd.trc
Grid Interprocess Communication (GIPC) daemon trace files
—//
—// gipc daemon trace files //—-
—//
myrac01 {/app/oracle/diag/crs/myrac01/crs/trace}: ls -lrt gip* | tail -2
-rw-rw—- 1 oracle oinstall 763055 Jan 13 04:04 gipcd.trm
-rw-rw—- 1 oracle oinstall 5507597 Jan 13 04:04 gipcd.trc
Cluster System Monitor daemon trace files
—//
—// sysmon daemon trace files //—-
—//
myrac01 {/app/oracle/diag/crs/myrac01/crs/trace}: ls -lrt osysmon* | tail -2
-rw-rw—- 1 root oinstall 557926 Jan 13 03:46 osysmond.trm
-rw-rw—- 1 root oinstall 2867760 Jan 13 03:46 osysmond.trc
Cluster Logger Service daemon trace files
—//
—// logger daemon trace files //—-
—//
myrac01 {/app/oracle/diag/crs/myrac01/crs/trace}: ls -lrt ologgerd* | tail -2
-rw-rw—- 1 root oinstall 410752 Jan 13 03:48 ologgerd.trm
-rw-rw—- 1 root oinstall 2742627 Jan 13 03:48 ologgerd.trc
Apart from the Clusterware service (daemon) trace files, the diagnostic location also contains the client utility trace files as shown below
OCRCHECK utility trace files
—//
—// ocrcheck client trace files //—-
—//
myrac01 {/app/oracle/diag/crs/myrac01/crs/trace}: ls -lrt ocrcheck* | tail -2
-rw-rw—- 1 oracle oinstall 137 Jan 13 01:30 ocrcheck_31338.trm
-rw-rw—- 1 oracle oinstall 310 Jan 13 01:30 ocrcheck_31338.trc
OCRCONFIG utility trace files.
—//
—// ocrconfig client trace files //—-
—//
myrac01 {/app/oracle/diag/crs/myrac01/crs/trace}: ls -lrt ocrconfig* | tail -2
-rw-rw—- 1 oracle oinstall 137 Jan 13 01:30 ocrconfig_30866.trm
-rw-rw—- 1 oracle oinstall 312 Jan 13 01:30 ocrconfig_30866.trc
CRSCTL utility trace files.
—//
—// crsctl client trace files //—-
—//
myrac01 {/app/oracle/diag/crs/myrac01/crs/trace}: ls -lrt crsctl* | tail -2
-rw-rw—- 1 oracle oinstall 175 Jan 13 01:31 crsctl_1103.trm
-rw-rw—- 1 oracle oinstall 729 Jan 13 01:31 crsctl_1103.trc
Cluster Monitor utility (oclumon) trace files
—//
—// oclumon client trace files //—-
—//
myrac01 {/app/oracle/diag/crs/myrac01/crs/trace}: ls -lrt oclumon* | tail -2
-rw-rw—- 1 oracle oinstall 166 Dec 17 03:54 oclumon_2175.trm
-rw-rw—- 1 oracle oinstall 387 Dec 17 03:54 oclumon_2175.trc
Grid Plug and Play utility (GPNP) trace files
—//
—// gpnp client trace files //—-
—//
myrac01 {/app/oracle/diag/crs/myrac01/crs/trace}: ls -lrt gpn* | tail -2
-rw-rw—- 1 oracle oinstall 205 Jan 13 01:31 gpnptool_664.trm
-rw-rw—- 1 oracle oinstall 608 Jan 13 01:31 gpnptool_664.trc
As we can observe, it is now very easy to locate all the Clusterware trace/log files under a single centralized log repository. This will indeed help DBAs to save their precious time while troubleshooting Clusterware related issues.
