Understanding how the clusterware startup occurs is critical to the diagnosis and resolution of Oracle RAC problems.
In Unix and Linux operating systems, there is a master daemon process named INIT that functions to start up additional system background processes. The INIT process first spawns the init.ohasd process, which in turn starts up the Oracle High Availability Services Daemon (OHASD). In turn, the OHASD daemon then spawns additional Clusterware processes at each startup level as shown next:
Cssdagent: Agent responsible for spawning CSSDOrarootagent: Agent responsible for managing all root-owned ohasd resourcesOraagent: Agent responsible for managing all Oracle-owned ohasd resourcescssdmonitor: Monitors CSSD and node health (along wth the cssdagent)
Level 2—OHASD rootagent spawns:
During the second level of startup for Clusterware, the oraagent spawns the following Clusterware processes for 11g R2:
MDNSD: Used for DNS lookup
GIPCD: Used for inter-process and inter-node communication
GPNPD: Grid Plug and Play Profile Daemon
EVMD: Event Monitor Daemon
ASM: Resource for monitoring ASM instances
orarootagent: Agent responsible for managing all root-owned CRSD resourcesoraagent: Agent responsible for managing all Oracle-owned CRSD resources
Level 4—CRSD rootagent spawns:
Network resource: To monitor the public network
SCAN VIP(s): Single Client Access Name Virtual IPs
Node VIPs: One per node
ACFS Registery: For mounting ASM Cluster File system
GNS VIP (optional): VIP for GNS
During this phase for Clusterware startup with 11g R2, the oraagent spawns the following processes:
ASM Resouce: ASM Instance(s) resource
Diskgroup: Used for managing/monitoring ASM diskgroups
DB Resource: Used for monitoring and managing the DB and instances
SCAN Listener: Listener for single client access name, listening on SCAN VIP
Listener: Node listener listening on the Node VIP
Services: Used for monitoring and managing services
ONS: Oracle Notification Service
eONS: Enhanced Oracle Notification Service
GSD: For 9i backward compatibility
GNS (optional): It is a grid naming service that performs name resolution
The important Clusterware daemon logs are located under the <GRID_HOME>/log/<nodename> directory. There are additional logfiles located under the <GRID_HOME>/log/<nodename>directory as listed next:
alert<NODENAME>.log - look here first for most clusterware issues ./admin: ./agent: ./agent/crsd: ./agent/crsd/oraagent_oracle: ./agent/crsd/ora_oc4j_type_oracle: ./agent/crsd/orarootagent_root: ./agent/ohasd: ./agent/ohasd/oraagent_oracle: ./agent/ohasd/oracssdagent_root: ./agent/ohasd/oracssdmonitor_root: ./agent/ohasd/orarootagent_root: ./client: ./crsd: ./cssd: ./ctssd: ./diskmon: ./evmd: ./gipcd: ./gnsd: ./gpnpd: ./mdnsd: ./ohasd: ./racg: ./racg/racgeut: ./racg/racgevtf: ./racg/racgmain: ./srvm:
The cleanup of these logfiles can be automated with a shell script that can be scheduled to run from the cron facility. The cfgtoollogs directory located under the $GRID_HOME and $ORACLE_BASE directories contains other important logfiles such as the rootcrs.pl script and configuration assistants, such as the ASM Configuration Assistant (ASMCA).
ASM logs are located under $ORACLE_BASE/diag/asm/+asm/<ASM Instance Name>/trace
Oracle provides a useful script named diagcollection.pl, which can be found under the <GRID_HOME>/bin directory. This script, when executed, will automatically collect important Oracle 11g Clusterware files that can be sent to Oracle support engineers for analysis. You must be logged into Linux as the root user in order to run this script.